DIM-Pack
DIM-Pack provides tools for conducting statistical dimensionality analyses on assessments consisting of dichotomously scored tasks. To understand these tools and what they do, one first needs to understand, of course, what is meant by “dimensions”. The dimensions of an assessment instrument can be thought of as the attributes that are intended to be measured on the test takers. For example, in an educational measurement setting, we may want to measure a student's level of achievement in mathematics, reading, science, and writing – a case of four dimensions. Any one of those four dimensions may be further hypothesized to be decomposable into lower dimensional skills. In general, it is not usually clear how many dimensions are on a test.
With this understanding of what we mean by dimensions, we can proceed to talk in more detail about dimensionality analysis. The purpose of a dimensionality analysis is generally two-fold:
- Determine if there is more than one statistically detectable dimension (the presence of one dimension is termed “unidimensionality”) and
- If there are multiple detectable dimensions (“multidimensionality”), describe the nature of the multiple dimensions.
Download
DIMTEST-Latent Unidimensionality Assessment
DIMTEST is a hypothesis testing procedure that assesses lack of latent unidimensionality for a dichotomously scored test. It does so by assessing the dimensional distinctiveness between two specified subtests, referred to as the Assessment Subtest (AT) and the Partitioning Subtest (PT). DIMTEST, originally developed by Stout (1987) has been refined and improved by many researchers (see in particular Nandakumar & Stout, 1993, and Stout, Froelich, & Gao, 2001)
DIMTEST operates in either a confirmatory mode or an exploratory mode. In the confirmatory mode, it assesses the user-selected AT that a priori, likely on substantive grounds, has been judged to be possibly dimensionally distinct from the user-selected PT. In the exploratory mode, using cross-validation, it assesses unidimensionality by using a statistically selected AT that is potentially maximally dimensionally distinct from PT. DIMTEST is completely nonparametric, requiring no parametric IRT modeling. In particular, DIMTEST does not require estimation of item response functions. It instead depends on estimation of conditional item pair correlations among the AT items, each conditional on the dominant latent ability measured by the PT items. Under the DIMTEST null hypothesis of unidimensionality, the AT and PT items measure the same ability, and the estimated conditional covariances should be within noise of zero. Under the DIMTEST alternative hypothesis, AT measures an ability that is distinct from the ability measured by PT; and, thus, the estimated conditional covariances on AT will be significantly greater than zero.
DETECT-Full Dimensionality Analysis
One of the most common types of test formats is one in which the items are grouped around several distinct reading passages or underlying psychological, instructional, curricular, or cognitive constructs–for example, algebra, geometry, and trigonometry in a mathematics test. Using estimated item pair conditional covariances, the DETECT procedure nonparametrically provides a detailed dimensional description of this type of test, which is said to exhibit approximate simple structure.
DETECT begins by using clusters obtained from the dimensionality-sensitive cluster analysis procedure HCA/CCPROX and then uses a customized genetic algorithm to efficiently search through all of the possible item cluster partitions to find the one that approximately maximizes the DETECT statistic. In addition to quickly finding a dimensionally highly appropriate grouping of the test’s items and thereby estimating the number of dominant dimensions present, it also provides two statistics that summarize of the total amount of multidimensionality on the test. One statistic measures the lack of fit by a unidimensional model, and the other measures the degree to which the test displays approximate simple structure. This information is often useful from a statistical robustness perspective for those wishing to use IRT methodologies based on unidimensionality. DETECT presumes dichotomous item scoring.
DETECT can be run in either confirmatory or exploratory mode. In confirmatory mode, the user specifies a set of non-overlapping clusters for the test items and DETECT calculates its indices for that clustering. In exploratory mode, the data are split into a training sample and a cross-validation sample. The genetic algorithm searches for the best clusters on the training sample and then the cross-validation sample is used to calculate the DETECT indices.
HCA/CCPROX-Hierarchical Cluster Analysis with Dimensionally-Sensitive Proximity Matrices
HCA/CCPROX performs a latent multidimensionality-sensitive hierarchical cluster analysis on dichotomously scored items. This nonparametric procedure is able to quickly cluster the items into progressively larger and larger relatively dimensionally homogeneous groups. It allows the user to examine the test’s dimensionality at a variety of agglomeration levels, ranging from which pairs of items are most closely dimensionally related, to which two-cluster solution best dimensionally summarizes the entire test. If the items of a test exist in approximately dimensionally homogeneous item clusters, then there should exist a level in the hierarchy at which the clusters found by HCA/CCPROX will maximally agree with this approximate simple structure. Even if approximate simple structure does not hold, HCA/CCPROX will tend to find dimensionally disparate clusters, and hence its uses are not limited to tests demonstrating approximate simple structure. HCA/CCPROX is useful in tandem with a DIMTEST assessment of dimensionality, because DIMTEST can test hypotheses concerning HCA/CCPROX selected clusters.