This book is Work in Progress. I appreciate your feedback to make the book better.

8.7 Items

Item response theory

Factor analysis begins with several indicators and asks what latent variable could explain why they covary. Item response theory turns the same measurement problem around and looks directly at each response:

How likely is a person with a particular level of competence to solve this particular item?

This shift makes persons and items equally visible. A response can be unlikely because the person has less of the relevant competence, because the item is difficult, or because the model does not adequately describe the response process.

8.7.1 Where IRT Sits among Latent Variable Models

A useful first map distinguishes the measurement level of the observed indicators from the form of the latent variable.

Observed indicators Continuous latent variable Categorical latent variable
Continuous factor analysis latent profile analysis
Categorical item response theory / categorical factor analysis latent class analysis

Item response theory is therefore not one isolated model in one cell. It is the main family of latent-trait models for categorical item responses. Categorical confirmatory factor analysis occupies much of the same mathematical territory, but uses the language of loadings and thresholds rather than item discrimination and difficulty.

The table is a map, not an exhaustive classification. Models may contain several continuous and categorical latent variables, mixed item types, repeated measurements, multilevel structures, testlets, response times, or latent classes of response processes.

IRT is best understood as a family of measurement models. The Rasch model is one especially important member of that family.

8.7.2 Persons and Items on a Common Scale

For a binary item, the Rasch model is

\[ P(X_{pi}=1\mid\theta_p,b_i) = \frac{\exp(\theta_p-b_i)} {1+\exp(\theta_p-b_i)}, \]

where:

  • \(X_{pi}=1\) means that person \(p\) solved item \(i\);
  • \(\theta_p\) is the person's latent competence;
  • \(b_i\) is the item's difficulty.

The same relationship can be written on the log-odds scale:

\[ \operatorname{logit}\{P(X_{pi}=1)\} = \theta_p-b_i. \]

This simple difference has an unusually clear interpretation:

  • if \(\theta_p=b_i\), the predicted probability is \(.50\);
  • if the person is one logit above the item, it is about \(.73\);
  • if the person is one logit below the item, it is about \(.27\).

A higher value means more competence for persons and more difficulty for items. Persons and items can therefore be placed on one common latent scale.

theta <- seq(-4, 4, length.out = 200)
difficulty <- c(-1.5, 0, 1.5)
probability <- sapply(difficulty, function(b) plogis(theta - b))

matplot(
  theta,
  probability,
  type = "l",
  lty = 1,
  lwd = 2,
  xlab = expression("Person competence " * theta),
  ylab = "Probability of a correct response",
  ylim = c(0, 1)
)
abline(h = 0.5, lty = 3)
legend(
  "topleft",
  legend = paste("difficulty =", difficulty),
  lty = 1,
  lwd = 2,
  bty = "n"
)
Three Rasch item characteristic curves. The item is solved with probability .50 where person competence equals item difficulty.

Figure 8.8: Three Rasch item characteristic curves. The item is solved with probability .50 where person competence equals item difficulty.

8.7.3 Why Begin with Rasch?

The Rasch model is attractive because it is small, interpretable, and demanding.

Its defining restrictions are:

  1. every item has its own difficulty \(b_i\);
  2. all items discriminate equally strongly;
  3. there is no item-specific guessing parameter;
  4. after conditioning on competence, responses are locally independent.

These restrictions produce several useful consequences.

8.7.3.1 One additive comparison

The response probability depends only on \(\theta_p-b_i\). A difference of one logit has the same meaning throughout the scale.

8.7.3.2 A raw score can be sufficient

For a complete response vector to the same set of binary Rasch items, the number correct is a sufficient statistic for the person parameter. Two pupils with the same raw score receive the same Rasch point estimate, even if they solved different items.

This result no longer holds so simply when pupils receive different item sets, responses are missing, items receive different discrimination parameters, or the model includes further dimensions.

8.7.3.3 Item and person comparisons can be separated

Under model fit, comparisons among persons do not depend on the particular items used, and comparisons among items do not depend on the particular persons sampled, apart from the uncertainty introduced by finite data. This property is often called specific objectivity.

8.7.3.4 The model can be tested rather than merely fitted

Rasch measurement treats equal discrimination and invariance as substantive requirements. If an item does not behave accordingly, the first question is whether the item, scoring, dimensionality, or population violates the intended measurement model—not merely whether another model has a better information criterion.

This is also the limitation. The Rasch model may be too restrictive. A two-parameter model may describe the data better when some items distinguish much more sharply between lower- and higher-competence pupils than others.

8.7.4 IRT Is a Family of Models

Different item formats and response processes require different response functions.

Situation Common model What varies by item?
Correct / incorrect Rasch model difficulty
Correct / incorrect 1PL model difficulty; one common discrimination may be estimated
Correct / incorrect 2PL model difficulty and discrimination
Multiple choice with lower asymptote 3PL model difficulty, discrimination, guessing
Ordered partial-credit categories partial credit model step difficulties
Same rating categories across items rating scale model item location, with shared category-step structure
Ordered categories with varying discrimination generalized partial credit or graded response model thresholds and discrimination
Unordered response categories nominal response model category-specific response parameters
Several competencies multidimensional IRT item relations to several latent traits
Items nested in common texts or stimuli testlet model general competence plus local testlet effects

The labels Rasch and 1PL are sometimes used interchangeably. Conceptually, both impose equal item discrimination. Software may distinguish them through scale identification: for example, mirt fixes Rasch slopes to one and estimates the latent variance, whereas its 1PL option handles the scale differently. After an appropriate rescaling, the substantive one-parameter response model is the same.

8.7.5 Models within the Rasch Family

The binary Rasch model is only the beginning.

8.7.5.1 Dichotomous Rasch model

Each item is scored \(0\) or \(1\). The item has one difficulty parameter.

8.7.5.2 Partial credit model

An item can award ordered scores such as \(0,1,2\). Each transition between adjacent score categories has its own step difficulty. The item may therefore distinguish between no credit, partial credit, and full credit without abandoning Rasch-style equal discrimination.

8.7.5.3 Rating scale model

This is a more restrictive model for repeated rating categories. It assumes that the category-step structure is shared across items, while items differ in overall location.

8.7.5.4 Many-facet Rasch model

Additional facets can enter additively, such as task difficulty, rater severity, occasion, or response mode. This is useful when performance depends not only on a person and an item but also on who scored the response.

8.7.5.5 Multidimensional and explanatory Rasch models

Several latent competencies can be modelled, or item difficulty can be explained by observable item features. The linear logistic test model, for example, decomposes item difficulty into contributions from item characteristics.

These extensions preserve parts of the Rasch logic, but each adds assumptions that need substantive justification.

8.7.6 Three Choices That Are Often Mixed Together

A large-scale assessment involves at least three distinct decisions.

8.7.6.1 1. The response model

This determines how item responses depend on person and item characteristics: Rasch, 2PL, partial credit, graded response, multidimensional IRT, and so forth.

8.7.6.2 2. Estimation of model parameters

This determines how item parameters and population parameters are calibrated.

Method Basic idea Typical role
Conditional maximum likelihood Conditions on the Rasch raw score so that person parameters disappear from item calibration Classical Rasch estimation
Marginal maximum likelihood Integrates over a population distribution of competence Large-scale assessment and modern IRT software
Joint maximum likelihood Estimates person and item parameters together Historically important; can be biased with short tests
Bayesian estimation Combines likelihoods with prior distributions and produces posterior distributions Complex and hierarchical IRT models

8.7.6.3 3. Person scoring after calibration

This determines how the fitted model is used to summarize the uncertain competence of a person. EAP, MAP, WLE, and plausible values are alternatives at this third stage. They are not different Rasch models.

8.7.7 From a Response Pattern to a Person Distribution

After the item model has been calibrated, a pupil's responses produce a likelihood for \(\theta_p\):

\[ L(\theta_p\mid\mathbf x_p,\widehat{\boldsymbol b}) = \prod_i P(X_{pi}=x_{pi}\mid\theta_p,\widehat b_i). \]

Combining this likelihood with a population distribution gives the posterior distribution:

\[ p(\theta_p\mid\mathbf x_p) \propto L(\theta_p\mid\mathbf x_p)\,p(\theta_p). \]

The important object is this whole distribution. A person score is a summary of it—or, in the case of a plausible value, a draw from it.

8.7.8 Person-Score Options

Result Definition Uses a prior? Main strength Main caution
ML value maximizing the response likelihood no based only on the response pattern and calibrated items undefined at all-wrong or all-correct extremes in the basic model
WLE bias-adjusted likelihood estimate no population prior reduces first-order bias of ML and is often useful for individual point scoring still a point estimate; uncertainty must be reported separately
MAP mode of the posterior distribution yes finite and stable, including near the extremes shrinks estimates toward the prior distribution
EAP mean of the posterior distribution yes minimizes posterior expected squared error and is easy to accompany with a posterior SD also shrinks; depends on the chosen population model
Plausible value random draw from the posterior distribution yes carries uncertainty into population analyses and regressions not an individual diagnostic score and not meant to be averaged into one “best” score

8.7.8.1 EAP: the centre of gravity

The expected-a-posteriori estimate is

\[ \widehat\theta_p^{EAP} = E(\theta_p\mid\mathbf x_p). \]

It is the posterior mean. If the posterior is symmetric, EAP and MAP can be almost identical. If it is skewed, they differ.

EAP estimates are stable for short tests and extreme response patterns because the population distribution contributes information. The price is shrinkage: highly unusual estimates are pulled toward the population mean, especially when the pupil answered few items.

8.7.8.2 MAP: the highest point

The maximum-a-posteriori estimate is

\[ \widehat\theta_p^{MAP} = \operatorname*{arg\,max}_{\theta_p} p(\theta_p\mid\mathbf x_p). \]

It is the mode of the posterior distribution. MAP can be easier to compute in high-dimensional models than EAP, because it finds a maximum rather than integrating over a large grid.

8.7.8.3 WLE: a likelihood score with bias correction

The weighted-likelihood estimate modifies maximum likelihood to reduce its small-sample bias. It does not use a population prior in the EAP/MAP sense and therefore usually shrinks less toward the population mean.

WLE is attractive when a single person-level point estimate is required. It should still be accompanied by a conditional standard error, because measurement precision differs across the competence scale.

8.7.8.4 Plausible values: several completed versions of the latent variable

A plausible value is not the posterior mean or mode. It is a random draw:

\[ \theta_p^{(m)} \sim p(\theta_p\mid\mathbf x_p,\mathbf z_p), \]

where \(\mathbf z_p\) may contain background variables used in a latent regression model.

Several plausible values create several completed datasets. A population analysis is run once with each plausible value and the results are combined. This is closely related to multiple imputation.

Plausible values are particularly valuable when the aim is to estimate population means, group differences, correlations, or regression coefficients. They are deliberately unsuitable for ranking individual pupils or reporting one supposedly exact competence value.

EAP, MAP, and WLE answer: “Which single value should summarize this pupil's uncertain competence?” Plausible values answer: “How can later population analyses retain that uncertainty rather than pretending the latent variable was observed?”

8.7.9 Why the Methods Give Different Scores

The differences become largest when:

  • the test is short;
  • many items are missing;
  • the response pattern is all correct or all incorrect;
  • the pupil is far from the centre of the item difficulties;
  • the posterior distribution is skewed;
  • background variables strongly predict the latent trait;
  • the item model is multidimensional.

With a long, well-targeted test, the likelihood is concentrated and ML, WLE, MAP, and EAP will often be close. With little information, the scoring rule matters much more.

8.7.10 VERA as an IRT Application

VERA stands for VERgleichsArbeiten. The assessments are administered in Years 3 and 8 and are intended primarily to support school and classroom development. They relate pupils' performance to the national educational standards and their competence-level models. The results are not intended as school rankings and are not graded.

The IQB describes competence as a continuous scale that is divided into a small number of substantively interpreted levels. Item difficulties and pupil competence are located on the same metric; the levels make that otherwise abstract scale interpretable for teaching.

VERA is therefore a particularly clear application of the basic IRT sequence:

\[ \text{educational standards} \longrightarrow \text{items} \longrightarrow \text{IRT scale} \longrightarrow \text{competence levels} \longrightarrow \text{feedback for instruction}. \]

The IQB also notes that a statistical competence value can be returned for an individual pupil when the difficulties of the administered items are known. At the same time, the individual result needs additional diagnostic evidence—such as learning progress and classroom observations—to support conclusions about one child.

A Rasch model is a natural starting point for VERA-style scoring because many responses are coded as correct or incorrect, items are calibrated on a common scale, and comparability is a central goal. It should not be assumed automatically, however. Dimensionality, local dependence, item fit, differential item functioning, and the consequences of different test booklets must be examined.

Research using VERA data has compared Rasch, 1PL, 2PL, and multidimensional models. This is useful evidence that “VERA uses IRT” does not settle every modelling choice. The appropriate model remains an empirical and substantive question.

VERA and PISA also have different reporting purposes. PISA plausible values are designed for population estimates and secondary analyses. VERA feedback may include an individual competence estimate and a competence-level classification for instructional use. The public IQB materials reviewed for this section do not specify one universal EAP, MAP, or WLE scoring rule for every Land and every assessment cycle. It would therefore be unsafe to label all VERA person scores as one particular estimator without the technical documentation for the relevant implementation.

8.7.11 Are VERA Items and IQB Data Public?

The answer depends on what is meant by the data.

8.7.11.1 Published example tasks: yes

The IQB task browser contains many VERA examples from earlier cycles together with solutions and didactic commentaries. They can be filtered by subject, competence domain, level, and other task characteristics. Many IQB-produced tasks are released under a Creative Commons licence.

These materials are excellent for understanding task design and competence levels, but they are not an item-response dataset. They do not provide one row per pupil and one scored response per item.

8.7.11.2 The operational item pool: not fully open

VERA uses newly developed tasks and may reuse tasks from earlier assessments. It follows that the current operational pool cannot simply be treated as a completely open item bank. Released examples represent the public part of a larger testing system.

8.7.11.3 IQB research data: available by application

The Research Data Centre at the IQB provides anonymized Scientific Use Files and Campus Use Files for non-commercial research and teaching. Access is generally free but requires an application and agreement to the conditions of use. Documentation, empty data structures, scaling manuals, and study reports may be available without a data application.

I did not find a nationwide VERA pupil-by-item response file offered as an unrestricted direct download. The most relevant German alternative is therefore an IQB education-monitoring dataset obtained through the FDZ, such as an IQB National Assessment or Trend in Student Achievement dataset.

8.7.11.4 Directly downloadable international alternatives

PISA provides public-use files, including cognitive item-response files and scored cognitive item-response files. These are close to the full large-scale-assessment workflow, but they are too large and complex for a first five-line Rasch example.

TIMSS item responses included in R packages provide a smaller bridge. The following example uses real Grade 4 mathematics responses distributed with the sirt package. It resembles VERA in age group, educational setting, binary items, missing-by-design responses, and a common competence scale, while remaining small enough for an introductory analysis.

Source Item texts Pupil response matrix Immediate access Best use here
IQB VERA task browser yes, selected past tasks no yes understand items and competence levels
IQB FDZ Scientific Use File depends on study and restrictions yes application required German secondary analysis
OECD PISA public-use files coded item information yes yes full large-scale-assessment analysis
sirt::data.timss item identifiers and metadata yes yes, with R package minimal Rasch demonstration

8.7.12 A Minimal Rasch Application in R

This example has four aims:

  1. load a small public educational item-response dataset;
  2. estimate a Rasch model;
  3. inspect item difficulties;
  4. compare EAP, MAP, WLE, and plausible values for the same pupils.

The package chunks are initially marked eval=FALSE so that the current book build does not depend on two new packages before they have been installed and tested locally. After installing mirt and sirt, remove eval=FALSE from the chunks.

8.7.12.1 Load Grade 4 TIMSS item responses

Rows are pupils, columns are items, 1 is correct, 0 is incorrect, and NA indicates that an item was not observed. In a booklet design, NA does not automatically mean that a pupil failed the item.

8.7.12.2 Inspect solution rates before fitting IRT

The proportion correct is an observed descriptive statistic. Rasch difficulty is a model parameter on the logit scale. The two will be strongly related, but they are not literally the same quantity—especially with different booklets or non-random missingness.

8.7.12.3 Estimate the Rasch model

model = 1 specifies one latent competence dimension. itemtype = "Rasch" fixes the item slopes to one. The item difficulties are estimated while the latent distribution supplies the scale.

8.7.12.4 Extract item difficulties

A lower \(b_i\) indicates an easier item. An item with \(b_i=1\) is predicted to be solved with probability \(.50\) by a pupil whose competence is \(\theta=1\).

8.7.12.5 Draw item curves and test information

In a Rasch model, all curves have the same shape and differ only in horizontal location. The test-information curve shows where the collection of items provides the greatest measurement precision.

8.7.12.6 Compute EAP, MAP, and WLE scores

The three point estimates should generally move together. Differences are expected near the extremes and for pupils with relatively little observed item information.

8.7.12.7 Draw plausible values

Each row now contains five different values because the latent competence remains uncertain. They should not be averaged to create a supposedly better pupil score.

For a population regression, the intended workflow is instead:

  1. fit the regression with PV1;
  2. repeat it with PV2 through PV5;
  3. pool coefficients and uncertainty using multiple-imputation rules.

For a genuine PISA-style analysis, plausible values should be generated from a latent regression model containing relevant background variables and analysed together with the survey weights and replicate weights. The simple draws above demonstrate the logic but do not reproduce the complete PISA production procedure.

8.7.13 What You Need for Your Own VERA-Style Analysis

Item difficulties alone are not sufficient to estimate pupil competence. At minimum, you need:

  • one row per pupil;
  • one column per item or sub-item;
  • scored responses such as 0, 1, and possibly ordered partial-credit categories;
  • NA for genuinely unobserved items;
  • stable item identifiers matching the difficulty table;
  • information about test booklets or modules;
  • ideally, item metadata and relevant grouping variables for fit and DIF checks.

A minimal data layout is:

pupil_id item_01 item_02 item_03 item_04
1001 1 0 1 NA
1002 1 1 1 0
1003 0 0 NA 0

With raw responses available, the same code pattern becomes:

If the item difficulties are already calibrated on the official scale, they should normally be fixed as anchor parameters rather than re-estimated from one class. That is a different task from the introductory calibration above and requires careful attention to the parameterization and scale transformation used by the original calibration software.

8.7.14 What Must Be Checked before Scores Are Trusted?

A fitted model is not automatically a valid measurement instrument.

8.7.14.1 Dimensionality

Do the items mainly measure one competence, or are reading, listening, vocabulary, and language form empirically distinct? A single total score can conceal meaningful multidimensionality.

8.7.14.2 Local independence

After controlling for competence, item responses should not remain strongly related. Several questions based on the same text may form a testlet and violate this assumption.

8.7.14.3 Item fit

Do observed response patterns follow the item characteristic curves implied by the model? Misfitting items may contain ambiguous wording, unexpected strategies, scoring problems, or multidimensional demands.

8.7.14.4 Differential item functioning

Does an item have the same difficulty for relevant groups after controlling for competence? Potential groups include gender, language background, school type, test mode, and federal state.

8.7.14.5 Targeting and information

Are item difficulties spread across the range occupied by the pupils? A test containing mostly easy items measures high competence poorly, even if it has many items.

8.7.14.6 Missingness and booklet design

Not administered is not the same as incorrect. Missing-by-design responses can be handled by IRT, but item linking and common anchor items are then essential.

8.7.14.7 Consequences of reporting

A competence estimate and its standard error are different from a competence level. Cutting a continuous uncertain estimate into levels adds classification uncertainty, especially for pupils near a threshold.

8.7.15 The Main Connections

The conceptual chain is now complete:

\[ \text{categorical indicators} \longrightarrow \text{latent response model} \longrightarrow \text{item calibration} \longrightarrow \text{person posterior} \longrightarrow \begin{cases} \text{EAP, MAP, or WLE point estimate},\\ \text{plausible values for population analysis}. \end{cases} \]

Rasch is valuable because it makes this chain especially transparent. It is not the only IRT model, and IRT is not the only method in the categorical-indicator/continuous-latent-variable cell. But Rasch provides the clearest starting point for understanding how an assessment can place item difficulty and person competence on one scale—and why the resulting person value always remains an estimate rather than an observed fact.