Understanding Test Reliability Calculation

Test reliability is a crucial concept in educational and psychological measurement. It refers to the consistency of a test score over time or across different forms of the test. A reliable test provides stable and dependable results. One common method for estimating the internal consistency reliability of a test with dichotomous items (like multiple-choice questions scored as right or wrong) is the Kuder-Richardson Formula 20 (KR-20).

The question provides us with the following information about a multiple choice test:

• Number of test items ($n$): 50
• Average difficulty of items ($\bar{p}$): 0.48
• Test variance ($S_t^2$): 81

We need to find the reliability coefficient of the test.

Applying the Kuder-Richardson Formula 20 (KR-20)

The Kuder-Richardson Formula 20 (KR-20) is given by:

$$r_{KR-20} = \left( \frac{n}{n-1} \right) \left( 1 - \frac{\sum pq}{S_t^2} \right)$$

Where:

• $n$ is the number of items in the test.
• $S_t^2$ is the total variance of the test scores.
• $\sum pq$ is the sum of the variances of each item. For an individual item $i$, its variance is $p_i q_i$, where $p_i$ is the difficulty of item $i$ (proportion of students getting the item correct) and $q_i = 1 - p_i$.

Calculating the Sum of Item Variances ($\sum pq$)

The problem states that the difficulty values vary in a "narrow range with an average value of 0.48". While we don't have the individual item difficulties ($p_i$) to calculate the exact $\sum p_i q_i$, a common approximation when only the average difficulty ($\bar{p}$) is provided is to assume that the average of the item variances is approximately the variance of an item with the average difficulty. Thus, $\sum p_i q_i \approx n \times \bar{p}(1 - \bar{p})$.

Using this approximation:

• Average difficulty ($\bar{p}$) = 0.48
• $1 - \bar{p} = 1 - 0.48 = 0.52$
• Estimated average item variance $\approx \bar{p}(1 - \bar{p}) = 0.48 \times 0.52 = 0.2496$
• Estimated sum of item variances ($\sum pq$) $\approx n \times \bar{p}(1 - \bar{p}) = 50 \times 0.2496 = 12.48$

Calculating the Reliability Coefficient

Now we can plug the values into the KR-20 formula:

• $n = 50$
• $n-1 = 49$
• $S_t^2 = 81$
• Estimated $\sum pq = 12.48$

$$r_{KR-20} \approx \left( \frac{50}{49} \right) \left( 1 - \frac{12.48}{81} \right)$$

First, calculate the fraction term:

$$\frac{50}{49} \approx 1.0204$$

Next, calculate the term inside the parenthesis:

$$\frac{12.48}{81} \approx 0.15407$$

$$1 - \frac{12.48}{81} \approx 1 - 0.15407 = 0.84593$$

Finally, multiply the two parts:

$$r_{KR-20} \approx 1.0204 \times 0.84593 \approx 0.8631$$

Comparing with Options

The calculated reliability coefficient is approximately 0.8631. Let's compare this value with the given options:

The calculated value 0.8631 is closest to 0.86.

Conclusion on Test Reliability

Based on the Kuder-Richardson Formula 20 and the provided test statistics (number of items, average difficulty, and test variance), the estimated reliability coefficient of the test is approximately 0.86. A reliability coefficient of 0.86 indicates a reasonably high level of internal consistency for the test scores, suggesting that the items are measuring a similar construct and the scores are quite consistent.

Revision Table: Key Concepts

Additional Information: Understanding Item and Test Variance

In test theory, understanding variance is key to calculating reliability. The total variance of a test ($S_t^2$) reflects the overall spread of scores among test-takers. It's influenced by both the true differences in ability (true variance) and random errors (error variance).

For dichotomous items, the item variance ($p_i q_i$) is highest when item difficulty ($p_i$) is 0.50 ($0.50 \times 0.50 = 0.25$) and decreases as difficulty approaches 0 or 1. Items with difficulty near 0.50 contribute more to the variance of the total test score (if they discriminate well) than very easy or very hard items. The sum of item variances ($\sum pq$) represents the variance attributed to the individual items, assuming their scores are independent.

The KR-20 formula essentially compares the sum of item variances ($\sum pq$) to the total test variance ($S_t^2$). If the sum of item variances is much smaller than the total test variance, it implies that the items are related to each other and contribute to a larger, common variance (the true score variance), leading to higher reliability. Conversely, if the sum of item variances is large relative to the total test variance, it suggests more random error or that items are not measuring the same thing consistently, resulting in lower reliability.

The term $\left( \frac{n}{n-1} \right)$ is a correction factor applied in the formula.