Pew Research Center’s survey on international knowledge covers facts about global leaders, international institutions and geography, among other topics.
The following criteria are used to evaluate how well the 12 items can be used as a scale or index of international knowledge more broadly: 1) the degree to which responses are internally consistent, 2) the degree to which the questions reflect a single underlying latent dimension, and 3) the degree to which the scale discriminates between people with high and low knowledge, providing information about people with varying levels of knowledge about foreign affairs.
The internal reliability of the scale as measured by Cronbach’s alpha is 0.83. Cronbach’s alpha does not increase if any of the items are dropped.
An exploratory factor analysis finds that the first common factor explains 67% of the shared variance in the items. The second common factor explains only 15% of the common variance. The factor loadings show that each of the 12 items is at least moderately correlated with the first common factor. This suggests that the set of items is the result of a single underlying dimension.
Note that all the international knowledge items are coded as binary variables (either correct or incorrect). Both Cronbach’s alpha and factor analysis are based on a Pearson’s correlation matrix. Pearson’s correlations with binary variables are restricted to a limited range, underestimating the association between two variables. We do not anticipate the use of a Pearson’s correlation matrix will affect the unidimensional factor solution for the scale.
We conducted item response modeling for the scale to evaluate how well it discriminates between people at different levels of knowledge. The analysis fits a two-parameter logistic model, allowing discrimination and difficulty to vary across items. Discrimination shows the ability of the question to distinguish between those with higher and lower international knowledge. Difficulty shows how easy or hard each question is for the average respondent.
All the items have acceptable discrimination estimates. The difficulty parameter estimates are negative for seven of the 12 items, and positive for five of the items. This shows that the average respondent had a good chance of getting the correct answer on most of the items.
The test information function shows the amount of information the scale provides about people with different levels of knowledge of foreign affairs. The test function approximates a normal curve and is centered below zero (Theta) at about -0.25. This indicates that the scale provides the most information about those with slightly below-average international knowledge. The scale provides comparatively less information about those with high international knowledge, especially those with very high levels of knowledge. Notably, being able to distinguish among those highly knowledgeable about foreign and international topics was not an important goal of this study. Moreover, the inclusion of the very difficult questions needed to make distinctions among the very informed could potentially have negative effects on the morale of the rest of the people taking the quiz.
