Association for Public Policy Analysis & Management

Currently, 17 states require personal finance courses in secondary schools, and six require financial literacy testing (Council for Economic Education, 2014). Assessment of students’ financial knowledge is often used to evaluate the impact of mandated programs (Huston, 2010). Few financial literacy tests, however, have been examined for reliability. Additionally, most tests show a gender gap in performance, with females typically less financially literate than males (Lusardi & Mitchell, 2014). This discrepancy has been attributed to differences in confidence and interest (Chen & Volpe, 2002). However, the gap may result from failures of invariance in financial literacy assessments, meaning that the ability of males and females are measured differently. Apparent sex-bias in financial literacy testing may cause school districts to incur costly legal ramifications, as the fairness of required tests is called into question. To address possible failures of gender-related invariance, we tested a pilot high school financial literacy assessment against the Rasch model and performed a DIF analysis. This project sought to answer the following research questions: 1) How well does the assessment’s item difficulty align with the curricular difficulty? 2) How effective is the assessment at measuring students’ performance? 3) How do students’ item responses vary by gender?

The assessment examined in this study was the Kentucky High School Financial Literacy post-test, which was administered to 12th graders following a personal finance curriculum. The Rasch method was selected as it examines test quality by determining test unidimensionality, test item difficulty, student ability levels, and item fit (Bond & Fox, 2007). A differential item function (DIF) analysis was conducted as part of the Rasch as a means to examine differences in student responses by gender. DIF occurs when students of different groups do not have the same probability of answering a question correctly, despite having the same ability level.

Results of the Rasch analysis shows the assessment had high item reliability at .93 and person reliability of .79. There were item fit issues, with item-infit statistics ranging from -2.0 to 3.3 and item-outfit statistics ranging from -1.8 to 2.2. Two items had misfit values, suggesting that they did not measure ability properly and there may be an issue with unidimensionality (Bond & Fox, 2007). The item difficulty levels ranged from -2.01 to 2.50, and the items did not align with the curriculum’s expected difficulty. One item experienced DIF (“Which of the following is true about inflation?”), with males having a greater probability of correctly answering the question. Therefore, the quality of this assessment would be improved by removing this item.

This study contributes to the literature by being among the first to apply a Rasch DIF analysis to a financial literacy instrument in the US. The findings show that these methods can aid in investigating the gender gap in financial education. This study also provides policymakers with an example of how DIF analysis can uncover biased items in financial literacy tests. The removal of such unfair assessment questions will help safeguard mandates from accusations of gender discrimination.