Panel Paper: Intra-Class Correlation and R2 Estimates for Commonly Used Outcome Measures in Research on High Schools and Postsecondary Institutions

Saturday, November 8, 2014 : 10:35 AM
Apache (Convention Center)

*Names in bold indicate Presenter

Fatih Unlu, Abt Associates, Lily Fesler, Abt Associates, Inc., Julie Edmunds, University of North Carolina at Greensboro and Elizabeth Glennie, RTI International, Inc.
One of the emerging fields in education research is concerned with providing estimates for district and school-level intraclass correlations (ICC) and how much of the variation at each of these levels is explained by the typically available covariates (i.e., district, school, and student-level R2 values) with the intention of informing the designs of future research studies (e.g., Bloom, Richburg-Hayes, and Black, 2005; Hedges and Hedberg, 2007; Schochet, 2008; Xu and Nichols, 2010; Hedges and Hedberg, 2014a and 2014b). One limitation of this line of research is that almost all of these studies exclusively utilize academic achievement as captured by test scores in standardized or state assessments and there is not sufficient evidence as to whether and to what extent the reported ICC and R2 values would be applicable to other outcome measures.

The proposed paper addresses this limitation and is concerned with the estimation of district, school, and student-level ICC and R2 values for outcome measures commonly used by researchers studying high school and postsecondary interventions and programs such as staying in high school, progressing in high school, high school graduation, enrollment and persistence in a postsecondary institution, grade point average (GPA) and credit accumulation through high school and postsecondary institutions, and college graduation. While these outcomes are frequently analyzed (e.g., three of the What Work Clearinghouse topic areas, College and Career Preparation, Dropout Prevention, and Postsecondary Education particularly examine these outcomes),  we are not aware of any studies that report design parameters for these outcomes. We aim to fill this important gap and report these parameters by analyzing the data we collected for the Longitudinal Evaluation of North Carolina’s Early College High Schools Initiative. This experimental study is following more than 4,000 students in North Carolina through high school and into postsecondary institutions. Data sources for the high school and college-level outcomes include the administrative databases maintained by North Carolina Department of Public Instruction and North Carolina Community College System as well as the National Student Clearinghouse. 

We will calculate ICC and R2 values for not only the full study sample but also for subgroups defined by school and student characteristics including higher vs. lower performing, underrepresented minority, and socioeconomic status. Finally, we also address a specific methodological question which is concerned with whether the unavailability “matching baseline/pretest” variables for some of these outcomes (e.g., high school graduation and college enrollment) lead to smaller R2 values than other outcomes with matching pretests (e.g., test scores, GPA). These analyses are underway and will be completed well before the conference.