Impact Evaluation of a Multi-State Consortium Taaccct Grant: Challenges in Data Collection and Lessons Learned
*Names in bold indicate Presenter
The Ray Marshall Center (RMC) conducted the impact evaluation of the GCIT TAACCCT grant. We conducted an outcome analysis where we tracked and studied the education and employment outcomes of GCIT participants, including number of college credits earned, student persistence, program completion, credential achievement, employment placement, employment retention and wage gains. We also conducted a quasi-experimental impact analysis to estimate the impacts of the GCIT TAACCCT program on key education and employment outcomes. We used propensity core matching to construct a comparison group and address the key issue of the counterfactual: what would have happened absent the intervention?
In this paper, we share the circumstances and evolution of our impact evaluation. We discuss the challenges involved in collecting the academic and employment data needed for the evaluation. We focus on: the difficulties in setting up access to the data, due to the different governing bodies and administrative structures in Louisiana and Mississippi; the complex logistics of data transfer, due to multiple agencies (7+) involved in data collection; difficulties in ensuring accuracy and completeness of data, due to very different data systems in Louisiana and Mississippi; and, challenges in tracking participant outcomes, due to long lags in data availability and insufficient follow-up time. We note the evolution of our quasi-experimental design over the program period, in response to these realities of data availability and data collection. We also note how the challenges related to data collection limit the robustness of our impact findings. Finally, we report outcomes and impacts of the program. Lessons learned from our experience evaluating a large consortium grant spanning two states should help inform future approaches to measuring the impact of large-scale federal investments like the TAACCCT program.