Association for Public Policy Analysis & Management

MDRC has been working with several CTE intermediaries (e.g., YouthForce NOLA and CareerWise Colorado) organized through the MDRC Center for Effective Career and Technical Education. The Center incubates new ideas, synthesizes findings and lessons learned from projects and disseminates this knowledge to policymakers, practitioners, and other researchers in order to ensure that expansion of CTE programs is informed by a growing evidence base. The programs being implemented by several of our intermediary partners include work-based learning, sequenced CTE courses leading to credentials or successful transitions into employment/postsecondary, and soft-skills training. This presentation examines (1) the efforts that several CTE intermediaries have undertaken to identify and adapt program metrics to understand whether access to program services is equitable, participants’ engagement in these services, and their progress through and completion of the program, and (2) the supportive role that a research and evaluation partner can play.

In these partnerships with CTE intermediaries, particularly those still early in their development, we have found that they are typically fully extended when they're trying to manage relationships between and link the work of high schools, colleges, and employers. This means that also learning systematically about the ongoing implementation of their programs in ways that they can funnel back into improvements of their program models can stretch them beyond capacity. These intermediaries often lack the data infrastructure needed to collect, store, and analyze information for the combined purpose of research and development. Thus, their research and development efforts are more likely to proceed in a piecemeal, disconnected way. For example, an organization might employ a management information system (MIS) to track some aspects of program implementation, such as participants’ engagement in internships or apprenticeships, but when the program team desires to analyze outcomes or use data to better target services or supports to specific individuals, they are often unprepared to integrate data from other systems and sometimes need to create an entirely new data system.

In recognition of these challenges, we have found that a research partner can be valuable in taking on investigations of program implementation as well as helping to sort through potential data sources and measures to support an intermediary's organization of information. The research partnership often starts with the creation of a logic model that clarifies links between measurable program components and desired outcomes for participants. This logic model provides the foundation for the development of a performance measurement framework. MDRC works with these CTE programs along the continuum of program improvement and evaluation to continue to build more rigorous evidence while strengthening program operations. In the early stages, our research has included implementation and descriptive outcomes while at the same time working alongside programs to help them think about what services they provide above and beyond what is available to its target population. This lays the groundwork work to eventually have their impact assessed as rigorously as is feasible, if and when that is appropriate, while building capacity for them to own much of this work increasingly over time.