Association for Public Policy Analysis & Management

To address large disparities in labor market participation rates, the US Department of Labor has committed resources to expand the capacity of the public American Job Center (AJC) system to serve people with disabilities (PWD) and ensure its accessibility to all PWD. IMPAQ conducted an evaluation of AJC accessibility, with the goal of developing a national accessibility score. Measurement challenges for our study included: a lack of existing measures of accessibility that could be validated; a need for new metrics for measuring participants’ experience of, ability to use, and equal access to, all phases of the service delivery process (rather than compliance, as most studies of accessibility do); and a need to investigate the magnitude of socially desirable responses, which are highly likely in a study of this type. The study we designed was comprehensive and highly complex, involving:

Designing and validating a measure of the complex and multidimensional construct of accessibility – Our mixed-methods approach involved developing and implementing a web-based survey of all AJCs in the country (2400+), supplemented with site visit interviews and architectural accessibility assessments, to provide a deeper understanding of the true accessibility of the AJCs and to validate the self-reported survey data. We will describe the challenges of implementing our survey, from addressing the limitations of relying on AJC self-report to gathering data across the universe of AJCs.

Designing tools and processes to manage large-scale qualitative data collection – IMPAQ’s AJC Accessibility study included visits by 20 researchers to 100 randomly selected job centers throughout the US over the course of seven months. The scale of this undertaking posed numerous challenges related to data collection, including: site selection, planning logistics, ensuring data consistency, and minimizing costs. We will discuss the tools and processes designed to manage these visits and describe promising practices and lessons learned.

Listening to customer voices – To effectively measure the usability of center services by PWDs and the extent to which PWDs have equal access to services, programs, and opportunities, it was essential to include the customer perspective, accomplished by conducting focus groups with center customers who have disabilities. In addition to discussing challenges in setting up and conducting the focus groups, we will offer insights from PWD on effective accessibility strategies and barriers to accessibility that were not captured in the center surveys or site visits.

Using Item Response Theory (IRT) to define levels of accessibility – We developed a method of using IRT to create a measurement model that statistically combined the results from the web survey and the site visits to develop domain and overall accessibility scores. It also included standard-setting procedures that utilized Subject Matter Experts to define levels of accessibility and develop rules to assign scores to accessibility levels. Our analysis included estimating any bias towards socially desirable responses. This paper will describe the challenges and possibilities regarding the use of IRT in social science research.