Opening the Black Box

Wiscape Banner
shadow

CONTACTING US

Main Office

WISCAPE
School of Education
UW-Madison
353 Education Building
1000 Bascom Mall
MadisonWI  53706-1326

Tel: 608/265-6342

Email: wiscape-info@education.wisc.edu
or by contact form

 

Facebook logo Twitter logo

 

Opening the Black Box: Quasi-Experimental Method Selection for Postsecondary Education Research and Evaluation

Project Leads

Elizabeth Vaade 

Policy Analyst,WISCAPE
University of Wisconsin-Madison

Bo McCready

Project Assistant, WISCAPE
University of Wisconsin-Madison

Overview

The push for data-driven decision-making and accountability has made evaluation research critical to successful postsecondary education policy and practice. When experimental methods are not the best choice, researchers and evaluators can use quasi-experimental methods to quantitatively estimate program effects. Although the literature describes how to execute these methods, this work offers little information about

  1. the process researchers use to select a method,
  2. the tradeoffs inherent in method selection, and
  3. how policy actors interpret results derived from a particular technique.

Without this kind of information, rigorous research and evaluation can seem like a black box—inputs and outputs are visible, but the processes remain hidden and largely inaccessible.

This project explores what factors influence quasi-experimental method selection in quantitative research and evaluation. Through surveys, interviews, and database scans, the project team investigates the factors that help determine which methods are used and how researchers and other policy actors differ in their interpretations of these techniques. Approximately 20 individuals—including researchers, evaluators, state government works, policymakers, and practitioners— participated in the project to date. View descriptions of the methods discussed in this study here.

PURPOSE AND OUTCOMES

This work provides a starting point for critical conversations about research and evaluation in postsecondary education. A discussion of feasible and appropriate comparison methods contributes to continuous improvement of not only the programs undergoing evaluation but also related research. Further, revealing how policy actors view and interpret these methods and the results they produce helps facilitate greater understanding between researchers and evaluators who produce findings and methods standards and the policymakers and practitioners who use this research and perform their own assessments.  Finally, considering issues of rigor and accessibility in quasi-experimental methods will help evaluators and researchers prioritize and balance these goals, as well as think about what they mean in practice.

Related Papers and Presentations

  • Vaade, E.S. & McCready, B. (2011, November 16).  Opening the black box: The process of selecting non-experimental evaluation methods and the impact on postsecondary education programs and policy. Presented at the Council on Public Policy and Higher Education pre-conference forum at the annual meeting for the Association of the Study of Higher Education in Charlotte, NC. Download paper. Download presentation slides.

RELATED Media

© 2017 Board of Regents of the University of Wisconsin System • Please contact the School of Education External Relations Office with questions, issues or comments about this site.