Mukaria J. J. Itang’ata

Doctoral Dissertation Announcement

---

Candidate: Mukaria J. J. Itang’ata

Degree of: Doctor of Philosophy

Department: Evaluation, Measurement, and Research

Title: A Comparative Study of Exact Versus Propensity Matching Techniques Using Monte Carlo Simulation

Committee:
Dr. Brooks Applegate, Chair
Dr. Joseph Kretovics
Dr. Warren E. Lacefield

Date: Wednesday, December 5, 2012 10:00 a.m. to noon
2401 Sangren Hall

Abstract:
Often researchers face situations where comparative studies between two or more programs are necessary to make causal inferences for informed policy decision-making. Furthermore, experimental designs employing randomization provide the strongest evidence for causal inferences. However, many pragmatic and ethical challenges may preclude the use of randomized designs.  In such situations, subject matching provides an alternative design approach for conducting causal inference studies. This study examines various design conditions hypothesized to affect matching procedures’ bias recovery ability.
The study examines three common social science research scenarios for case matching where discrete, continuous, or both types of covariates are employed. For each scenario, the following hypothesized factors are experimentally arranged in a factorial design: (a) Bias amount, (b) effect size (ES), (c) covariance among the covariates (CV), and (d) correlation (CR) between the bias amount and the covariate group, as the between group factors and six matching methods (MM) (random sampling, exact matching, propensity score matching, nearest neighbor matching, radius matching, and Mahalanobis metric matching) as the within group factor. Study conditions are investigated using Monte Carlo techniques. One thousand replicates are drawn from a theoretically defined population. Each replicate includes a treatment sample of N=200 and a control population of N=10,000 subjects. From the control population, random samples of N=200 are drawn via each matching method to form comparison group samples with known bias and effect size amounts added.
Results reveal that in the discrete covariate scenario there are significant group effects for Bias and ES and a MM*Bias interaction. In the continuous covariate scenarios there is a significant 4-way interaction involving MM* ES*CV*CR, while in the mixed scenario there is a 4-way interaction among Bias*ES*CV*CR and a significant main effect for MM.

These findings suggest that differences exist among the six MM’s ability to recover experimentally induced bias. Moreover differences are noted in matching success rates for all but the random sampling and nearest neighbor methods. Study implications suggest that social science researchers need to carefully consider multiple factors when employing a propensity-based or exact matching procedure in quasi-experimental designs.  Recommendations for further research are offered.