Methodology

Data for the evaluation of the Kalamazoo Promise will be collected through five main activities:

  1. Survey of Middle and High School Students
  2. Survey of Teachers and Staff
  3. Interviews with Key Informants
  4. Review of Public and District Documentation
  5. Analysis of School-Level Available Data

Survey of Middle and High School Students

The College of Education will survey high school students on an annual basis to establish a baseline with which to compare the effects of the Kalamazoo Promise on future students. The purpose of the survey is to capture student aspirations, teacher expectations for students, and school climate, and to collect information from students regarding their plans following graduation, and intentions to pursue postsecondary education. Students will also be asked to recall and report on changes in their schools since the announcement of the Kalamazoo Promise. Information also will be collected that will help develop and refine the scholarship program. Surveys in future years will include questions about perceived obstacles to using the Kalamazoo Promise. All students will be sampled. The survey will be anonymous and will not be linked to student-level information. It will include key background items so the results can be disaggregated and correlational and multivariate analyses can be conducted.

Survey of Teachers and Staff

The College of Education will develop and administer an electronic survey for teachers and staff. The survey would collect information on perceived changes in the district, including changes in student aspirations, teacher expectations, responses from the Community and from the district itself. Information also will be collected that will help develop and refine the scholarship program. The survey will collect some data on general background items so that the results can be disaggregated and correlational and multivariate analyses can be conducted.

Interviews with Key Informants

A purposive sample of 60-70 key informants from the community and from the local school district will be identified for interviews. A semistructured interview protocol will be developed to ensure that relevant and critical topics are covered. Key informants from the community are likely to be civic and local government officials, leaders of community organizations, faith-based groups, and neighborhood associations. Within the district, key informants will include district officials, school board members, principals, and guidance counselors. Data from interviews will be entered and coded for thematic analysis and consistency of patterns within narrative data.

Review of Public and District Documentation

Program and school district documentation would be reviewed for the evaluation. Additionally, a review of media coverage would be conducted in order to help gain an understanding of the scope of the potential impact of the Kalamazoo Promise. Additionally, we will request relevant documentation in connection with the interviews of key informants. The documentation would be thematically coded and summarized. Archiving these public documents allows us to track growth over time within the district, as well as the evolution of community groups response to the scholarship. It also serves as a source of data to triangulate our findings from the other data collection activities in this evaluation.

Analysis of School-Level Available Data

The College of Education would conduct a secondary analysis of school level data. This would track the performance of each of the district schools over the past 3-5 years relative to how demographically similar schools in Michigan perform. We have refined a rather sophisticated methodology called "residual gains analysis" that we would use. This type of analysis uses school level test and demographic data that we can obtain from the state department of education, so the district is not burdened with requests for data.

To estimate the effect of the scholarship, we developed a set of statistical filters that remove most of the changes in student composition in the Kalamazoo schools over time. The remaining portion of the score changes provides a reasonable estimate of program effectiveness. While calculating the filtered scores requires statistical techniques, the basic idea is relatively simple. The filters work by comparing each district school with a set of demographically and geographically similar non-Kalamazoo schools. Instead of focusing on absolute scores or values, the filtered scores focus on the differences between each Kalamazoo school and its specially selected comparison group created using regression analyses. Variables used in the filters will be obtained from state and federal databases and include family income, ethnicity, special education status, urbanicity code, and school enrollment.

Inasmuch as the comparison schools will be similar to the district schools in most relevant respects save for not being a Promise school, the filtered (difference) scores will provide the best approximation of the program effect using group or school-level data. Another advantage of the filtered scores is that they have a straightforward interpretation.

Since the filtered scores represent the difference between a district school and its specially selected comparison group, a score of zero indicates that the school is performing exactly on par with its comparison group. Unlike most test metrics, filtered or residual scores can take negative values, indicating that a school's performance is below that of its comparison group. For example, a filtered score of -15 indicates that the average student in a Kalamazoo school scored 15 points lower than predicted or relative to the average student in the school's comparison group. By contrast, a filtered score of 25 indicates that the average student in a Kalamazoo school scored 25 points higher than the average student in the school's comparison group.

Changes in scores or indicators over time provide a better estimate of value added than a snapshot from a single point in time. Each Kalamazoo school gain in residual scores over time indicates that the average student score or school-level indicator is catching up with the average student or the average school in the school's comparison group. Similarly, declines in a school's residual scores suggest that it is falling behind the average school in the comparison group.

 

 

Kalamazoo Promise Evaluation
Western Michigan University
Kalamazoo MI 49008-5283 USA
(269) 387-3883 | (269) 387-3696 Fax
gary.miron@wmich.edu