Educational Leadership, Research and Technology

Evaluation of the Kalamazoo Promise

This is an outline of work that the College of Education and Human Development at Western Michigan University is conducting regarding the Kalamazoo Promise, a scholarship program funded by anonymous donors that was announced in November 2005. Since its announcement, there has been substantial interest in the potential effects of the universal scholarship. Because of this interest and because other communities are replicating or planning to replicate the program, an evaluation that examines the impact of the scholarship program is critical. The purpose of the evaluation is to determine whether the Kalamazoo Promise, through its transformative effect on the education culture within the school district, improves the progress of students through their K-12 experience and better prepares them for entering a postsecondary education program.

Some imagine that at least four to five years will be required to measure the anticipated long-term outcomes (i.e., increased access and success in post-secondary education). Our evaluation, however, maps out and tracks the impact of the program which can be measured with specific short-term and intermediate outcomes that are linked to the long-term outcomes. Aside from anticipated outcomes, our evaluation will also consider unanticipated outcomes stemming from the inception of the scholarship.

In addition to this, the W.E. Upjohn Institute for Employment Research plans to conduct student level analysis to examine the impact of the Kalamazoo Promise on student-level indicators.

Evaluation team

About the Kalamazoo Promise

The Kalamazoo Promise scholarship program was announced in November 2005 and provides four years of tuition and fees at any public college or university in Michigan for students who have attended Kalamazoo Public Schools. The Kalamazoo Promise is unique in its scope and basis. While most other scholarship programs are based on merit or need, the Kalamazoo Promise is based only on location. The program seeks to remove the financial barriers to enrolling in college for those students who have attended Kalamazoo schools and lived within its boundaries for at least the four years of high school.

The Kalamazoo Promise has received accolades from local and state government officials. The plan has also received wide media attention, with newspapers from around the nation covering the scholarship program. Each news report introduces new testimonials from parents and students directly affected by the Promise. Individuals tell heartwarming stories of how college was a long-hoped goal that finances rendered unattainable; now, the Kalamazoo Promise has changed their lives' course. Many other cities have replicated the scholarship, or have undertaken feasibility studies, indicating the growth of a movement toward universal K-16 education. Educators, politicians, journalists, and families have all praised the initiative's focus on education and the new opportunities that have been given to Kalamazoo's students. For more information about the Kalamazoo Promise, please visit the scholarship's website.

Community-school maps

Many relationships exist between the schools in the Kalamazoo Public School district and the Kalamazoo community. Often, partnerships are arranged through Kalamazoo Communities in Schools, a nonprofit organization that acts as a bridge between the schools and service providers, or even concerned citizens. In other cases, community organizations or individuals contact the school directly to begin a program. For these reasons, the number of community-school partnerships varies significantly from school to school. The partnerships listed are current for the 2007-08 school year.

Community-school maps

Updates

Logic model

A detailed description of the logic model and its outcomes is available in Working Paper #1.

Methodology

Data for the evaluation of the Kalamazoo Promise will be collected through five main activities:

Analysis of school-level available data.
Interviews with key informants.
Review of public and district documentation.
Survey of middle and high school students.
Survey of teachers and staff.

Survey of middle and high school students

The College of Education and Human Development will survey high school students on an annual basis to establish a baseline with which to compare the effects of the Kalamazoo Promise on future students. The purpose of the survey is to capture student aspirations, teacher expectations for students, and school climate, and to collect information from students regarding their plans following graduation and intentions to pursue post-secondary education. Students will also be asked to recall and report on changes in their schools since the announcement of the Kalamazoo Promise. Information also will be collected that will help develop and refine the scholarship program. Surveys in future years will include questions about perceived obstacles to using the Kalamazoo Promise. All students will be sampled. The survey will be anonymous and will not be linked to student-level information. It will include key background items so the results can be disaggregated and correlational and multivariate analyses can be conducted.

Survey of teachers and staff

The College of Education and Human Development will develop and administer an electronic survey for teachers and staff. The survey would collect information on perceived changes in the district, including changes in student aspirations, teacher expectations, responses from the Community and from the district itself. Information also will be collected that will help develop and refine the scholarship program. The survey will collect some data on general background items so that the results can be disaggregated and correlational and multivariate analyses can be conducted.

Interviews with key informants

A purposive sample of 60 to 70 key informants from the community and from the local school district will be identified for interviews. A semi-structured interview protocol will be developed to ensure that relevant and critical topics are covered. Key informants from the community are likely to be civic and local government officials, leaders of community organizations, faith-based groups, and neighborhood associations. Within the district, key informants will include district officials, school board members, principals and guidance counselors. Data from interviews will be entered and coded for thematic analysis and consistency of patterns within narrative data.

Review of public and district documentation

Program and school district documentation would be reviewed for the evaluation. Additionally, a review of media coverage would be conducted in order to help gain an understanding of the scope of the potential impact of the Kalamazoo Promise. Additionally, we will request relevant documentation in connection with the interviews of key informants. The documentation would be thematically coded and summarized. Archiving these public documents allows us to track growth over time within the district, as well as the evolution of community groups' response to the scholarship. It also serves as a source of data to triangulate our findings from the other data collection activities in this evaluation.

Analysis of school-level available data

The College of Education and Human Development would conduct a secondary analysis of school level data. This would track the performance of each of the district schools over the past three to five years relative to how demographically similar schools in Michigan perform. We have refined a rather sophisticated methodology called "residual gains analysis" that we would use. This type of analysis uses school level test and demographic data that we can obtain from the state department of education, so the district is not burdened with requests for data.

To estimate the effect of the scholarship, we developed a set of statistical filters that remove most of the changes in student composition in the Kalamazoo schools over time. The remaining portion of the score changes provides a reasonable estimate of program effectiveness. While calculating the filtered scores requires statistical techniques, the basic idea is relatively simple. The filters work by comparing each district school with a set of demographically and geographically similar non-Kalamazoo schools. Instead of focusing on absolute scores or values, the filtered scores focus on the differences between each Kalamazoo school and its specially selected comparison group created using regression analyses. Variables used in the filters will be obtained from state and federal databases and include family income, ethnicity, special education status, urbanicity code, and school enrollment.

In as much as the comparison schools will be similar to the district schools in most relevant respects save for not being a Promise school, the filtered (difference) scores will provide the best approximation of the program effect using group or school-level data. Another advantage of the filtered scores is that they have a straightforward interpretation.

Since the filtered scores represent the difference between a district school and its specially selected comparison group, a score of zero indicates that the school is performing exactly on par with its comparison group. Unlike most test metrics, filtered or residual scores can take negative values, indicating that a school's performance is below that of its comparison group. For example, a filtered score of -15 indicates that the average student in a Kalamazoo school scored 15 points lower than predicted or relative to the average student in the school's comparison group. By contrast, a filtered score of 25 indicates that the average student in a Kalamazoo school scored 25 points higher than the average student in the school's comparison group.

Changes in scores or indicators over time provide a better estimate of value added than a snapshot from a single point in time. Each Kalamazoo school gain in residual scores over time indicates that the average student score or school-level indicator is catching up with the average student or the average school in the school's comparison group. Similarly, declines in a school's residual scores suggest that it is falling behind the average school in the comparison group.