The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.
Volume lll, Number 2, 1997
Issue Topic: School-Linked Services
Promising Practices
Using Evaluation Results: The Program Staff Perspective
Alfredo Tijerina of School of the Future explains the program staff perspective of using evaluation results.
Evaluations of school-linked service programs are frequently commissioned by outside agencies, often the very groups which support these programs' survival. While in many cases, the inclusion and cooperation of the program staff are crucial to the evaluation process, the information from the evaluation is not always fed back to these staff to inform program development. While the lessons which funders extract from evaluation data are important, the information provided in the evaluation process can be valuable for program staff as well.
The Houston School of the Future has had such an experience. Houston was one of four Texas sites selected by the Hogg Foundation for Mental Health's School of the Future initiative. Each site was provided a five-year grant (1990–1995) to establish a school-based services program. The grant provided for the services of an on-site project coordinator who was primarily responsible for project administration, service and community outreach, and public relations. The goal of the project was twofold: to enable selected schools to coordinate and implement school-based social and health services on their campuses and to evaluate the effectiveness of this method of service delivery. Today, 2 years beyond the end of the grant period, the project is continuing at each site.
The evaluation had both a quantitative and a qualitative component. The quantitative evaluation, the purpose of which was to determine the effectiveness and impact of the project on students and their schools, used nationally known instruments to collect student-level information on emotional well-being, self-esteem, and perceptions of school climate. These data were supplemented by school district data on academic achievement. The qualitative component showed the process of project development. Data for this portion was collected through two surveys and interviews and included information on project strengths, weaknesses, and barriers; perceptions of the project and the project coordinators; and opportunities for project continuation and potential incorporation into the educational system.
As project coordinator of the Houston School of the Future for more than 5 years, I have found that well-designed and implemented evaluations can offer information integral to supporting service programs, such as those that the Houston School of the Future provides. During one of the interviews with evaluators conducted early in the grant period, I stated that the schools would not abandon the project after the funding period ended. My prediction proved accurate, and the evaluation and its outcomes have played a part in generating the support necessary for the project's continuation in a number of ways:
-
Identifying key areas of concern. Each School of the Future site conducted a survey in which stakeholders identified key areas of concern. This was invaluable to us. The results of our survey supported our claim regarding the need for health services and led to the establishment of the health clinic, which has become the hallmark of the Houston school. Subsequent data collection efforts also assisted in the development of the project. For example, because a depression scale revealed a need to have a psychologist and a psychiatrist on staff, the health clinic now has these.
-
Putting proposals together. Since the Hogg Foundation grant funded only the coordinators and evaluation, we had to work hard to gain the support of others. The evaluation findings both confirmed what we knew early on and helped us to galvanize the support of others. For example, the evaluation of Compadres, a violence prevention program, helped us with our proposal for Ensueños, a substance-abuse program. We wove the findings into the narrative, which made our proposal stronger and resulted in our getting the grant.
-
Building partnerships. In order to encourage additional support from potential collaborative partners, we give them booklets describing the project and its services, its relationship with the community, and the lessons learned from its experiences. Because of the successes identified early in the project, we have been able to build on the schools' resources and strengths and on partnership strengths. It is important to recognize the resources in the community, not just money. These publications also are used by participants for planning project expansion. Furthermore, these publications have provided us with valuable contacts and media attention, which has led to inquiries and visits. It is important to be able to show that you have done something innovative.
Some aspects of the evaluation were not as useful to us. For example, although the stakeholder survey helped the project to bring in the clinic, as a result of either confusion of records or inconsistency in record keeping, data on outcomes for students who used this service have been only moderately useful. Quantitative findings have been problematic for other reasons as well. So many variables influence outcomes from school-based service programs that it is difficult to determine which interventions led to the outcomes obtained. From the program perspective, this is particularly troublesome. Additionally, the great amount of mobility within the population left us too small a sample to study longer-term program results effectively. It also would have been helpful had project staff members been able to contribute to the evaluation design. As the project brought in services, we could have identified some of the evaluation pieces that concerned us in order to obtain information that would be more useful to us. However, although the quantitative findings were problematic in several ways, they did enable us to demonstrate to stakeholders that there were unmet needs in the community, and this helped us to gain support for the program.
Documents describing the the Future initiative can be obtained by contacting the Hogg Foundation for Mental Health, University of Texas, Post Office Box 7998, Austin, TX 78713-7998. Tel: 512-471-5041.
Alfredo Tijerina
Coordinator
School of the Future
Houston, TX