Volume lll, Number 2, 1997

Issue Topic: School-Linked Services

Promising Practices

Bringing It Home: Supporting the Local Use of Data

Mary Wagner and Shari Golan of SRI International share information from their evaluation of California's Healthy Start School-Linked Services Initiative and the use of evaluation information at local sites.

The statewide evaluation of California's Healthy Start School-linked Services Initiative, funded by a consortium of California-based foundations, has compiled the most comprehensive database available on the processes and outcomes of school-linked services. Evaluators from SRI International drew on that database throughout the evaluation to inform local sites and the Healthy Start state partnership about the implementation of the initiative and the results for children, families, and schools. SRI's experience underscores both the value of sharing evaluation information regularly with programs and the difficulty of doing so in a timely and useful way.

Through Healthy Start, the California Department of Education (CDE) awards competitive grants of up to $400,000 to local education agencies that work in partnership with other public and private community organizations to develop new or expand existing efforts to provide comprehensive, integrated services linked to schools. The first 65 grantees, funded in 1992 and 1993, were involved in the evaluation, which was completed in 1996. The evaluation did the following:

• Described the sponsoring collaboratives and their operations
• Collected detailed information on the the services provided to children and their families
• Assessed outcomes of participating families in the areas of basic needs, health and health care, mental health, employment and financial independence, education, teen risk behaviors, and legal system involvement
• Assessed school-wide impacts regarding student mobility, attendance, school climate, parent involvement, disciplinary actions, and academic performance
• Linked characteristics of collaboratives and services to child, family, and school impacts in order to identify factors contributing to greater benefits
• Reported on the sustainability of local initiatives after their initial grants were expended and factors that supported sustainability

During the evaluation, SRI prepared several reports and briefs summarizing the results at the state level. SRI also shared local information and results with each grantee on a regular basis through the following:

Quarterly Reports

One mechanism for the dissemination of information was a quarterly report of service and client data. SRI developed a single-sheet, two-sided report form that recorded for the quarter and project-to-date the number of clients who had approached the local project for different kinds of services, the number of clients served and the numbers of units of different services that they received, the number of clients on waiting lists for different services, the number of referrals made for outside services, and the results of those referrals. In addition, the form reported the number of children and families served and demographic characteristics of those clients. A report for each local initiative was accompanied by a statewide aggregate compilation for comparison.

Many sites reported sharing their quarterly reports with their advisory boards to assess their accomplishments. These reports suggested program improvements and examined the amount and quality of data being submitted to SRI.

To support sites in their analysis of quarterly reports, SRI encouraged them to address the following questions: (1) Does your program reach the clients you intend to serve? (2) Do the types and frequencies of services delivered match your program's goals? (3) Is there a need for services that you currently are not able to provide? (4) To what extent are clients being referred outside for services, and to what extent are follow-ups being conducted? and (5) How often do referrals result in actual service delivery?

Even among sites that used the reports, however, data were sometimes perceived as late and not user-friendly. Some sites believed that the 3-month lag between the end of a quarter and receipt of a report—required by SRI to process the thousands of individual service records from 65 sites—made information too out-of-date to guide programs in a rapidly changing environment. In addition, the desire to keep the report to one page for easy dissemination to many interested parties created a form which was dense with data. This format was not practical for many people who were not used to relying on so much data.

Improvements could have been made in both timeliness and user-friendliness if the evaluation had included a component for helping sites develop the capacity to compile their own service- and client-related information. Assisting with such capacity-building activities for the 65 sites exceeded the scope of the evaluation tasks, however.

Evaluation Coaches

Use of the quarterly report at the local level was supported by local evaluation “coaches,” who helped sites adapt the statewide evaluation activities to local needs. Coaches also helped sites interpret and use the information that SRI reported back. The extent to which the coach emphasized and drew on the quarterly report in working with his/her sites was a key factor in whether staff found the information useful in the program improvement efforts.

Annual Surveys

The results of annual surveys of collaborative members and school staff were reported to sites for use as a self-assessment of their functioning. The surveys covered such topics as collaborative processes, levels of trust, support for Healthy Start, and integration of services into the school.

Reports on Outcomes

At the conclusion of the evaluation, the outcomes were reported for participating children and families for each site. Effective local initiatives used evidence of benefits to help marshal resources to sustain their efforts after the conclusion of their grants. These data were of keen interest to sites and were well received. The 3 years between the start-up of local initiatives and the site-specific answer to the key question, “Are we helping children and families?,” however, was a long wait.

Site Reports

In addition to this reporting to local sites, all information on individual sites was also shared with the state-level partnership, composed of members of several state agencies and the foundation consortium. From these site reports, the state partners obtained a clearer understanding of the wide variation in local programs under the Healthy Start umbrella. Site reports also were used to help identify local collaboratives that might benefit from technical assistance or other support from the state's Healthy Start Field Office. The state partnership also succeeded in securing budgetary support for expanding Healthy Start to more sites as early evaluation findings gave indications of benefits for children and families.

Post-Evaluation Guidebook

After the conclusion of Sri's evaluation, CDE decided to develop an evaluation that could be carried out by the local sites and still provide comparable data that could be aggregated by CDE. Local sites did not have the capacity to sustain the amount and level of information gathered through the statewide evaluation, however. SRI worked with CDE and an advisory panel of local site representatives to create evaluation requirements that would produce information that was useful to each local site and also met the California legislature's need for comparable statewide data. Everyone agreed to have all sites report on a limited number of data items and have each site choose from a menu of outcome areas one small set of data items that best fit their local goals and services. SRI wrote a guidebook that provided clear instructions on how all data items were to be measured and reported to ensure the collection of comparable and reliable data. Measures reflected commonly used practices whenever possible. In the first year of the guidebook's use, more than 120 Healthy Start grantees used it to capture baseline and follow-up data on outcomes at the school, family, and individual levels.

Extensive time and resources were required to provide quarterly site-specific data to 65 local collaboratives, but this investment was worthwhile. With access to site-level reports and coaches who could translate the findings of those reports, local staff and collaborative members had information on the strengths of their efforts and areas in which their programs might need improvement. This effort laid the foundation for a data-driven improvement process that continues today.

Mary Wagner
Program Director

Shari Golan
Senior Education Policy Analyst

SRI International
333 Ravenswood Avenue
Menlo Park, CA 94025-3493
Tel: 415-326-6200
Fax: 415-326-5512