The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.
Volume VII, Number 2, Spring 2001
Issue Topic: Out-of-School Time Issue #2
Evaluations to Watch
CORAL: A Multi-Site Initiative to Improve Academic Achievement and Build Community Capacity
Kathleen Hebbeler of SRI International describes the evaluation of CORAL, which seeks to help communities view academic achievement as the shared responsibility of multiple sectors of the community.
The Initiative
Communities Organizing Resources to Advance Learning (CORAL) is a 10-year initiative to improve academic achievement through out-of-school activities. Funded by the James Irvine Foundation (the Foundation), CORAL is designed to build the capacity of communities to design and provide high quality programs. CORAL seeks to help communities view academic achievement as the shared responsibility of multiple sectors of the community, including business, media, public and private agencies, higher education, and the faith community. SRI International, a private research firm, is working with the Foundation to design and conduct the evaluation of CORAL.
Five communities in California are participating in the initiative. These communities receive financial and technical assistance from the Foundation for a pre-planning and planning phase. Funding is given to a lead agency to oversee the planning process, which involves broad representation across the community. Implementation grants that support CORAL activities for five to seven years are made following a successful planning period. The target age groups for CORAL programs are kindergarten through grade 8, although communities are also expected to provide some programming for high school-aged youth. The communities are being phased in over several years, with the first implementation in fall of 2000.
The Evaluation
The evaluation of CORAL is based on a theory of change that was developed over a six-month period through discussions with Foundation staff and consultants, including the evaluation team. The theory of change identifies improved academic achievement as the long term outcome of CORAL. The theory also identifies the coordination of providing agencies and the broad support of communities for academic achievement, out-of-school activities, and CORAL as two critical mechanisms leading to the provision of high quality out-of-school programs. Interim outcomes include children and families having knowledge and access to CORAL programs and children and families using the programs.
The evaluation will address research questions in five areas: implementation, utilization, child and family outcomes, contextual factors, and sustainability. The design includes both cross-community and community-specific components, with data for each community analyzed separately and then analyzed across the communities to portray the overall initiative. Communities also have the opportunity to adapt or individualize the evaluation design to include issues unique to their community.
In the evaluation, SRI will use a variety of data collection methods including both quantitative and qualitative measures. In order to understand issues that affect implementation, evaluators will interview participants of the governing coalition, program directors, and school administrators, conduct staff surveys, and review written documentation. An important source of implementation data will be structured on-site observations and curriculum reviews. This observation and review process will provide important feedback to the program and data for the evaluation. It will be conducted by a team of representatives including, for example, staff from the program being reviewed, another program, the lead agency and the evaluation team.
One of the major data collection sources for information related to utilization will be the CORAL management information system (MIS). An MIS is being developed to assist CORAL communities to enter and store information about enrollment and participation in CORAL activities. Several options, such as a “card swipe” system, are being explored as possible ways to track participation. The CORAL MIS will produce a variety of reports for the program and the lead agency and it will provide a wealth of data for the evaluation.
The measurement of academic achievement was one of the most critical and challenging facets of the evaluation design. Much discussion during the development of the theory of change and the evaluation focused on what is meant by academic achievement in the context of CORAL. Is it having the knowledge and skills to succeed in school? Is it actually succeeding in school? What is success in a school setting? Since CORAL programs rely on project-based learning and innovative uses of technology, it is conceivable that children could succeed in the out of school setting and yet still perform poorly in school. A clear consensus emerged from discussions among the Foundation and the communities that academic achievement should not only be defined as scores on standardized tests. Scores on tests being administered as part of the California educational accountability system will be included as one outcome measure, but so will grades, teacher ratings, program staff ratings, self-ratings by the child, parent ratings, promotion to next grade level, referrals for special services, school attendance, disciplinary actions, and possibly individually administered assessments of reading and mathematics. The ideal outcome is that children will see themselves and be seen by others as successful learners both in school and out-of-school. By measuring academic achievement with multiple measures from multiple perspectives, the evaluation will be able to examine how many children partially or completely achieve this ideal.
The voices of parents are also critical to the evaluation. Communities are expected to vary in the extent to which they develop programs targeted directly for parents (for example English language classes or parent support programs). The theory of change stipulates that CORAL will ultimately contribute to parents becoming better advocates for their children in a school setting. Parent surveys and focus groups will be an important source of information about how parents perceive the impact of CORAL on their child’s and family’s life. The family perspective on CORAL will also be explored in-depth through case studies of participating children in each community.
As mentioned, CORAL is intended to impact communities by strengthening the organizational capacity of participating organizations, including the lead agency. One aspect of organizational capacity building is the ability to use information to make decisions and to improve performance. The evaluation has been designed so it can provide information useful to both the communities and the Foundation. The evaluation team at SRI remains committed to infusing the community-based philosophy of CORAL throughout the evaluation design, data collection, and reporting processes. Exactly what this means in practice is being constructed each day as the evaluation moves from design to implementation.
For further information, including the timeline for reports, contact Kathleen Hebbeler at kathleen.hebbeler@sri.com or Elizabeth Iida at elizabeth.iida@sri.com.
Kathleen Hebbeler
Principal Investigator
SRI International
600 Mockingbird Place
Davis, CA 95616
530-758-7483