The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.
Volume VI, Number 1, Fall 2000
Issue Topic: Out-of-School Time Issue #1
Promising Practices
Strengthening Local Evaluation Capacity in Rural Communities
Using a participatory/empowerment evaluation approach with Save the Children, Linda Morrell and Kenneth Terao from the Aguirre Group offer reflections and lessons learned from their experience.
In 1997, Save the Children U.S. Programs (SCF) implemented an innovative initiative, the Web of Support, to promote quality out-of-school time programs for children and youth in some of America’s most economically distressed communities. This initiative provided funding and technical assistance to grantees (called community partners) in rural and urban areas, emphasizing elements—constructive activities (including leadership development), caring adults and safe environments—that contribute to children’s learning and personal growth. To familiarize and ensure adoption of the Web of Support, Save the Children provided community partners with training and technical assistance in program development (how to improve out-of-school time programs for children) and evaluation (how to measure outcomes of activities and services provided to children).
The National Institute on Out-of-School Time (NIOST), a nationally recognized organization with expertise in strengthening out-of-school time programs, led training and technical assistance (T/TA) exercises to ensure programs’ continuous quality improvement. The Aguirre Group (Aguirre), an organization specializing in evaluation, provided T/TA in internal outcome evaluation and conducted a comprehensive evaluation of the Web of Support.
Evaluation Approach
Save the Children, in collaboration with Aguirre, chose a participatory/empowerment approach to evaluate the Web of Support because it promotes self-sufficiency within the organization and among associated community partners, allows for customized evaluations of a diverse selection of programs/activities, and supports efforts for continuous program improvement.
The participatory/empowerment approach also allows community partners to have ownership and involvement at all levels of the evaluation process. They are able to develop their own definitions of positive change and improvement based on the populations they serve and needs within the community, identifying the stakeholders and their roles in evaluation design and implementation. This approach also allows the contributions and viewpoints of individuals at all levels of the Web of Support to be acknowledged.
Train the Trainer
To support the development and institutionalization of internal evaluation, Aguirre trained Save the Children staff at the national office in Westport, Connecticut, and the three rural regional offices in Berea, Kentucky (Appalachian Region), Asheville, North Carolina (Eastern Region), and Albuquerque, New Mexico (Western Region), to be evaluation trainers for rural community partners. These trainers initially observed on-site assistance, regional trainings, and telephone assistance conducted by Aguirre to community partners. After they understood evaluation in theory and practice, the trainers began to play a larger role in conducting training and providing assistance. Over the 3 years of the initiative, Save the Children staff developed the skills to provide ongoing support to community partners, with Aguirre providing technical expertise and support when necessary. Training Save the Children staff to be trainers supported evaluation sustainability at both the national and local levels. This fit naturally into existing staff responsibilities given that staff contact community partners frequently and have strong personal relationships with them.
Evaluation Assistance to Community Partners
Save the Children regional staff and Aguirre assisted community partners in developing plans to evaluate their out-of-school time activities in each of the three elements of the Web of Support: constructive activities, caring adults, and safe places. More than 60 community partners at 230 partner sites throughout the rural United States have adopted Save the Children’s Web of Support principles and work to improve the services they provide to children.
Based on the empowerment approach, rural community partners determined activities to be evaluated, instruments, and methods of data collection, data analysis strategies, and ways to report findings and use information. Save the Children held at least two annual evaluation trainings in each region and provided ongoing assistance in person or via phone to partners throughout the evaluation cycle. The initial training was designed to assist community partners as they developed evaluation plans, instruments, and data collection methods for specific activities. Partners used a variety of evaluation methods and instruments, including pre/post children and youth surveys, pre/post adult surveys (staff, teacher, and parent), observation checklists, parent interviews, collection of existing records (e.g., grades and test scores), and focus groups.
The second training was designed as a hands-on workshop in which partners analyzed their data and created local evaluation reports. Partners were also trained to use their results by discussing ways in which programs/activities could be strengthened.
Initiative-Wide Evaluation
The local evaluation reports developed by community partners and the additional surveys conducted by Aguirre were used to evaluate the Web of Support at the national level, providing a snapshot of the effect Save the Children-supported programs had on children across the country. The findings showed that more than 6,000 children participated in constructive activities that were evaluated. The activities examined were designed to enhance children’s cognitive ability, intellectual skill, and social maturity, and specifically included:
- Tutoring programs for all school subjects
- Reading and math programs to sharpen academic skills
- Computer training to enhance technical understanding
- Recreation programs to develop social and team building skills
- Library programs to encourage interest in reading
- Arts programs to enhance self confidence
- Youth councils to develop leadership skills
Lessons Learned and Best Practices
Since the implementation of the Web of Support, Save the Children and affiliated organizations dealt effectively with a variety of challenges as they determined how to best approach and conduct empowerment evaluation in rural communities. Following are some best practices and lessons learned:
Time and experience are necessary to develop skills. Using the empowerment approach to evaluation provides partners with the autonomy to choose the activities they are interested in evaluating and the methods to conduct their evaluation. However, this approach can place a burden on program staff (e.g. maintaining an evaluation schedule, collecting data, analyzing data, etc.). Partners must appreciate that the burden experienced in learning how to develop and implement outcome evaluation is outweighed by the benefits of improving program quality and contributing to their organization’s infrastructure and sustainability.
Partners need to learn and experience evaluation skills over time. In the Web of Support model, the first year in which partners are introduced to evaluation they are urged to start with basic evaluation to learn procedures (i.e. designing an evaluation plan, creating/identifying instruments, collecting data, analyzing data, and writing an evaluation report) associated with developing and implementing outcome evaluation. After partners become familiar with internal evaluation principles and procedures, they can focus on the evaluation and improvement of services.
Aggregating and interpreting data are skills that take effort to develop. The more partners are involved in the analysis and interpretation of their data, the better equipped they will be to understand the implications and use the results to make meaningful changes to their program(s). During workshops provided by SCF, partners analyzed and interpreted their own data with the assistance of trained evaluators and SCF staff.
Programs need continued guidance as they come to understand the connection between evaluation and improvement in program quality. Continued training, resources, and guidance help develop higher- level evaluation skills. Given this, partners and staff can develop evaluation materials (including instruments) for their programs. For example, the staff at Iyanbito Learning Center near Gallup, New Mexico, wanted to evaluate the Center’s Summer Cultural Enrichment Program. After participating in the Web of Support evaluation trainings and engaging in a year of outcome evaluation, they developed an instrument that measured how much participants learned about their Navajo culture. Staff, along with Alvin Smith (and SCF field manager in the Western Region), led the development of the instrument and Aguirre coaches helped refine it. Partners find data analysis trainings most helpful and productive when they use their own data or relevant program information, which is both meaningful and familiar.
Thoughtful consideration of data findings demonstrates the link between program evaluation and program improvement. It is beneficial for program staff to reflect on findings immediately following data analysis in a forum facilitated by someone knowledgeable in program quality improvement and evaluation. In many cases, SCF staff moderated discussions with individual partners on site. The discussions were designed to lead partners to conclusions about program changes based on evaluation results. Often this was done just before partners completed their annual program plan (a yearly SCF requirement).
Involvement at all levels is necessary for institutionalization of evaluation. Institutionalizing the empowerment approach to outcome evaluation is realistic. The level of interest, resources allocated, and degree of importance that senior staff place on the empowerment approach concept determines the level of commitment participants at all levels have in the adoption of evaluation.
It is important that, at all levels, evaluation is seen as a companion to program quality and improvement. To facilitate this at the national level, SCF conducted monthly conference calls on program improvement and evaluation among SCF home and regional office staff and consultants. At the local level, partners were provided with integrated program development/improvement and evaluation technical assistance by SCF staff.
To understand and integrate results to strengthen programs, discussion of evaluation findings among stakeholders is essential. With more people involved, it is easier to contemplate findings with individual partners and to disseminate new and innovative ideas. Partners are encouraged to take work completed during trainings back to their community for discussion before materials are finalized (including evaluation plans, instruments, and, later, reports). After stakeholders at the local level are satisfied with their information (i.e. evaluation findings), they can provide the information to SCF for national distribution via newsletter, telephone conference call, regional gathering, retreat, and the Inter/intranet.
Community partners with the ability to demonstrate accountability when applying for funding have an improved chance of gaining financial support. This ability promotes program sustainability along with an appreciation of evaluation and its benefits. For example, a community partner in Letcher County, Kentucky, was able to enhance the evaluation section of a 21st Century Community Learning Center’s proposal to the US Department of Education and ultimately receive funding partially because they demonstrated knowledge of and experience in evaluation.
The limitations of the evaluation approach must be recognized. Using the empowerment approach to evaluation, partners are free to define improvement based on their individual program standards. Because definitions often vary among programs, generalizing results is not recommended. The type and rigor of data collection, analysis, and reporting varied between community partners. Therefore, in most cases, program-to program comparisons are not appropriate.
The authors wish to acknowledge the support of Catherine Milton, vice president and executive director, Save the Children US Programs; Terry Russell, associate vice president, Save the Children US Programs; and Claudia Stevenson, research associate, The Aguirre Group.
Linda C. Morell
Evaluation Specialist
Email: lmorell@aiweb.com
Kenneth L. Terao
Project Director
Email: kterao@aiweb.com
The Aguirre Group
480 East 4th Ave., Unit A
San Mateo, CA 94401-3349
Tel: 650-373-4979