Volume VIII, Number 2, Fall 2002

Issue Topic: Evaluation for Continuous Improvement

Evaluations to Watch

Evaluating Citizen Schools

Charlie Schlegel of Citizen Schools explains how their evaluation strategy successfully balances the need to determine program impact with the need for continuous improvement.

Founded in 1995, Citizen Schools operates after school programs in 16 public schools, including 12 in Boston and 4 affiliated sites nationally. Each week, almost 1,000 middle school students—most from low-income households—participate in up to 16 hours of engaging, hands-on learning activities to improve their skills in writing and data analysis as well as their leadership and civic interest. Students also participate in “apprenticeships” taught by adult volunteers, who share their expertise and professionalism through 10-week projects, culminating in a final product or performance.

Citizen Schools Evaluation
Evaluation plays two important roles in the success of our program. First, regular and rigorous assessments of the program quality and impact offer us, as well as policymakers, objective measures of its effect on student achievement and “life trajectory.” Second, using the balanced scorecard and other approaches, evaluation generates the performance data and constituent feedback necessary for continuous improvement.

Evaluation at Citizen Schools translates our broadly-stated mission—“to educate kids and strengthen communities”—into four outcome areas: community, leadership, access, and skills (CLAS). Each aspect of the program—the writing and data projects, volunteer-led apprenticeships, homework assistance, and community explorations—are intended to improve students’ lives in one or more of the outcome areas. We use four evaluation approaches to assess program impact and analyze data for continuous improvement.

Approach One – Skill Assessments
Citizen Schools assesses students’ writing, data, and oral skills at the beginning and end of each school year. Our staff uses this information to identify the specific skill areas in which students need the most help and where they improve over the school year. We use the results and analysis of each skills assessment to make changes in the way our different sites allot time and staff resources towards each skill area and to adjust the curriculum and structure of our skill-based projects.

Approach Two – Constituent Surveys
Midway through each school year and at the close of the program Citizen Schools asks students, parents, volunteers, school teachers, and staff for feedback through its constituent surveys. The surveys ask about constituents’ experience in the program, their assessments of its impact, and suggestions for improvement. We analyze constituent feedback, looking for trends and specific responses that may lead to improvement in the quality of the program and its measures of impact. Over time survey feedback has resulted in programmatic changes, such as additional supports for adult volunteers and different marketing strategies for different constituents.

Approach Three – The Balanced Scorecard
The balanced scorecard is a management system that enables organizations to clarify their vision and strategy and translate them into action. It provides feedback around both internal program processes and external outcomes in order to continuously improve strategic performance and results. (The Balanced Scorecard Institute)

Citizen Schools uses a balanced scorecard to monitor our performance in all areas. Our balanced scorecard includes several program objectives, as well as fundraising, staff management, and national and state policy objectives. We report our progress on each of the balanced scorecard measures to our Board of Directors, staff, and key stakeholders quarterly. Each year the leadership of Citizen Schools establishes the specific objectives and measures of the balanced scorecard for that year based on feedback from staff and other stakeholders within the organization. Staff members collect data for the scorecard and track progress on objectives specific to their departments.

Over time Citizen Schools has found the balanced scorecard to be an effective organizational lever, helping to draw managerial attention and resources to under-resourced initiatives and priorities. For instance, our 2002 balanced scorecard includes “measures” to develop plans for regularly collecting student academic data and to design baseline measures to assess the program’s impact on adult volunteers. In short, the balanced scorecard helps keep the scarce resources of the organization focused on the essentials of running high-quality after school programs.

Approach Four – The Impact Study
The Impact Study is a longitudinal, quasi-experimental evaluation conducted by Policy Studies Associates (PSA), an evaluation firm headquartered in Washington D.C. Begun in the fall of 2001, this evaluation uses several methods of data collection and analysis to addresses questions of program implementation, outcomes, and impact on students’ school performance and social maturity. This externally-driven evaluation helps us collect the necessary outcomes data so our staff can focus on collecting data for continuous improvement.

The relationship between internal efforts to use data for continuous improvement and externally-generated data for proving impact is complementary. Citizen Schools and PSA both collect data about students’ school performance and social maturity—data that we use to measure the impact of our program as well as inform our decisions about quality improvements in our services.

Charlie Schlegel
Director of Research and Evaluation
Citizen Schools
Museum Wharf
308 Congress Street
Boston, MA 02210
Tel: 617-695-2300
Email: charlieschlegel@citizenschools.org