Volume lll, Number 2, 1997

Issue Topic: School-Linked Services

Evaluations to Watch

Evaluation of an Extended Services School in Boston

Mary Walsh and George Madaus, Professors at the Graduate School of Education at Boston College, talk about the development of a model extended services school in Boston.

With funding from the DeWitt Wallace Reader's Digest Foundation, a model extended services school is being developed at the Gardner Elementary School in the Allston–Brighton neighborhood of Boston. The Gardner is a K–5 school (550 children; 31 languages spoken) which buses 60 percent of its population from across Boston. The Gardner Extended Services School (GESS) is being adapted from the model of the Children's Aid Society Schools in New York and will focus on education and career development. It will extend the school day, link the school with community agencies, and respond to the needs of two generations. A partnership among Boston College, the Gardner School staff and parents, the local YMCA, and a community coalition (Healthy Boston) is coordinating the planning and eventual implementation of the GESS.

Evaluation Design

Two evaluations will be conducted. The process of implementing the GESS will be evaluated by the DeWitt Wallace Reader's Digest Foundation. The effectiveness of the GESS will be evaluated by George Madaus of the Boston College Center for the Study of Testing, Evaluation and Educational Policy.

The effectiveness evaluation reflects the Stufflebeam CIPP Model (consisting of Context, Input, Process, and Product evaluations) modified by adding strong naturalistic components. The time frame is 5 years and includes assessing longitudinally the inputs, processes, and outcomes of the project. The evaluation will include four phases. Each phase will examine the needs of the four target groups (children, parents, teachers, and community), the resources necessary to address those needs, the delivery of those resources to each of the four target groups, and the outcomes—intended and unintended—associated with the delivery of those services and resources. For each evaluation question, within each phase, a rationale is provided so that target groups and participants have a clear understanding of the nature and purpose of the evaluation. The phases are as follows:

1. Early planning phase. This phase will gather needs assessment/planning information on the four populations that are the focus of the GESS effort. It is anticipated that many unforeseen issues will arise at this early stage of the project, and part of the role of the evaluators will be to help project planners deal with emerging issues. In essence, this phase of the evaluation will be responsive to the needs of the planners.

2. Initial project implementation phase. This phase is designed to answer formative questions about early initiatives/plans/processes necessary to enable each of the four target groups to reach program goals. It will also take into account the issues faced by the planners who, at this point in the process, may not have a completely clear notion of which services are effective. Thus, the evaluation team will concentrate on providing information that will guide decision making about the participants' needs, the resources devoted to providing services, and the actual delivery of services.

3. Full-scale project implementation phase. This phase, which will last several years, is designed to collect ongoing quantitative and qualitative data on the four target groups. While it is expected that the project will be more clearly defined at this stage, this evaluation is intended to provide planners with the information necessary to meet ongoing decision-making needs.

4. Impact assessment phase. This phase is designed to give funders necessary information on outcomes—intended and unintended—so that the project services can be improved, extended, evaluated, and funded in the future.

In each phase, it was deemed necessary to identify the stated goals, inputs, process, and outcomes of the project, and then, through a process of study and inference, to delineate the actual objectives, goals, inputs, process, and outcomes for each of the target populations. A significant effort was made to design evaluation questions that probed the perceptions of the four target groups on the stated and de facto goals, inputs, process, and outcomes of this project.

For each phase it is necessary to rely on both quantitative and qualitative methods of data collection in order to obtain a richer, triangulated look at the questions within each of the phases. In choosing the variables for each population at each phase, the evaluators were guided by literature reviews on what seemed to work. They designed evaluation indices and data, gathering methodologies around techniques known to maximize the richness of the information vis-à-vis the projected goals.

The long-term, continuous nature of this evaluation will provide project planners, managers, and partners with ongoing information about the program's strengths and weaknesses that can be used for program development and improvement. While the evaluation plan has provided specific details on a number of key aspects of the evaluation effort (e.g., data collection instruments, reporting plans, management plans, and budget) the evaluation has been designed to be flexible and responsive to issues that arise over the course of the project. As the project develops, additional issues not included in this proposal may arise and need to be addressed. Conversely, because issues deemed to be important at this time may eventually not be critical issues for evaluation, the program planners and their evaluation team will need to adjust and direct the evaluation as necessary.

Mary Walsh
Professor

George Madaus
Boisi Professor of Education and Public Policy

Graduate School of Education
Boston College
Campion Hall 309
Chestnut Hill, MA 02167
Tel: 617-552-4710
Fax: 617-552-1981