Volume XII, Number 1 & 2, Fall 2006

Issue Topic: Building and Evaluating Out-of-School Time Connections

Evaluations to Watch

Measuring Change: Out-of-School Time Programs and School Culture

Holly Morehouse describes how out-of-school time programs connected to the school day transformed one district's school culture.

Though geographically the smallest school district in Vermont, the Winooski School District (WSD) is one of the state's most diverse, with students who speak over 20 different languages. Close to 35% of the student population transfers out of or into WSD every year, and 54% qualify for free or reduced-price lunch. In 2003, WSD received a 21st Century Community Learning Centers (21st CCLC) grant, which we used to build an AfterZone1 that offers 60 different programs after school, over vacations, and during the summer.

The AfterZone programs have had a profound and welcome impact on our schools, particularly in regard to school culture. WSD teachers are working together in new ways, students and teachers are building multidimensional relationships, and students are demonstrating greater involvement in extracurricular learning. Today, approximately 46% of the K–12 student population and 35% of the teaching faculty participate in our out-of-school time (OST) programs.

To measure the effectiveness of these programs, we have devised a three-pronged evaluation. Based on evaluation resources from OST and other fields, our approach uses both measures of effort¸ which look at outputs (the products and services generated by the program and its activities) and measures of effect, which track “changes in knowledge, skills, attitudes, or behaviors in [the] target population.”2 We have added a third data category—measures of change, which looks at changes in school culture—that is, sustained changes in how teachers, principals, and other staff “relate to each other in different ways and actually do something differently” as a result of the program.3

In evaluating our OST programs' impacts on school culture, we have identified four indicators of change:

1. Number of teachers from the district participating in OST programs. Last year, 31 out of 89 district teachers and 8 para-educators taught in the Winooski AfterZone. These teachers learned new skills, shared ideas, and engaged students. Teachers have come to perceive WSD schools as a place where they will be supported in developing innovative curricula and opportunities for learning that go beyond the regular school day.

2. Number of student-initiated programs. Twelve of Winooski's AfterZone programs this past year were initiated by students. Students developed the concepts, approached teachers to be the instructors, and worked with the instructors to design the programs. WSD sees student initiative in identifying and developing AfterZone programs as a powerful indicator of growing student involvement.

3. Connections to the regular school day. While it is difficult to record every way that ideas, strategies, and relationships from an OST program carry over into the regular school day, last year WSD documented 14 teacher reports of such carryover. Examples include:

• Two teachers who pair up regularly for teaching AfterZone courses and now meet over lunch to discuss teaching strategies and project ideas for their classrooms
• The after school literary journal club whose monthly “magazine” is distributed throughout the school, sparking student discussions
• A science teacher who includes a unit on rockets in his high school courses after working with an outside expert over the summer

4. Opportunities for teacher collaboration. Nearly half of the programs offered in 2004–2005 and 2005–2006 were taught by teams of two or three teachers, often from different grade levels and sometimes from different schools. Research has shown that school culture benefits from teacher collaboration, communication, and direct involvement by continually renewing instructional methods and curriculum and by building an atmosphere of “collegiality, trust, and shared mission.”4 WSD also builds teacher collaboration by partnering with the University of Vermont to offer graduate-level summer courses to all teachers in Winooski's summer school and AfterZone summer programs.

Since establishing the AfterZone, Winooski has seen increased communication and exchange across the district's three schools. Teachers from all grade levels come together in professional development workshops and team up to develop new school-day and OST curricula, design thematic units, and try out new strategies. We have also seen a new level of excitement, with students, parents, educators, and community members participating in events and experiencing that special “ah ha!” of hands-on learning.

The next step in our evaluation is to build a comprehensive database expanding the definition of the Winooski AfterZone to include all program activities taking place after school in the district and the community. This working partnership will bring in the school's athletic programs, the Winooski Parks & Recreation Department, the Boys & Girls Club, the Teen Center, and more.

1 For more information about the Winooksi AfterZone, please contact Bruce Chattman, Superintendent, Winooski School District, 60 Normand Street, Winooski, VT 05404. Tel: 802-655-0485. Email: bchattman@winooski.k12.vt.us
2 Little, P. M. D., Harris, E., & Bouffard, S. (2004). Performance measures in out-of-school time evaluation. Cambridge, MA: Harvard Family Research Project. [Click here to view.]
3 Kelleher, P., & Levenson, M. R. Can school culture change? School Administrator, 61(8), 54–55.
4 Peterson, K., & Brietzke, R. (1994). Building collaborative cultures: Seeking ways to reshape urban schools. Oak Brook, IL: North Central Regional Educational Laboratory.

Holly Morehouse, Ph.D.
Vermont 21st CCLC State Coordinator
Vermont Department of Education
120 State Street
Montpelier, VT 05620
Tel: 802-828-0619
Email: hollymorehouse@education.state.vt.us