Search Publications | The Evaluation Exchange | FINE Network | OST Database

The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.

A Profile of the Evaluation of 21st Century Community Learning Centers—District of Columbia

Program Description

Overview

The District of Columbia 21st Century Community Learning Center (DC 21st CCLC) program began in 1999 as part of a broader initiative, the DC Children and Youth Investment Partnership (DC CYIP), which aims to coordinate youth activities in the District of Columbia so that all youth receive a “seamless web” of services that help them grow up to be healthy and productive adults. The DC 21st CCLC was run by the District of Columbia Public School System (DCPS) and offered school-based after school, summer, and Saturday programs for youth and evening classes for adults. The DC 21st CCLC's three major goals were to: (1) offer significant, expanded learning opportunities for children, youth, and adults in the local school community; (2) help middle school youth meet or exceed state standards in reading and mathematics; and (3) reduce substance abuse and teen violence.

Start Date	fall 1999 (completed summer 2002)
Scope	local
Type	after school, summer/vacation, comprehensive services
Location	urban
Setting	public schools
Participants	kindergarten through middle school students
Number of Sites/Grantees	Ten school sites. DCPS is the grantee, receiving funds directly from the U.S. Department of Education. DCPS, in turn, funded 10 public junior high and middle schools in Washington, D.C. starting during the 1999–2000 school year and going though to 2001–2002. In general fewer sites operated during the summer. For instance, only 9 schools were funded for the summer program in 2000. A few of these schools combined programs and some programs shared sites so that there were a total of only five sites. During the summer of 2001, programming operated at 8 of the 10 21st CCLC school sites. The program was closed during the summer of 2002 due to funding constraints. The after school program took place at all 10 middle/junior high schools during the regular school years from the spring of 2000 to the spring of 2002.
Number Served	1,371 in the after school program during 1999–2000; at least 989 children in the summer program in 2000; 920 children in the summer program in 2001
Components	Each DC 21st CCLC site was to deliver offerings in each of the following areas: (1) education (reading, math, and technofluency); (2) sports/health (physical and mental well-being); (3) arts (visual and performing); and (4) community service. Each DC 21st CCLC summer program was located in a public school and managed by an Assistant Principal (AP) who was responsible for a staff of instructors called facilitators. The program operated from 12:30pm to 6:30pm each weekday during the summer of 2000 and in the morning and afternoons during the summer of 2001. The APs and staff from DCPS collaboratively plan the programs. During the summer of 2001, an academic summer school program, which involved the use of computers in many activities, was offered in the mornings and a nonacademic (e.g., sports, arts, community service, and technology) program was offered in the afternoon. The DC 21st CCLC after school program was established with a flexible design for program implementation. Within the DC 21st CCLC framework, APs worked with school staff, students, and community members to identify and deliver activities that served the needs and interests of the local student population. Most APs selected DCPS teachers as facilitators to conduct specific student activities at their sites. APs were also charged with involving local businesses and community leaders in planning and/or implementing after school programming, as well as with forming a Neighborhood Advisory Council to provide community input on the 21st CCLC programming.
Funding Level	approximately $15 million over three years, according to original plans
Funding Sources	Federal government. DCPS received $4.1 million from the US Department of Education through the 21st CCLC Program. Most of the remaining funds came from the Temporary Assistance for Need Families (TANF) block grant.
Other	The DCPS provided training to APs prior to program implementation and then reviewed implementation with them in meetings and special workshops once the program began. Trainings and meetings covered such topics as the technology and sports program components, how to document program activities, and how to manage and train staff. There were also DCPS trainings held for facilitators regarding the philosophy of the program, activities to be conducted, and specific support for sports coordinators..

Evaluation

Overview	The Urban Institute conducted formative evaluations of the DC 21st CCLC program that focused on implementation and the identification of possible student outcome areas to guide future evaluation work. One evaluation focused on the after school program during the 1999–2000 school year, a second on the summer program during the summer of 2000, and a third on the summer program during 2001 (with a particular focus on the use of computer technology to improve academic achievement).
Evaluators	Jacqueline Raphael, Duncan Chaplin, Zakia Redd, Luke Miller, Meredith Liu, Victoria Russell, Helen Fu, and Emily Anthony from the Urban Institute
Evaluations Profiled	Formative Report on the DC 21st Century Community Learning Center After-School Program Formative Report on the District of Columbia 21st Century Community Learning Center Summer Program Using Technology to Improve Academic Achievement in Out-of-School-Time Programs in Washington, DC
Evaluations Planned	An impact evaluation should be completed by October 2003.
Report Availability	Raphael, J., Chaplin, D., & Redd, Z. (2000). Formative report on the District of Columbia 21st Century Community Learning Center summer program. Washington, DC: The Urban Institute. Available at: www.urban.org/url.cfm?ID=409651. Raphael, J., Chaplin, D., Miller, L., & Redd, Z. (2000). Formative report on the DC 21st Century Community Learning Center after-school program. Washington, DC: The Urban Institute. Available at www.urban.org/url.cfm?ID=410440. Liu, M., Russell, V., Chaplin, D., Raphael, J., Fu, H., & Anthony, E. (2002). Using technology to improve academic achievement in out-of-school-time programs in Washington, DC Washington, DC: The Urban Institute. Available at www.urban.org/url.cfm?ID=410578.

Contacts

Evaluation	Jacqueline Raphael The Urban Institute 2100 M St., NW Washington, DC 20037-1207 Tel: 202-261-5809 Fax: 202-833-2477 Email: jac.8@juno.com	Duncan Chaplin The Urban Institute 2100 M St., NW Washington, DC 20037-1207 Tel: 202-261-5771 Fax: 202-833-2477 Email: dchaplin@ui.urban.org
Program	Saundra Handy Shaw JHS 925 Rhode Island Ave., NW Washington, DC 2002 Tel: 202-607-4208 Fax: 202-673-2463 Email: handy_s4@yahoo.com
Profile Updated	August 18, 2003

Evaluation 3: Using Technology to Improve Academic Achievement in Out-of-School Time Programs in Washington DC

Evaluation Description

Evaluation Purpose	To describe the implementation of the DC 21st CCLC program during the summer of 2001, with a particular focus on the use of computer technology to improve academic achievement.
Evaluation Design	Non-Experimental: Evaluators collected data from all individual summer program sites, including data from program documents, facilitators/instructors, APs, and students.
Data Collection Methods	Document Review: Evaluators reviewed school site grant proposals and monitoring reports prepared by each site in order to learn about program offerings and challenges/accomplishments. Interviews/Focus Groups: Short interviews were conducted with nine APs and 12 facilitators. APs provided data about such topics as hiring, professional development, recruitment of students, data collection, and challenges to program implementation. Facilitators were asked about the goals of activities, support for instruction, and professional development. Observation: Observations were made once each for 17 academically focused technology activities and 31 nonacademic activities.
Data Collection Timeframe	Data were collected during the summer of 2001.

Findings:
Formative/Process Findings

Activity Implementation	Each site used two computer software programs, one for remedial reading and one for math enrichment. Reading classes using this software typically lasted an hour, while math classes typically lasted an hour and a half, and both were offered either every day or twice a week. In a typical session, students used the software to do exercises, complete reading lessons, take tests, and pursue other activities dictated by the software. Students worked independently, occasionally asking the facilitator questions. Many facilitators used progress charts to communicate progress to their students. Most also used a reward system for student progress, such as food, coupons, money, movie passes, and CDs, although some facilitators seemed to give rewards unconditionally rather than as an incentive for good behavior or progress. Many facilitators seemed to pay for these rewards using their own money. In addition to rewards, the reading and math programs generated certificates for mastering skills and lessons. On average, 93% of students were found to be “very” or “somewhat” engaged in the reading and math activities, based on activity observations, although this engagement was found to drop over the course of each day's class. When students disengaged, some sites made little effort to reengage them, and efforts that were made often failed. Despite high levels of engagement, many students reported being disinterested and bored by the programs, although many other students seemed to find them interesting and fun. The reading program was less interesting to students than the math program, although some students commented that they wished the math program were also more challenging and exciting. Students at over half of the sites were found to be using the Internet when they were not supposed to be. The evaluators found that many students had discovered how to shrink their Internet browser when the teacher was coming by to avoid being caught. Despite a four-year range in participants' grade level abilities, nearly all students began both the reading and math software at the same beginning level, which was found to be too easy for many students, especially older students. This lack of individualized treatment contributed to students finding the software boring and not challenging enough.
Program Context/Infrastructure	The quality and quantity of equipment and facilities were judged to be excellent by the evaluators. Classrooms were clean, well lit, and spacious. All but three sites had air conditioning. In every observed activity, each child had access to his/her own computer, and most classrooms had more computers than students. All but one classroom had Internet access. All but one site had access to technical assistance for the computer facilities, although the quality of this technical assistance varied from site to site.
Recruitment/Participation	Students gave many different reasons for attending the morning summer school program, but they predominantly mentioned poor academic performance (either low grades or low SAT-9 scores), parental requests, and having “nothing better to do.” Despite one more site being open in 2001, fewer children participated in this year than in the year prior (920 vs. 989). Enrollment and attendance issues were found to have impeded implementation, with six of the observed activities reporting such difficulties. Most participants (27 of 33 interviewed in focus groups) were already very comfortable with computers and used them outside of school, with access to computers at home.
Staffing/Training	The average student to facilitator ratio across sites was six to one. A number of facilitators lacked knowledge of common computer application packages and only a few considered themselves to be computer experts. All facilitators felt comfortable teaching the programs they were using with their students, however, since they received training in those programs. Students seemed comfortable with facilitators. A few facilitators seemed especially helpful, telling students not to get frustrated, joking with them, patting them on the back, and giving positive feedback. In addition to the facilitators, a number of other adult aides were present for many activities. These aides were found to be of mixed quality. In about half of the cases, the aides were found to be of the same high quality as the facilitators, but in other cases the aides seemed to have little interaction with students and simply occupied computers and looked at papers. One problem may have been the lack of familiarity with computer activities on the part of these aides.

Summative/Outcome Findings

Academic

Evaluators found little evidence that the reading and math programs contributed to students' excitement or motivation regarding their academic progress.

A Profile of the Evaluation of 21st Century Community Learning Centers—District of Columbia

Profile Information