The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.
HFRP
> Out-of-School Time
> OST Database & Bibliography
> Database
> 21st Century Community Learning Centers—District of Columbia
A Profile of the Evaluation of 21st Century Community Learning Centers—District of Columbia
Profile Information
- Full Text (HTML)
- Request reprint permission
- How to cite
- About the OST Database
Program Description
Overview
The District of Columbia 21st Century Community Learning Center (DC 21st CCLC) program began in 1999 as part of a broader initiative, the DC Children and Youth Investment Partnership (DC CYIP), which aims to coordinate youth activities in the District of Columbia so that all youth receive a “seamless web” of services that help them grow up to be healthy and productive adults. The DC 21st CCLC was run by the District of Columbia Public School System (DCPS) and offered school-based after school, summer, and Saturday programs for youth and evening classes for adults. The DC 21st CCLC's three major goals were to: (1) offer significant, expanded learning opportunities for children, youth, and adults in the local school community; (2) help middle school youth meet or exceed state standards in reading and mathematics; and (3) reduce substance abuse and teen violence.
| Start Date | fall 1999 (completed summer 2002) |
| Scope | local |
| Type | after school, summer/vacation, comprehensive services |
| Location | urban |
| Setting | public schools |
| Participants | kindergarten through middle school students |
| Number of Sites/Grantees | Ten school sites. DCPS is the grantee, receiving funds directly from the U.S. Department of Education. DCPS, in turn, funded 10 public junior high and middle schools in Washington, D.C. starting during the 1999–2000 school year and going though to 2001–2002. In general fewer sites operated during the summer. For instance, only 9 schools were funded for the summer program in 2000. A few of these schools combined programs and some programs shared sites so that there were a total of only five sites. During the summer of 2001, programming operated at 8 of the 10 21st CCLC school sites. The program was closed during the summer of 2002 due to funding constraints. The after school program took place at all 10 middle/junior high schools during the regular school years from the spring of 2000 to the spring of 2002. |
| Number Served | 1,371 in the after school program during 1999–2000; at least 989 children in the summer program in 2000; 920 children in the summer program in 2001 |
| Components | Each DC 21st CCLC site was to deliver offerings in each of the following areas: (1) education (reading, math, and technofluency); (2) sports/health (physical and mental well-being); (3) arts (visual and performing); and (4) community service. Each DC 21st CCLC summer program was located in a public school and managed by an Assistant Principal (AP) who was responsible for a staff of instructors called facilitators. The program operated from 12:30pm to 6:30pm each weekday during the summer of 2000 and in the morning and afternoons during the summer of 2001. The APs and staff from DCPS collaboratively plan the programs. During the summer of 2001, an academic summer school program, which involved the use of computers in many activities, was offered in the mornings and a nonacademic (e.g., sports, arts, community service, and technology) program was offered in the afternoon. The DC 21st CCLC after school program was established with a flexible design for program implementation. Within the DC 21st CCLC framework, APs worked with school staff, students, and community members to identify and deliver activities that served the needs and interests of the local student population. Most APs selected DCPS teachers as facilitators to conduct specific student activities at their sites. APs were also charged with involving local businesses and community leaders in planning and/or implementing after school programming, as well as with forming a Neighborhood Advisory Council to provide community input on the 21st CCLC programming. |
| Funding Level | approximately $15 million over three years, according to original plans |
| Funding Sources | Federal government. DCPS received $4.1 million from the US Department of Education through the 21st CCLC Program. Most of the remaining funds came from the Temporary Assistance for Need Families (TANF) block grant. |
| Other | The DCPS provided training to APs prior to program implementation and then reviewed implementation with them in meetings and special workshops once the program began. Trainings and meetings covered such topics as the technology and sports program components, how to document program activities, and how to manage and train staff. There were also DCPS trainings held for facilitators regarding the philosophy of the program, activities to be conducted, and specific support for sports coordinators.. |
Evaluation
| Overview | The Urban Institute conducted formative evaluations of the DC 21st CCLC program that focused on implementation and the identification of possible student outcome areas to guide future evaluation work. One evaluation focused on the after school program during the 1999–2000 school year, a second on the summer program during the summer of 2000, and a third on the summer program during 2001 (with a particular focus on the use of computer technology to improve academic achievement). |
| Evaluators | Jacqueline Raphael, Duncan Chaplin, Zakia Redd, Luke Miller, Meredith Liu, Victoria Russell, Helen Fu, and Emily Anthony from the Urban Institute |
| Evaluations Profiled | Formative Report on the DC 21st Century Community Learning Center After-School Program Formative Report on the District of Columbia 21st Century Community Learning Center Summer Program Using Technology to Improve Academic Achievement in Out-of-School-Time Programs in Washington, DC |
| Evaluations Planned | An impact evaluation should be completed by October 2003. |
| Report Availability | Raphael, J., Chaplin, D., & Redd, Z. (2000). Formative report on the District of Columbia 21st Century Community Learning Center summer program. Washington, DC: The Urban Institute. Available at: www.urban.org/url.cfm?ID=409651. Raphael, J., Chaplin, D., Miller, L., & Redd, Z. (2000). Formative report on the DC 21st Century Community Learning Center after-school program. Washington, DC: The Urban Institute. Available at www.urban.org/url.cfm?ID=410440. Liu, M., Russell, V., Chaplin, D., Raphael, J., Fu, H., & Anthony, E. (2002). Using technology to improve academic achievement in out-of-school-time programs in Washington, DC Washington, DC: The Urban Institute. Available at www.urban.org/url.cfm?ID=410578. |
Contacts
| Evaluation | Jacqueline Raphael The Urban Institute 2100 M St., NW Washington, DC 20037-1207 Tel: 202-261-5809 Fax: 202-833-2477 Email: jac.8@juno.com |
Duncan Chaplin The Urban Institute 2100 M St., NW Washington, DC 20037-1207 Tel: 202-261-5771 Fax: 202-833-2477 Email: dchaplin@ui.urban.org |
| Program | Saundra Handy Shaw JHS 925 Rhode Island Ave., NW Washington, DC 2002 Tel: 202-607-4208 Fax: 202-673-2463 Email: handy_s4@yahoo.com |
|
| Profile Updated | August 18, 2003 | |
Evaluation 1: Formative Report on the DC 21st Century Community Learning Center After-School Program
Evaluation Description
| Evaluation Purpose | To describe the implementation of the DC 21st CCLC after school program between October 1999 and May 2000 and the implications of the current implementation for further evaluation of the program. |
| Evaluation Design | Non-Experimental: Data were collected from all individual after school program sites, including APs and students. |
| Data Collection Methods | Document Review: DCPS' grant proposal to the US Department of Education for the DC 21st CCLC program, as well as each of the 10 schools' after school program grant proposal to DCPS were analyzed. Also reviewed was the DCPS Performance Report for the program, submitted to the US Department of Education in April 2000. Interviews/Focus Groups: Semi-structured interviews were conducted with the 10 APs, which lasted about one hour and covered issues of program implementation at the sites. Semi-structured focus groups were also conducted with between seven and twelve students at each site, which lasted about 30 minutes and covered issues of students' participation and their impressions of the program, including any potential benefits to them from their participation. Observation: Informal observations of program facilities and students' participation in program activities were made to provide information about program implementation. |
| Data Collection Timeframe | Data were collected during the 1999–2000 school year. |
Findings:
Formative/Process Findings
| Activity Implementation | Due to Department of Education oversight rules prohibiting access to funds until late February 2000, many after school activities could not be implemented until that time; these include: direct instruction in reading and mathematics, development of technology and academic skills on DC 21st CCLC computers, substance abuse, pregnancy, and violence prevention activities, intergenerational activities, and educational and entertainment field trips. Despite these limitations, nearly all sites reported student activities being conducted in all four key component areas (education, sports/health, arts, and community service) prior to this date, although the intensity and duration of such activities varied widely across the 10 program sites. Most sites offered activities in all four components every week or two. The evaluators found that a wide variety of activities were offered across the 10 sites, indicating that APs were tailoring program offerings to local needs and staff capabilities. APs frequently mentioned that students' interests were a significant factor in deciding what activities to offer. At most sites, students participated in two or three activities per day. Several programs gave students a “choice” day, typically Fridays, when students were able to select the activities in which they wished to participate. The degree of student choice varied considerably from site to site. |
| Parent/Community Involvement | APs reported moderate levels of collaboration with two or more community entities, including neighborhood businesses, churches, religious groups, foundations, sports facilities, and various guest speakers. Collaboration usually entailed donated services, facilities, staffing, and/or transportation to the sites. At the “high end” of community involvement, one site had local business owners teach students entrepreneurial skills, while students at another site received individual tutoring from local college student volunteers. All but one site reported having formed Neighborhood Advisory Councils, but none had yet conducted meetings at the time of the site visits. Many APs noted it was difficult to find meeting times that accommodated the majority of members' schedules. APs and students reported that parents were not yet very involved in the after school programs, but several APs reported having plans to engage more parents. |
| Program Context/Infrastructure | APs implemented various policies for maintaining discipline and student safety in the program, including: suspending disruptive students from the program, employing security guards and metal detectors, requiring written parental permission for students to participate, making sure nonparticipants didn't “hang out” in the school or on the fields during after school program operation, and locking doors during the program. Students at several schools mentioned that the program helps to keep them off the streets. Many said that if they were not in the program they would be hanging out on the streets, doing “nothing,” watching television, sleeping, or participating in unsupervised recreational or academic activities. |
| Recruitment/Participation | APs publicized the program by making announcements over the school intercom, posting notices in the school, talking about the program at PTA and YMCA meetings, sending home flyers/letters/brochures, and soliciting referrals from guidance counselors. The program reached almost 70% of its overall enrollment goal of serving 2,000 students across all sites, but routinely served less than half this number on a day-to-day basis. APs suggested that family needs, distance between students' homes and school, lack of initial funding (inhibiting the provision of popular activities), other after school activities, and lack of student interest may have inhibited greater student participation. |
| Satisfaction | Students reported that they enjoyed the program and participated in activities to which they would not otherwise have access. Students were especially enthusiastic about the sports activities offered, and the individualized academic attention they received from facilitators. Students expressed positive feelings in regards to their interactions with peers at the program, especially opportunities to interact with students from other grade levels. |
| Staffing/Training | Seven out of 10 APs reported having worked in some capacity at their sites for at least 10 years, three of them between 17 and 30 years. Several reported having worked at after school programs that existed before the 21st CCLC was initiated. Most APs personally recruited program facilitators. They sought staff with flexible skills and the ability to nurture and motivate students. Several APs specifically recruited facilitators with skills in areas such as computer literacy and mathematics. Many students reported meaningful interactions with the adult program staff. Students reported especially enjoying interaction with the staff in a non-classroom setting, such as playing sports. A number of APs expressed concerns about how their roles were defined at the school, particularly in regard to their responsibilities in relation to the regular school day vs. the after school programming. Some expressed frustrations related to insufficient time to devote attention to the after school program, lack of program assistants, and redundant paperwork required for the 21st CCLC program. Some APs expressed concerns about facilitator staffing, in that some facilitators were sometimes late due to regular day positions at other schools. Some APs wanted to hire facilitators from outside the DCPS system, but were advised not to by DCPS because of the extra time needed to process non-DCPS applicants, leading some APs to feel limited in their hiring powers. Some APs reported difficulty dealing with the 1:20 staffing ratio, given that program attendance fluctuated quite a bit on a daily basis. APs reported some staff had difficulty managing students and maintaining discipline. Both students and APs indicated a need for more staff to ensure student safety, particularly in getting students home safely from the program. |
Evaluation 2: Formative Report on the District of Columbia 21st Century Community Learning Center Summer Program
Evaluation Description
| Evaluation Purpose | To examine the implementation of the DC 21st CCLC program during the summer of 2000, particularly in regard to DCPS' site support, and to identify potential student outcomes to be explored further in the upcoming impact evaluation. |
| Evaluation Design | Non-Experimental: Data were collected from all individual summer program sites, including facilitators/instructors, APs, and parents. |
| Data Collection Methods | Document Review: Each of the nine school program grant proposals to DCPS were reviewed along with DCPS documents outlining professional development activities for DC 21st CCLC summer program staff. Interviews/Focus Groups: Short interviews were conducted with nine APs and 12 facilitators. APs provided data about such topics as hiring, professional development, recruitment of students, data collection, and challenges to program implementation. Facilitators were asked about the goals of activities, support for instruction, and professional development. Focus groups were held with parents. Parents were asked about reasons why their children participated, the extent of their children's enthusiasm, and their perceptions of the benefits of the program for their children, including changes in attitudes and behaviors. DCPS staff who oversee the 21st CCLC summer program were also interviewed about the program's theory of action, goals, and how the program was intended to be implemented. Observation: Twenty-two 30- to 60-minute program activities were observed. At least two observations took place at each site. Nine hundred and eight-nine children between the ages of 4 and 13 were observed being served at the five sites. Presumably a larger number participated at some point during the summer. |
| Data Collection Timeframe | Data were collected during the summer of 2000. |
Findings:
Formative/Process Findings
| Activity Implementation | About one quarter of the program time was devoted to the development of academic skills including reading, writing, mathematics, and problem solving. Most activities were hands-on, including such tasks as dancing, singing, artwork, computer use, and educational games. Some activities were designed to meet specific needs or interests of a particular group of students, such as: weightlifting for boys preparing for football season, a fashion show for interested students, and a wall mural activity for those interested in mural making. Most program activities involved groups of children of similar ages, but children across ages were combined for such activities as field trips and athletic competitions. Most activities were led by one facilitator assisted by one to four aides. Activity periods were usually 90 minutes with one site offering three-hour activity blocks. Evaluators concluded that most activities were well planned with little idle time. Students frequently took leadership roles in activities, such as leading peers in athletic stretches. Generally, facilitators and aides maintained discipline and students stayed on task. Some facilitators displayed better classroom management than others. In two observed activities, poor classroom management was evident, particularly when relating to older junior high/middle school students. Students were generally given choices about which activity to pursue and given the option of switching activities at any time. Students were generally observed to persist in one activity rather than switch frequently. In many activities, particularly technology activities, students were able to pace themselves according to their individual abilities and needs. Students were encouraged to compete with each other in a number of areas including sports, board games, typing speed tests, and a stock market game. Students were given ample opportunities to interact freely with one another and were able to select with whom they interacted, often choosing friends. Evaluators observed that this provides an opportunity for children to practice social skills with adults present to facilitate and intervene should problems arise. Although students appeared more engaged in some activities than others, they appeared to enjoy almost all of the observed activities. Students demonstrated high levels of engagement in the activities, often discussing the task at hand with one another. Students appeared to be comfortable using technology. In almost all small group activities, and many large group activities, children received individual attention from adults. This attention took the form of instruction, encouragement, advice, and constructive feedback. A high level of collaboration was observed between students, particularly older students. The collaboration included working together to create joint products, helping each other with skills, and jointly resolving conflicts. |
| Program Context/Infrastructure | The program was implemented largely as planned at five sites covering nine of the 10 intended schools. Transportation issues limited participation in the program. Because a number of the school summer programs had to be combined due to summer closures for renovations at several sites, some children had further to travel since their program was not located at their own school's site. Families that were unable to transport their children to the alternative site or unwilling to allow their children to take public transportation to get there were not able to participate in the program. DCPS also runs other summer programs that operate in the mornings and end when the DC 21st CCLC program begins (12:30pm). For some 21st CCLC participants, the morning program and the afternoon 21st CCLC program were held at different sites, which was noted as an inconvenience for parents. One AP expressed concerns about the safety of allowing children to return home alone after dark. Nearly all APs and facilitators reported having adequate facilities, equipment, and supplies to implement the program. Some APs and facilitators felt that more upfront planning time would have improved program quality. One facilitator mentioned the need to start planning three to four weeks before the program began. In general, some staff felt rushed to begin the summer program. |
| Recruitment/Participation | The program initially intended to serve 2,000 children in grades six through eight, but changed plans to reflect a more limited capacity and an acknowledgment of the greater attendance rates of younger children as compared to older children. The revised goal was to serve between 1,000 and 1,400 children from nine middle and junior high schools and from the elementary schools that feed into the middle and junior high schools. Three APs noted that “competing” summer programs in the communities they served hindered their ability to recruit students. Two APs spoke of the need to coordinate the 21st CCLC program with existing summer programs. In small group activities, there were an average of 13 students in each group and a range of 4 to 20 students. A few larger athletic group activities included as many as 80 students. |
| Satisfaction | Parents provided numerous examples of their children's enjoyment of the program, including being ready to go to the program early, instructing parents not to pick them up early, and discussing activities with parents at home. Some parents expressed surprise that the program maintained their children's interest for the program's duration since previous summer programs in which their children had participated did not. Parents expressed satisfaction with the fact that their children were spending time with students from different schools and of different ages. Some parents appreciated that the program was free; they indicated that they would have had to pay to enroll their children in other programs. |
| Staffing/Training | Facilitators who were hired in time for a scheduled training received four days of preservice training. One AP felt that the facilitators, particularly those who had not worked in the after school component during the regular school year, needed more training. This was corroborated by a new facilitator who said he could have used more training on program procedures. On the whole, APs felt adequately prepared to implement the summer program in terms of planning and staff training. Most facilitators reported receiving only a half-day or less of training; either they did not attend the four-day orientation or did not consider it to be training. The athletic facilitators reported that the training they received from DCPS was helpful. Most facilitators reported receiving adequate support from their APs, particularly with respect to requests for materials, payroll issues, and disciplinary proceedings. APs felt that they had hired the appropriate number of staff for the students served. However, two APs felt that it would have been more efficient to postpone hiring decisions until the second or third week of the program (instead of the first) since enrollment often picked up and programs had to fire and then rehire staff as numbers grew. Facilitators generally reported that they enjoyed working with children of varied ages and that they were successful with them despite the fact that most facilitators were teachers at junior high/middle schools. The one exception to this appreciation for the age diversity was found at one site where activity blocks were longer-three facilitators at this site indicated less comfort in working with younger children because they felt that they were harder to engage. Some APs were concerned about their schedules, which had them arrive at about the same time as the students and participate in nightly meetings from 7 to 9pm. Two APs said that they were not comfortable arriving at the same time as the students because they would have preferred to make sure the site was prepared before student arrival. One AP mentioned that the DCPS should avoid evening meetings since APs were too tired after working with children all day and needed more time to prepare for their programs. Facilitators were assisted by student aides, AmeriCorps members, and a few parents during most activities. The facilitators valued this assistance. Student aides appear to have been underused. Many student aides attended the same school or lived in the same neighborhood as participants, which may have made it difficult for them to serve as supervisors or mentors. Student aides were provided with good work experience through the program. The AmeriCorps volunteers were more active members of the staff, providing structured support to facilitators and students particularly in the areas of technology and art. AmeriCorps aides were older than student aides and tended to have more experience. Facilitators and DCPS staff reported a reliance on AmeriCorps aides for teaching and supporting the children, especially in the technology component. Parents also spoke highly of AmeriCorps members, noting that children developed meaningful relationships with them. Observed activities were deemed by the evaluators to be adequately staffed. |
Summative/Outcome Findings
| Family | Time away from their children reduced stress in some families, according to parents. Some parents appreciated the efficiency and familial benefits of having a place to send all of their elementary through junior high school aged children. That children from the same family could be together in the program was seen as convenient and beneficial to the family. |
| Youth Development | Some parents reported that their children were “around a better group of students” and had the opportunity to build “more positive” friendships through the program. Some parents reported that their children were displaying increased self-confidence at home: dancing, singing, and expressing more opinions. These parents were not certain, however, that these changes in their children would not have taken place without the DC 21st CCLC summer program. Some parents of children with attention deficit disorders or behavioral problems mentioned that their children were learning to be calmer and more focused by taking part in activities they enjoyed in the program. |
Evaluation 3: Using Technology to Improve Academic Achievement in Out-of-School Time Programs in Washington DC
Evaluation Description
| Evaluation Purpose | To describe the implementation of the DC 21st CCLC program during the summer of 2001, with a particular focus on the use of computer technology to improve academic achievement. |
| Evaluation Design | Non-Experimental: Evaluators collected data from all individual summer program sites, including data from program documents, facilitators/instructors, APs, and students. |
| Data Collection Methods | Document Review: Evaluators reviewed school site grant proposals and monitoring reports prepared by each site in order to learn about program offerings and challenges/accomplishments. Interviews/Focus Groups: Short interviews were conducted with nine APs and 12 facilitators. APs provided data about such topics as hiring, professional development, recruitment of students, data collection, and challenges to program implementation. Facilitators were asked about the goals of activities, support for instruction, and professional development. Observation: Observations were made once each for 17 academically focused technology activities and 31 nonacademic activities. |
| Data Collection Timeframe | Data were collected during the summer of 2001. |
Findings:
Formative/Process Findings
| Activity Implementation | Each site used two computer software programs, one for remedial reading and one for math enrichment. Reading classes using this software typically lasted an hour, while math classes typically lasted an hour and a half, and both were offered either every day or twice a week. In a typical session, students used the software to do exercises, complete reading lessons, take tests, and pursue other activities dictated by the software. Students worked independently, occasionally asking the facilitator questions. Many facilitators used progress charts to communicate progress to their students. Most also used a reward system for student progress, such as food, coupons, money, movie passes, and CDs, although some facilitators seemed to give rewards unconditionally rather than as an incentive for good behavior or progress. Many facilitators seemed to pay for these rewards using their own money. In addition to rewards, the reading and math programs generated certificates for mastering skills and lessons. On average, 93% of students were found to be “very” or “somewhat” engaged in the reading and math activities, based on activity observations, although this engagement was found to drop over the course of each day's class. When students disengaged, some sites made little effort to reengage them, and efforts that were made often failed. Despite high levels of engagement, many students reported being disinterested and bored by the programs, although many other students seemed to find them interesting and fun. The reading program was less interesting to students than the math program, although some students commented that they wished the math program were also more challenging and exciting. Students at over half of the sites were found to be using the Internet when they were not supposed to be. The evaluators found that many students had discovered how to shrink their Internet browser when the teacher was coming by to avoid being caught. Despite a four-year range in participants' grade level abilities, nearly all students began both the reading and math software at the same beginning level, which was found to be too easy for many students, especially older students. This lack of individualized treatment contributed to students finding the software boring and not challenging enough. |
| Program Context/Infrastructure | The quality and quantity of equipment and facilities were judged to be excellent by the evaluators. Classrooms were clean, well lit, and spacious. All but three sites had air conditioning. In every observed activity, each child had access to his/her own computer, and most classrooms had more computers than students. All but one classroom had Internet access. All but one site had access to technical assistance for the computer facilities, although the quality of this technical assistance varied from site to site. |
| Recruitment/Participation | Students gave many different reasons for attending the morning summer school program, but they predominantly mentioned poor academic performance (either low grades or low SAT-9 scores), parental requests, and having “nothing better to do.” Despite one more site being open in 2001, fewer children participated in this year than in the year prior (920 vs. 989). Enrollment and attendance issues were found to have impeded implementation, with six of the observed activities reporting such difficulties. Most participants (27 of 33 interviewed in focus groups) were already very comfortable with computers and used them outside of school, with access to computers at home. |
| Staffing/Training | The average student to facilitator ratio across sites was six to one. A number of facilitators lacked knowledge of common computer application packages and only a few considered themselves to be computer experts. All facilitators felt comfortable teaching the programs they were using with their students, however, since they received training in those programs. Students seemed comfortable with facilitators. A few facilitators seemed especially helpful, telling students not to get frustrated, joking with them, patting them on the back, and giving positive feedback. In addition to the facilitators, a number of other adult aides were present for many activities. These aides were found to be of mixed quality. In about half of the cases, the aides were found to be of the same high quality as the facilitators, but in other cases the aides seemed to have little interaction with students and simply occupied computers and looked at papers. One problem may have been the lack of familiarity with computer activities on the part of these aides. |
Summative/Outcome Findings
| Academic | Evaluators found little evidence that the reading and math programs contributed to students' excitement or motivation regarding their academic progress. |