The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.
HFRP
> Out-of-School Time
> OST Database & Bibliography
> Database
> 21st Century Community Learning Centers—Owensboro Kentucky Public Schools
> Evaluation #1 (2000-2001): Formative Evaluation Year 1
A Profile of the Evaluation of 21st Century Community Learning Centers—Owensboro Kentucky Public Schools
Profile Information
- Full Text (HTML)
- How to cite
- About the OST Database
Program Description
| Overview | The Owensboro Public Schools (OPS) 21st Century Community Learning Centers (21st CCLC) project is an after school program for students in select public schools in Owensboro, Kentucky. The program, which consists of five Community Learning Centers that operate year round, began on August 1, 2000 and is scheduled to run through June 2003. The goal of the program is to provide safe, supervised, and fun learning opportunities for students in kindergarten through twelfth grade that serve to increase their academic skills, enhance their ability to interact positively with peers and adults, and reduce behavior problems. |
| Start Date | August 2000 |
| Scope | local |
| Type | after school, summer/vacation, before school, weekend, comprehensive services |
| Location | urban |
| Setting | public school |
| Participants | kindergarten through high school students |
| Number of Sites/Grantees | five schools |
| Number Served | 662 in 2000–2001 and 1,323 in 2001–2002 |
| Components | Program objectives include providing the following: (1) after school programs five days a week for students when school is not in session, (2) extended day/extended year academic assistance for students, (3) day care services when school is not in session, (4) parenting workshops in cooperation with existing community agencies, (5) increased access to school technology and media centers for students and parents, (6) age-appropriate recreation and enrichment activities for students in cooperation with community agencies, (7) research-based substance abuse prevention and intervention activities for students and families, (8) career and employment counseling for students and adults, and (9) expanded learning opportunities for Emotional or Behavioral Disorders (EBD) students for seven weeks when school is not in session. Each school's 21st CCLC center has a full-time Coordinator who handles the program's day-to-day management. Coordinators plan, organize, and help implement program activities, hire and supervise staff, communicate with parents, maintain an evaluation database, and collaborate with other school and community-based programs. A Project Director, who reports to the OPS Director of Personnel and 21st CCLC Advisory Council, oversees and supervises each site and provides an annual update to the U.S. Department of Education. Coordinators combined available resources with student interest data to begin programming. Coordinators also consulted with other resource representatives, such as Family Resource/Youth Service Center (FRYSC) staff, teachers, parents, and other after school programs to determine what services were already available and what services were needed. The initial focus was on developing high-interest programming to attract students, followed by programs related to other areas. Programs are offered in areas of literacy, math, science, humanities, sports/games, technology, community service, culture, health, youth development, and adult services. Examples of programs offered at the Foust site include: community-led activities, Computer Wizards, Enrichment Clubs, Fun N' Fitness, Fun N' Fitness/Dream Team, Girl Scouts, humanities, Phonemic Awareness, prevention activities, and targeted tutoring. Examples of programs offered at the Estes site include: academic enrichment, community service, Is That Your Final Answer?, social/behavioral development, summer activities, and technology. The 5/6 Center site offers learning lab, model builders club, newspaper, and STAR. Examples of programs offered at the Owensboro middle school site include: Boy Scouts, Champions Against Drugs, Choices, craft class, Gents, lock-in at the YMCA, photography, recreation, technical school, tutoring, and We Care. Examples of programs offered at the Owensboro High School site include: arts and crafts, chess, Christmas Cookery, Computer Fun, Cooking with Meschko, drawing and sketching, employment search, knitting circle, Mad Potters, Manuel's Muralists, photography, Raku Fest, Sew Much More, teen support group, and tutoring. Students were able to choose activities. |
| Funding Level | Total funding from the US Department of Education is $1,856,740 and total nonfederal funding is $2,191,742 (August 2000 through June 2003). |
| Funding Sources | U.S. Department of Education |
Evaluation
| Overview | The Owensboro Public School system contracted with REACH of Louisville, Inc. to conduct a systematic assessment of the Owensboro Public School system contracted with REACH of Louisville, Inc. to conduct a systematic assessment of the 21st CCLC program for each year of implementation. The overarching evaluation purposes are to: (1) develop an evaluation framework for organizing and gathering data, (2) create a computer software application to “house” program data and facilitate program evaluation, (3) gather, aggregate, and analyze program data to provide program staff, key stakeholders, community members, and others information about the overall implementation and effectiveness of the program, and (4) make recommendations for ongoing program development and improvement. A logic model outlining the project theory was developed and served as a guide for determining implementation benchmarks and interim outcomes, which were then tracked and analyzed over time to explore program processes and degree of implementation. Several outcomes were developed to evaluate the success of the program. Specifically, by the end of the three years of operation, the following should be achieved:
|
| Evaluator | Robert J. Illback, Psy.D. and Ben W. Birkby, Psy.D. of REACH of Louisville, Inc. |
| Evaluations Profiled | Formative Evaluation of the 21st Century Community Learning Centers, Year 1 Evaluation of the 21st Century Community Learning Centers Program, Year 2 |
| Evaluations Planned | The year 3 evaluation will be summative in nature, focusing on long-term distal program outcomes (e.g., academic achievement, reduction in suspensions). |
| Report Availability | Illback, R. J., & Birkby, B. W. (2001). Formative evaluation of the 21st Century Community Learning Centers, year 1. Louisville, KY: REACH of Louisville. Birkby, B. W., & Illback, R. J. (2002). Evaluation of the 21st Century Community Learning Centers program, year 2. Louisville, KY: REACH of Louisville. |
Contacts
| Evaluation | Robert J. Illback, Psy.D. Benjamin Birkby, Psy.D. REACH of Louisville, Inc. 101 East Kentucky Street Louisville, KY 40203 Tel: 502-585-1911 Fax: 502-589-1592 Email: illbackr@reachoflouisville.com (Robert J. Illback) birkbyb@reachoflouisville.com (Benjamin Birkby) |
| Program | 21st Century Community Learning Centers Owensboro Public Schools 1335 West 11th Street Owensboro, KY 42301 Tel: 270-686-1000 Fax: 270-686-5756 |
| Profile Updated | June 20, 2003 |
Evaluation 1: Formative Evaluation of the 21st Century Community Learning Centers, Year 1
Evaluation Description
| Evaluation Purpose | To answer the following questions about the initial year of program implementation: (1) who is being served by the 21st CCLCs? (2) how are these individuals and families being served? (3) what initial outcomes are associated with the 21st CCLC program? (4) is the 21st CCLC program being implemented as planned? |
| Evaluation Design | Non-Experimental: The research sample included students, parents, teachers, program staff, and key informants involved in the five Community Learning Centers in Owensboro High School, Owensboro Middle School, Owensboro 6/5 Center, Foust Elementary School, and Estes Elementary School. |
| Data Collection Methods | Document Review: Evaluators examined literature and written communication related to the programs during a two-day visit in May 2001. Interviews/Focus Groups: Semi-structured interviews were conducted with parents (n=11), site coordinators (n=5), and key informants (n=21) at each location during a two-day visit in May 2001. Key informants interviewed were a representative sample of individuals who were familiar with the program, and included principals (n=5), teachers involved in the programs (n=5), teachers not involved in the programs (n=6), FRYSC staff (n=4), and the Director of Personnel at the Board of Education. Teachers were selected for interviews by the site coordinators as being representative of those serving in center schools. Parent focus groups were conducted at the five sites, each with two to three parents. Parents selected for focus groups were solicited by site Coordinators and agreed to participate in the evaluation. Interviews and focus groups examined the following aspects of the program: operations, school climate, family-centeredness, barriers and facilitators to implementation, marketing and engagement, mission, accessibility, strengths and weaknesses, community involvement, resource development, market penetration, and participant satisfaction. Observation: Evaluators observed programs over a two-day period in May 2001. Secondary Source/Data Review: Student referral needs data were examined. Referral data included factors that led to students being assessed as needing 21st CCLC services, including such factors as academic underachievement and discipline or behavior problems. Referrals came from parents, teachers, and occasionally from students themselves. Coordinators completed initial referral and tracking forms for each student that included information about the selection and engagement strategy used to attract or recruit the student, barriers or problems encountered in program implementation, reasons for referral of the student to the program, and information about the student's program attendance patterns. School discipline records were also reviewed. Surveys/Questionnaires: Students, teachers, and parents were administered surveys. Teacher surveys were collected during two time periods to assess student progress from pretest to posttest-at three months and again at six months into the program. Teachers were asked to report whether or not students in their classes who were involved in the 21st CCLC had made improvements in school in: submission and completion of homework, participation, volunteering, regular attendance, class attentiveness, behavior, academic performance, class readiness and preparedness, ability to get along well with peers, and number of suspensions and discipline referrals. Teachers were also asked whether they found the program to be helpful for students. Student surveys were given to all students who participated in the program in grade four and above at the end of the year or program activity. They were asked to answer yes or no to five statements relating to whether they found the activity to be enjoyable and of high quality. Surveys were given to all parents of program participants at the completion of the activity. Parents were asked to indicate their level of satisfaction (very satisfied, somewhat satisfied, not satisfied, or very unsatisfied) with the program in which their child participated. They were also asked to respond yes or no to statements about areas in which the program may or may not have helped their child, including improvements in: their child' school behavior, eagerness to go to school, and academic performance; their own (the parents') ability to remain employed and to remain in school; and the family's reduction in stress. At the time of the report, three of the five sites had collected parent, student, and three-month teacher survey data. Two of five sites had gathered six-month teacher survey data. Approximately 18% of the total parent population across sites (n=119) completed satisfaction surveys. A total of 71 students completed surveys, which represented 11% of student enrollment across sites. Seventy students had three-month teacher surveys completed on them and 77 students had six-month teacher surveys completed on them. It is not known if teacher surveys were completed on the same sample of students at three months and six months. Evaluators warn that these preliminary results should be interpreted with caution, since the sample represents only a small portion of the groups examined. |
| Data Collection Timeframe | Data were collected during the 2000–2001 school year. |
Findings:
Formative/Process Findings
| Activity Implementation | Programs focusing on humanities (n=26) were the most common type of program offered. Youth development (n=15), literacy (n=12), and sports (n=12) programs were also frequent. Science (n=9), math (n=8), health (n=7), technology (n=5), adult-related (n=5), and community service (n=1) programs were less common. The number of cultural programs provided is unknown. All programs at Foust were well attended except the targeted tutoring. “Well attended” means that the majority of students were either regular attendees or had completed a program (i.e., those that did not drop out). At Foust, only very small numbers of students dropped out or were irregular attendees. Enrichment clubs, Computer Wizards, community-led activities, and humanities had the highest numbers of students participate, followed closely by Fun 'N Fitness and prevention activities. Girl Scouts, Phonemic Awareness, and Fun 'N Fitness/Dream Team had less student participation. The available data suggests that at Estes academic enrichment activities had the highest numbers of students participate, followed by summer activities, sports, and social/behavioral development. At the 5/6 Center, the available data suggest that the learning lab and newspaper programs had the highest number of students participate. Several programs at Owensboro Middle School were well attended, including the lock-in at the YMCA, Champions Against Drugs, and photography. Boy Scouts, Choices, craft class, Gents, recreation, technical school, tutoring, and We Care were not well attended. At six and nine months these patterns were similar and the number of programs offered decreased over time. Owensboro High School implemented a wide variety of programs throughout the first year. Nine out of the 15 programs were well attended. Arts and crafts, Cooking with Meschko, drawing and sketching, Manuel's Muralists, Sew Much More, and teen support group were not always well attended. These patterns were roughly similar at six and nine months, with fewer programs being offered later in the year. Some parents mentioned a lack of tutoring in a particular subject area as a program implementation barrier. Parents stated that their child's preferences and interests drove their child's participation in 21st CCLC activities. Some Coordinators note that the enthusiasm and eagerness of students wanting to try new things contributed to successful program implementation. |
| Parent/Community Involvement | Community involvement happened in two ways. In the most common way, the Coordinator solicited involvement from an outside community partner (e.g., individual, agency) and invited them to donate their time, materials, or activities to the 21st CCLC. The second way was for Coordinators to bring students to a community partner for a specific activity (e.g., field trips). The degree of community involvement that key informants and Coordinators described was quite variable. According to key informants, some centers incorporated many community undertakings while others were more limited. Many informants noted a lack of community partnerships as an implementation barrier. Successes seemed to occur more often with Coordinators that had existing relationships with community partners. Parent involvement mostly consisted of Coordinators orienting parents to the program, initiating general information phone calls to parents, and having some face-to-face contact when parents pick up their children. Some sites had more involvement in family activities than others, and in exceptional situations parents in specialty professions served as instructors for some activities. One center initiated a relatively successful weekly Family Night. All but one of the parents interviewed had met the Coordinator or other staff in person. Parent involvement was relatively low compared to other components of the Owensboro 21st CCLC. Most parents reported peripheral involvement. All Coordinators described parent/family involvement as somewhat limited and as an area for improvement. In addition, many key informants noted a lack of parental involvement as a program implementation barrier. According to informants, although parents were “kept in the loop” and regularly apprised of their children's progress in the program, actual activity participation by parents was sparse, which they attributed to a lack of adequate family/parent-oriented activities and parental interest or availability. Parents described different levels of program engagement by Coordinators and center staff, and all parents interviewed were satisfied with their levels of interaction, describing the program as “very accessible.” When asked what they thought the purpose of the 21st CCLC was, most parents gave responses that indicated that they have a realistic awareness of the program's purpose. |
| Program Context/Infrastructure | Eight of the nine objectives stated in the original 21st CCLC grant proposal were at least partially implemented after 10 months in operation. The objective of providing expanded learning opportunities for students with emotional and behavioral disabilities was hampered by a lack of EBD staff availability. Many key informants and Coordinators noted problems with access to physical space as a barrier to program implementation. Physical space was a problem in that several of the centers were sharing office, storage, and activity space with other programs already in existence that had first priority on space. Coordinating schedules and establishing a method for reserving space was a challenge for some centers. However, Coordinators rarely mentioned experiencing “turf battles” with other program administrators or staff. Many Coordinators actually mentioned that support from existing school programs contributed to the programs' successful implementation. Many key informants, coordinators, and parents mentioned lack of transportation services as a barrier to program implementation. Transportation was problematic due to a lack of available bus drivers in the district. Plans to provide daily transportation to take students home from after school activities and on weekly field trips had not yet been implemented. Many parents noted that the program was well organized and structured (e.g., clear timelines for activities, division of responsibility, etc.). |
| Program-School Linkages | Key informants indicated that the level of program “presence” and integration that centers achieved in schools ranged from exceptional to fair, with all five centers generally familiar to school staff and students, but to varying degrees across sites. However, they noted that the level of acceptance that centers achieved from school staff and students was favorable across all five centers. According to informants, problems with presence and integration in the schools included slow student participation, lack of initial engagement with high risk students, lack of school staff awareness of what specific activities were available through the 21st CCLCs, confusion distinguishing the roles and responsibilities of the 21st CCLC from other after school activities, and lack of physical space available to 21st CCLC activities. Positive examples of program presence included high levels of collaboration with existing Family Resource and Youth Service Centers, academic enrichment activities being closely tied to classroom curriculum, lack of “turf battles” with other after school programs, successful booths at back-to-school “Ready Fests,” and such high student satisfaction at one center that a waiting list was required. Some key informants identified the lack of integration of academic enrichment activities with classroom curricula as a program implementation barrier. Most 21st CCLC programs appeared to be facilitated by 21st CCLC staff or existing school staff. An area of concern noted by Coordinators was that school staff who also worked in the 21st CCLCs may be “burned out” by the end of the regular school day. Nearly all Coordinators identified having supportive and involved principals and school staff, including teachers, who gradually “bought in” to the program as a factor that contributed to successful implementation of the program. |
| Recruitment/Participation | Twenty-four percent (662) of the total school population across all five sites (2,782) was enrolled in the 21st CCLC program during the first year of implementation. Although this figure falls short of the original program objective of serving 1,230 total students, the program served a substantial number of students for the first year of operation. Student participation varied across sites, ranging from 11% to 47%. Coordinators reported variable levels of student and school staff awareness of the 21st CCLC program across sites, although it was considered to be generally favorable for the first year of program operation. Many students are made aware of the 21st CCLC through announcements, signs, letters sent home, and word of mouth. Elementary students seem to be more aware than middle and high school students. Principals and administrators are generally aware of the program, but may not know its intended purpose or what activities are offered. Many teachers have heard of the program, some refer kids to it, and others have very limited awareness. Some Coordinators described tension with other after school programs that may contribute to varying awareness levels. For instance, confusion may arise over who is in charge after school, what activities go with what programs, and who has priority to use school facilities. Parents reported learning about the program through various methods. Many parents received mailers or other advertisements at the beginning of the school year introducing the 21st CCLC. Others learned of the program through public announcements (e.g., PTA, open houses, back-to-school fests, signs in hallways), and some first learned about the program through their children. One of the implementation barriers noted by key informants and Coordinators was the need to engage students who could benefit from the program, but weren't yet participating. Some parents identified limited student participation early in the year as a program implementation barrier. One key informant mentioned that participation dropped 60% when transportation became unavailable. Students were considered to have continued in the program throughout the year if they attended more than 30 days of activities. During the year of the evaluation, 340 students (51%) attended more than 30 days of activities, which falls slightly short of the goal of having 60% of enrolled students continue in the program throughout the year. As a whole, the 21st CCLC program serves roughly equal numbers of females and males. This pattern was observed across sites as well. Of the 662 students enrolled in the 21st CCLC program during the first year, 450 (68%) were Caucasian, 150 (23%) were African American, 31 (5%) were biracial, two (<1%) were Hispanic/Latino, one (<1%) was American Indian/Alaskan, and one (<1%) was other. Data were unavailable for the remaining 27 students (<1%). While each site varied, overall the 21st CCLC program appears to have strong minority representation. For instance, 21% of all Caucasian students across sites were enrolled in the program, while 32% of all African Americans were enrolled. One exception appears to be Asian students; of the approximately 30 Asian students enrolled at the participating schools (4.5% of the student population), 22 of which were enrolled at one school, none were enrolled in the 21st CCLC program. Twenty-two of the student participants during the first year were ages zero to five, 217 were ages six to ten, 333 were ages eleven to fifteen, and 73 were ages sixteen to twenty. Data were not available for 17 of the students (<1%). Age characteristics tended to correspond to the type of school (e.g., elementary, middle, and high school), and student population at each school, which ranged from a low of 313 students at one school to a high of 1,019 students at another. Middle and high schools tended to have larger student populations, and thus larger numbers of students served, meaning the age of the participants is skewed toward the teen years. The most common referral characteristic among students served at the Foust site is that they tend to be negatively affected by poverty (96%). About two thirds experience boredom or a lack of constructive activities (62%) and about one half have limited access to technology (51%) and to social and recreational interactions (49%). Less often, students experience academic underachievement (44%), have limited access to peer interaction outside of school (43%), are fearful, anxious, or afraid (30%), have repeated discipline or behavior problems (25%), or are quiet or withdrawn (19%). The most common referral characteristic among students served at the Estes site is limited access to technology (83%), followed by poverty (78%), limited access to social and recreational interactions (67%), and academic underachievement (61%). Less often, students enrolled are quiet or withdrawn (43%), have limited peer interaction outside of school (35%), or are experiencing repeated discipline or behavior problems (23%). A small percentage of enrolled students are fearful, anxious, or afraid (13%) or are bored and have a lack of constructive activities (5%). Data on the referral characteristics at the 5/6 were not available. The most common referral characteristics among students served at Owensboro Middle School are academic underachievement (39%) and poverty (38%). About one third of students have limited access to technology (32%), and about one fourth have limited access to social and recreational interactions (24%). Less than 20% are experiencing repeated discipline or behavior problems, are bored or lack constructive activities, or are quiet and withdrawn. A very small percentage (6%) is fearful, anxious, or afraid. At Owensboro High School, the most common referral characteristics among students enrolled in the program are that they are bored, lack constructive activities (87%), and have limited access to social and recreational interactions (84%). About two thirds of students are negatively affected by poverty. Smaller percentages of students are experiencing repeated behavior or discipline problems (39%), and about one fourth experience limited access to technology (26%) or academic underachievement (25%). Less often, students were reported to be quiet and withdrawn (11%), fearful, anxious, or afraid (11%), or have limited peer interaction outside of school (10%). When asked what they perceived to be the main purpose of the 21st CCLCs, key informants gave responses that closely matched the goals of the program as stated in the Procedural Handbook, especially those related to increasing academic and social interaction skills and providing after school programming. These data suggest that initial recruitment strategies employed by Coordinators were largely successful. |
| Satisfaction | The majority of student participants liked the 21st CCLC program (95%), looked forward to coming to the program (94%), were comfortable with program staff (92%), and felt that someone was available to help them after school (91%). Ninety-three percent of parents were very satisfied with the 21st CCLC program, 6% were somewhat satisfied, and 1% were not satisfied. Zero percent indicated that they were very unsatisfied. Many parents mentioned the quality of specific programs offered as a factor that contributed to successful program implementation. |
| Staffing/Training | Site Coordinators' understanding of the purpose of the program was in line with program expectations. Parents were collectively pleased with the interpersonal and professional qualities of Coordinators. Across sites, parents described Coordinators as organized, respectful, responsive, welcoming, dedicated, and open to parent input. Some parents noted that the personal and professional qualities of Coordinators and staff and the level of supervision of children as factors that contributed to successful program implementation. Some Coordinators noted that the training and consultation available to them and the support of the Program Director contributed to successful program implementation. |
Summative/Outcome Findings
| Academic | About two thirds of students (67%) believed they were doing better in school as a result of participating in the program. The majority of parents felt that the 21st CCLC helped their child to do better in school (86%), look forward to school (85%), and improve school behavior (77%). Three months into the program, classroom teachers reported that the majority of program participants improved in: regular attendance (77%), getting along with peers (76%), good behavior (73%), academic performance (73%), homework submission (70%), class readiness and preparedness (69%), homework completion (67%), class participation (67%), attentiveness in class (60%), and volunteering (56%). Teachers reported some reductions in discipline referrals (19%) and suspensions (6%) of participants. Six months into the program, classroom teachers reported that the majority of participants improved in: academic performance (75%), class participation (66%), getting along with peers (61%), class readiness and preparedness (60%), homework submission (57%) and completion (56%), good behavior (55%), attentiveness in class (55%), volunteering (55%), and regular attendance (53%). Teachers reported some reductions in discipline referrals (12%) and suspensions (3%). |
| Family | Eighty-one percent of parents thought that the 21st CCLC program helped their family to have less stress. |
| Workforce Development | Sixty-six percent of parents thought that the 21st CCLC program helped them to remain employed. Thirty-one percent of parents felt that the 21st CLLC program helped them (the parents) stay in school (adult education). |