The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.
HFRP
> Out-of-School Time
> OST Database & Bibliography
> Database
> Maryland After School Community Grant Program
A Profile of the Evaluation of Maryland After School Community Grant Program
Profile Information
- Full Text (HTML)
- Request reprint permission
- How to cite
- About the OST Database
Program Description
| Overview | The goal of the Maryland After School Community Grant Program (MASCGP), which served Maryland youth in grades 4 through 8, was to strengthen youth resiliency and prevent substance abuse, violence, and delinquency among youth by increasing the availability of high quality, structured after school programs. The program's objectives were to increase participants' supervised after school time, academic performance, social skills, attachments to prosocial adults, aversion to substance use and illegal behavior, and involvement and investment in constructive activities. |
| Start Date | 1997 (completed in June 2002) |
| Scope | state |
| Type | after school |
| Location | urban, rural, and suburban |
| Setting | public school, community-based organization, faith-based organization, private facility, recreation center |
| Participants | elementary and middle school students (fourth through eighth graders, some third graders) |
| Number of Sites/Grantees | approximately 40 |
| Number Served | 469 in 2001–2002 |
| Components | MASCGP activities varied by sponsoring agency (county or state government, public schools, youth agencies, church, and private organizations), program site (community center or public school), number of youth served, participation “dosage” (number of hours participants experience the program), and fee schedule. All MASCGP centers were required to include three basic components: academic achievement; social skills; and “bonding” activities, such as sports, arts, crafts, and other recreational activities aimed at retaining participants. |
| Funding Level | $1.1 million for the 2001–2002 school year |
| Funding Sources | Safe and Drug Free Schools Program of the U.S. Department of Justice, Maryland Governor's Office of Crime Control & Prevention, parent fees, other sources |
| Other | MASCGP was incorporated into a larger “Youth Strategies” Activity Initiative that combines eight federal and state funding streams and funds a wide variety of prevention activities to reduce or prevent youth substance abuse and crime. The 5-year consolidated grant initiative offered $15 million over 18 months (January 2002–June 2003) to 24 Maryland communities for prevention, intervention, and aftercare services for youth. |
Evaluation
| Overview | The goals of the evaluation were to examine program implementation and impact on participants' behavior and levels of risk and resiliency. Further studies focused on predictors of dropping out of programs, mechanisms linking participation to reduced delinquency, and relationships between program implementation and youth outcomes. |
| Evaluator | Denise C. Gottfredson, Stephanie A. Weisman, Shannon C. Womer, Melissa Kellstron, Sean Bryner, Amy Kahler, Lee Ann Slocum, Shaoli Lu, and David A. Soulé, University of Maryland, College Park |
| Evaluations Profiled | Maryland After School Community Grant Program: Report on the 1999–2000 School Year Evaluation of the Phase 1 After School Programs Maryland After School Community Grant Program Part 1: Report on the 2000–2001 School Year Evaluation of the Phase 2 After School Programs Maryland After School Community Grant Program Part 1: Report on the 2001–2002 School Year Evaluation of the Phase 3 After School Programs Attrition From After School Programs: Characteristics of Students Who Drop Out Do After School Programs Reduce Delinquency? After-School Programs, Antisocial Behavior, and Positive Youth Development: An Exploration of the Relationship Between Program Implementation and Changes in Youth Behavior |
| Evaluations Planned | none (The “Youth Strategies” Activity Initiative will be evaluated by Dr. Denise C. Gottfredson of the University of Maryland.) |
| Report Availability | Weisman, S. A., Soulé, D. A., & Womer, S. C., under the direction of Gottfredson, D. C. (2001). Maryland After School Community Grant Program: Report on the 1999–2000 school year evaluation of the Phase 1 after school programs. College Park: University of Maryland. Weisman, S. A., Womer, S. C., Lu, S., Soule, D. A., Bryner, S. L., Kahler, A., et al., under the direction of Gottfredson, D. C. (2002). Maryland After School Community Grant Program part 1: Report on the 2000–2001 school year evaluation of the Phase 2 after school programs. College Park: University of Maryland. Weisman, S. A., Womer, S. C., Kellstrom, M. A. Bryner, S., Kahler, A., & Slocum, L. A., under the direction of Gottfredson, D. C. (2003). Maryland After School Community Grant Program part 1: Report on the 2001–2002 school year evaluation of the Phase 3 after school programs. College Park: University of Maryland. Weisman, S. A., & Gottfredson, D. C. (2001). Attrition from after school programs: Characteristics of students who drop out. Prevention Science, 2, 201–205. Gottfredson, D. C. Weisman, S. A., Soulé, D. A., Womer, S. C., & Lu, S. (2004). Do after school programs reduce delinquency? Prevention Science, 5, 253–266. Weisman, S. A., Soule, D. A., Gottfredson, D. C., Lu, S., Kellstrom, M. A., Womer, S. C., & Bryner, S. L. (2005). After-school programs, anti-social behavior, and positive youth development: An exploration of the relationship between program implementation and changes in youth behavior. In J. L. Mahoney, J. S., Eccles & R. W. Larson (Eds.), Organized activities as contexts of development: Extracurricular activities, after-school, and community programs. Hillsdale, NJ: Lawrence Erlbaum Associates. |
Contacts
| Evaluation | Denise C. Gottfredson, Ph.D. Project Director Department of Criminology and Criminal Justice 2220D LeFrak Hall University of Maryland Tel: 301-405-4717 Fax: 301-405-4733 Email: dgottfredson@crim.umd.edu |
| Program | Andrea Alexander Youth Services Division Chief Governor's Office of Crime Control & Prevention 300 E. Joppa Rd, Suite 1105 Baltimore, MD 21286-3016 Phone: 410-321-3521 ext. 356 Fax: 410-321-3116 Email: andrea@goccp-state-md.org |
| Profile Updated | May 19, 2006 |
Evaluation 1: Maryland After School Community Grant Program: Report on the 1999–2000 School Year Evaluation of the Phase 1 After School Programs
Evaluation Description
| Evaluation Purpose | To evaluate the implementation and outcomes of 15 programs. The implementation portion measured standards in four areas: administration, social bonding, academic achievement, and social skills. The outcomes portion focused on programs' impact on participants' attachment to school, rebellious and delinquent behavior, drug use, attitudes about drug use, number of unsupervised hours and hours of parental supervision, commitment to education, and belief in rules. |
| Evaluation Design | Experimental, Quasi-Experimental, and Non-Experimental: Implementation data were collected for all 15 programs to measure programs' performance against implementation standards for attendance and the three program components: bonding, academic achievement, and social skills. The following data were collected on youth at all 15 sites: school records for the treatment and comparison/control groups, and program registration and attendance for the treatment group. Outcome data were collected on 14 programs. These 14 programs were given the choice of using a randomized control or comparison group design. Three programs chose a randomized control group design, in which youth interested in MASCGP were either assigned to participate in the program immediately, were placed in a comparison/control group (ineligible to participate), or were placed on a waiting list during the 1999–2000 school year. Youth on the waiting list were selected randomly to fill vacancies as youth withdrew from the program, while the remainder of this group was added to the comparison/control group. Eleven programs used a comparison group design in which comparison group members were selected from schools or areas outside the population served by their program. Surveys were administered to participants and control/comparison group members in the 14 programs participating in the outcomes study. In total, 97% of both the treatment and control/comparison groups completed pretests, and 89% of the treatments group and 97% of the control/comparison groups completed a posttest. In total, 417 youth were in the treatment group (this group includes those who dropped out of the program, had inconsistent attendance, or participated later in the year after being taken off the waiting list). A total of 408 youth were in the control/comparison group, including waiting list youth who were never recruited to participate. The treatment group was significantly more likely to be non-White (p < .01), to be older (p < .05), and to have spent more hours per week unsupervised after school prior to the program (p < .05) than the comparison group. Demographic differences were controlled for in all analyses. There were no differences between the groups in attrition bias. Evaluators examined the patterns of results to see if they differed by evaluation design (control vs. comparison group); they concluded that programs' use of nonrandomized designs did not bias estimates of the effects of program participation. Thus, data from both types of sites were collapsed together for the remainder of the analysis. |
| Data Collection Methods | Document Review: Logs and forms completed by program staff provided data on daily facets of program administration and components (academic, recreation, and social skills), as well as program implementation standards. Secondary Sources/Data Review: School records (grade point average [GPA] and attendance data), for the 1998–1999 and 1999–2000 school years. Program registration forms (with demographic data) and program attendance data were collected from program sites. Surveys/Questionnaires: The survey, What About You? (Gottfredson, 1991), measured youth's attachment to school, rebellious and delinquent behavior, drug use, attitudes about drug use, peer relationships, parental supervision, commitment to education, belief in rules, unsupervised after school time, and involvement in constructive activities. Tests/Assessments: Two assessments were added to the survey: one was drawn from the Social Skills Rating System (SSRS) Elementary Level Student form (Gresham & Elliott, 1990), which measured youth's social skills, and the other from the Communities That Care Youth Survey (Arthur, Pollard, Hawkins & Catalano, 1995), which measured youth's attachment to prosocial adults. References Arthur, M., Pollard, J., Hawkins, J. D., & Catalano, R. (1995). Communities that care youth survey. Seattle, WA: Developmental Research and Programs. Gottfredson, G. (1991). What about you? Ellicott City, MD: Gottfredson Associates. Gresham, F., & Elliott, S. (1990). Social skills rating system. Circle Pines, MN: American Guidance Service. |
| Data Collection Timeframe | Data were collected in the beginning (pretest) and end (posttest) of the 1999–2000 school year. |
Findings:
Formative/Process Findings
| Activity Implementation | Bonding activities were provided on 85% of program days across all sites. Only one program offered social bonding activities every single day during the program year, and two offered them on approximately half of all program days. The most popular bonding activity was structured recreation/sports. Eighty-six percent of the programs offered at least one field trip; 60% of the programs met the standards they set for number of field trips completed during the school year. All but one program completed academic assessments for over 85% of their youth. However, several programs fell short of the standards they set for conducting academic assessments and plans. Seventy-three percent of the programs met the standard requiring every youth to participate in 1.5 hours of structured educational activities per week. Many programs offered several hours of academic activities each week, but youth dropout and nonattendance resulted in youth receiving only an average of 1.6 hours of academic activities per week. Eighty percent of programs met the standard requiring every youth to participate in social skills activities for at least 1.5 hours per week. On average, youth received 2.5 hours of structured social skills activities per week. |
| Parent/Community Involvement | Two thirds of programs offered at least one family activity during the school year, but only three programs met their standards for the number of family activities. The main problem that programs faced in conducting youth's academic assessments was that youth/parent input was rarely sought. |
| Program Context/Infrastructure | After 3 years of operation, on average, programs met only 53% of implementation standards, ranging from 30% to 63% among programs. Although some programs did not meet the administrative standard for number of program days offered throughout the school year, on average the programs met this standard. Analysis indicated that an increase in youth's social bonding was found to be significantly related to a decrease in delinquency (p < .01) but not drug use. Stronger attitudes against illegal activities and increases in social skills among youth were found to be related to significant decreases in delinquency (p < .01) and drug use (p < .05). Increased negative peer influence was related to significant increases in delinquency (p < .01) and drug use (p < .05) among youth. Increasing youth's hours supervised and their involvement in solitary constructive activities were not found to translate into significant declines in delinquency or drug use. Changes in youth's GPA averages did not correlate significantly with changes in delinquency and drug use. |
| Recruitment/Participation | Only 73% of programs were able to maintain a core group of 20 youth throughout the 1999–2000 school year. Every program recruited 20 or more participants, but due to attrition, four programs failed to serve 20 youth during each month of the school year. On average, programs kept 18 youth continually enrolled for the entire year. Poor attendance was a major concern in the programs. Over the 9 months of the program year, the core youth for all programs combined were in attendance 74% of the days. Only 53% of youth met the standard of attending 80% of the sessions. Across programs, the percentage of youth that met the 80% standard ranged from 18% to 93%. Only 4 of the 15 programs met the standard requiring 100% of their youth to attend 80% of the sessions at least 1 month during the school year. On average, elementary school youth attended 82% of program days for a total of 71 days, and middle school youth attended 70% of program days, for a total of 49 days. Only one program investigated every instance of nonattendance, and overall only 37% of nonattendance was investigated. Programs generally failed to call parents of youth who were absent from the program without an excuse. |
| Staffing and Training | Most staff were deemed competent in the areas in which they delivered service based on records of prior experience. As a whole, program staff did not receive the required number of staff development sessions during the school year. Interactions with projects suggest that staff turnover presented a major problem. Each program set individual standards for ratios of staff to youth in academic activities. The average ratio of staff to youth in academic activities was 1:4. |
Summative/Outcome Findings
| Academic | Based on the incomplete academic data available, programs did not seem to have an effect, either positive or negative, on academic achievement. However, shorter programs (those meeting less than 9.5 hours a week) appeared to have more significant negative effects on academic performance than longer ones (p < .05). |
| Prevention | Overall, program participation was not significantly associated with delinquency and drug use, but several individual programs had significant results in these areas; participation in one program was associated with reduced delinquency (p < .05), and participation in two of the programs were associated with reduced drug use (p < .05 for each). Program participation was associated with reductions in positive attitudes toward illegal activities. Most programs showed moderate to large effect sizes in this area, although only one was statistically significant (p < .05). Two programs had moderate, though not significant, negative effect sizes—in other words, there was an association between program participation and increased positive attitudes toward illegal activities. Four programs demonstrated moderate to large effect sizes for decreasing negative peer influence, although only one was statistically significant (p < .05). Programs that had a high emphasis on social skills and a low emphasis on academics had a significant effect on reducing delinquency (p < .05), drug use (p < .01), and positive attitudes toward illegal activity (p < .01) compared to other programs. These programs were also more effective at decreasing negative peer influence and improving social skills than other programs. Participation in the programs seemed to reduce delinquent behavior for middle school youth, but not for elementary school youth. For older youth, participation in the programs decreased delinquent behavior primarily by decreasing peer drug models and increasing intentions not to use drugs (p < .01 for each). Programs serving primarily low-risk youth had a significant effect on reducing delinquency (p < .05) whereas programs serving high-risk youth did not. |
| Youth Development | Participation in programs significantly decreased the number of hours participants were unsupervised and increased their involvement in constructive activities (p < .05). All but three programs showed positive effect sizes on youth's social skills, although none were significant. |
Evaluation 2: Maryland After School Community Grant Program Part 1: Report on the 2000–2001 School Year Evaluation of the Phase 2 After School Programs
Evaluation Description
| Evaluation Purpose | To provide information on implementation of the 21 programs during the 2000–2001 year, to identify the major risk and protective factors associated with delinquency and drug use, and to identify program characteristics related to changes in these risk and protective factors and in delinquency and drug use. |
| Evaluation Design | Quasi-Experimental and Non-Experimental: Implementation data were collected from program staff records and observations at all 21 sites. Pretest/posttest attendance data were received for 53% of pretested youth, while pretest/posttest grade point average (GPA) data were received for 60% of pretested youth. Each program director was interviewed and 8 site visits were conducted at each program. Program outcomes were assessed from data collected from program participants at the beginning and end of the program year. A total of 486 of the 625 youth who had ever registered for these programs during 2000–2001 were administered the pretest survey, while 402 (83%) of these youth were administered the posttest survey. Stayers (those who took both pretest and posttest surveys) and dropouts (those who did not take the posttest) did not appear to be significantly different from one another on pretest measures, with the exception that dropouts were involved in significantly fewer constructive activities than stayers (p < .01). The final sample (those completing both the pretest and posttest) was 46% male and 82% non-White. The average age of the sample was 10.45 years. |
| Data Collection Methods | Document Review: Logs and forms completed by program staff provided data on daily facets of program administration and components (academic, recreation, and social skills). Interviews/Focus Groups: Program directors were interviewed at the end of the year about the programs' populations served, staff, and components. Observation: Site visits examined program characteristics, including the structure of program components, youth engagement levels, social climate, and behavior management. Secondary Sources/Data Review: Students' school records (grades and attendance data) were collected from the prior (1999–2000) and present (2000–2001) school years. Surveys/Questionnaires: Youth completed the What About You? survey (Gottfredson, 1991) at the beginning and end of the school year. The survey measures youth' attachment to school, rebellious and delinquent behavior, drug use, attitudes about drug use, peer relationships, parental supervision, commitment to education, belief in rules, and unsupervised after school time and involvement in constructive activities. Tests/Assessments: Additional items were added to the What About You? survey from the Social Skills Rating System (SSRS) Elementary Level Student form (Gresham & Elliott, 1990), which measured youth's social skills, and from the Communities That Care Youth Survey (Arthur, Pollard, Hawkins & Catalano, 1995), which measured youths' attachment to prosocial adults. References Arthur, M., Pollard, J., Hawkins, J. D., & Catalano, R. (1995). Communities that care youth survey. Seattle, WA: Developmental Research and Programs. Gottfredson, G. (1991). What about you? Ellicott City, MD: Gottfredson Associates. Gresham, F., & Elliott, S. (1990). Social skills rating system. Circle Pines, MN: American Guidance Service. |
| Data Collection Timeframe | Data were collected during the 2000–2001 school year. |
Findings:
Formative/Process Findings
| Activity Implementation | According to program staff logs, all programs provided at least one fun, constructive, interesting activity. On average, such activities were provided 95% of days, with three out of five programs providing these activities every program day. According to program observations, only 19% of programs provided recreational activities with a high level of structure. (Thirty-eight percent provided medium-structure recreational activities and 43% provided low structure.) Nearly all programs offered arts and crafts and performing arts activities, while about half offered wilderness or challenge activities and nearly one third offered an entrepreneurship activity. Two thirds of programs met the administrative standard of offering group field trips outside the after school program setting throughout the program year. In total, 169 field trips were taken during the year. Program director surveys indicated that educational activities used only 49% of best practices specified by the evaluators as such prior to the evaluation. According to survey responses, just over a quarter of programs (27%) used academic assistance programs developed by researchers. In 90% of programs, every youth participated in structured educational activities at least 1.5 hours per week, with an average of 2.6 hours of such instruction per week. All programs provided social skills development training through curricula or activities designed to teach, model, or reinforce social skills and character. Observations revealed that few programs met administrative standards related to the specified content of social skills development training. Program director surveys indicated that social skills training used only 37% of “best practices.” Forty-two percent of programs met the administrative standard of having every youth participate in an average of 1.5 hours per week of structured social skill and character development activities. Program director surveys and interviews revealed that social skills training components were much more structured than the academic assistance or recreation components. Based on survey responses, 53% of programs used social skills training curricula developed by researchers. |
| Costs/Revenues | Seventy-one percent of programs charged no fee for after school services. Of programs that charged fees, fees ranged from $5 per year to $50 per month. One third of program directors reported in interviews that program funding was one of their program's primary difficulties. |
| Parent/Community Involvement | Seventeen programs (81%) provided at least one activity for families, according to program staff logs. Ten percent of programs met the administrative standard of contacting a parent/guardian regarding youth's progress throughout the program year. In total, there were 34 times during the program year that programs contacted a parent/guardian of a MASCGP youth. One program met the administrative standard of seeking parent/guardian input for academic assessments. Overall, parent/guardian input was sought in 18% of assessments. Of the individual academic plans that were developed, 48% were shared with the youth and his/her parent or guardian (as opposed to the administrative standard of 100%). One third of program directors reported in interviews that parent cooperation, involvement, and understanding were primary difficulties encountered in the program. |
| Program–School Linkages | Four programs (19%) met the administrative standard of conducting an assessment of all youth in their program. In total, academic assessments were made for 66% of youth. Three programs (14%) met the administrative standard of having these academic assessments done by an educational professional in the youth's school or by a qualified after school staff member. In total, a qualified professional conducted 63% of academic assessments. Four programs (19%) met the administrative standard of developing individual academic plans for all youth based on the assessment of educational need. In total, academic plans were made for 66% of youth. Of these plans, 79% included general and subject-specific learning objectives (as opposed to the administrative standard of 100%). All programs provided structured educational activities based on youth's academic plans. |
| Program Context/Infrastructure | Overall, programs met 47% of their implementation standards, ranging from 25% to 71%. Forty-eight percent of the programs operated 4 days per week, 28% operated 5 days per week, and 24% operated 3 days per week. The majority of programs operated between 2.5 and 3 hours per week. Only a quarter of programs appeared to use a discipline management system consistently and to reward good behavior, while just over half of programs (57%) had clear expectations for youth behavior and demonstrated consistency in managing youth. Observations suggested that only 29% of the programs were effective at monitoring youth. Twenty-nine percent of elementary school programs, 60% of middle school programs, and 67% of mixed elementary/middle school programs met the administrative standard of offering after school activities for at least a specified number of days (at least 90 days for middle school youth and at least 120 days for elementary school youth). The average number of days of offered activities was 105 at middle school programs, 117 for elementary school programs, and 135 for mixed programs. |
| Recruitment/Participation | All but one site served both genders, with the one remaining site serving only boys. Seventy-one percent of programs served youth with behavior problems, nearly half served youth at risk of being expelled from school, and nearly a fifth (19%) served gang members. About half of the programs met the administrative standard of having a core group of 20 youth who participated regularly. Programs served an average of 18 youth regularly. Ten percent of programs met the administrative standard of having all youth attend 80% of all sessions. Programs had an average of 68% of participants attend 80% of sessions. Throughout the school year, 29% of participants dropped out of the program. Those who dropped out attended 53 program days on average, as compared to 112 days for stayers. More program dropouts than stayers tended to be white (p < .01). |
| Staffing/Training | The majority of programs (62%) experienced staff turnover during the year (average rate of 6%). Nine programs (43%) reported difficulty recruiting and/or retaining qualified staff. Of these nine programs, eight mentioned a lack of qualified applicants as a reason for this difficulty, four mentioned the time/hours of the program, one mentioned childcare licensure requirements, and one mentioned transportation issues. In 95% of observations, adults involved in the program appeared to be caring, genuinely interested in the youth, and accessible to the youth. Adult–youth relations were rated as good or exceptional in 71% of observations. Relations among staff were rated as good or exceptional in 76% of observations. Almost 60% of programs met the administrative standard of having all of their direct service staff judged as competent in the areas they were delivering services, as indicated by prior training. On average, 87% of staff were judged to have enough relevant experience in this regard. Six programs (29%) met the administrative standard of having 90% of their direct service staff participate in four staff development sessions per year in their service delivery area. In total, 42% of staff participated in at least four staff development sessions. The average ratio of trained academic assistance staff to youth was 1:6 across programs. |
Summative/Outcome Findings
| Academic | Participants in programs offering more than 30 lessons in social skills had significantly higher gains in GPA than did participants in programs with fewer such lessons (p < .01). Participants in programs offering more than 1 hour per week of social skills lessons had significantly higher gains in school attendance than did participants in programs with lower levels of such lessons (p < .01). Participants in elementary school programs that had more than 3 hours of academics per week had significantly larger gains in the percentage of days they attended school than participants in programs with lower levels of academics (p < .05). Participants in programs offering more than 2.5 hours of recreational activities per week had significant decreases in GPA compared to participants who had fewer hours of such activities (p < .01). |
| Prevention | Programs rated by observers as having more efficient procedures had greater reductions in rebellious behavior than programs with less efficient procedures (p < .05). Programs rated by observers as having higher levels of overall structure had greater reductions in rebellious behavior than programs with low or medium structure (p < .05). Programs rated by observers as having higher levels of behavior management had greater reductions in rebellious behavior than programs with lower levels of behavior management (p < .01). Programs rated by observers as having higher levels of overall structure had greater gains in youth's intentions not to use drugs than programs with low or medium structure (p < .05). Participants in elementary school programs that had more than 2.33 hours of academics per week had significantly greater decreases in last year variety of drug use (a scale that counts the number of different drugs they admitted using) than participants in programs with lower levels of academics per week (p < .01). For middle school participants, the dedication of lower levels of program time to recreation is associated with lower delinquent behavior, while higher percentages of program time concentrated on recreation is related to more delinquent behavior (p < .05). |
Evaluation 3: Maryland After School Community Grant Program Part 1: Report on the 2001–2002 School Year Evaluation of the Phase 3 After School Programs
Evaluation Description
| Evaluation Purpose | To examine MASCGP's effectiveness at influencing how youth spend their time and how much time youth spend in supervised settings, and in improving youth outcomes. Also, to examine what program characteristics are related to program effectiveness. |
| Evaluation Design | Quasi-Experimental and Non-Experimental: Implementation data were collected from all sites. Evaluators conducted 5 site visits at each program. In addition, pretest and posttest data were collected for MASCGP participants and a comparison group of nonparticipants. The comparison group was comprised of youth recruited by programs who were demographically similar to program participants. Data were collected from all MASCGP participants who received parental consent for involvement in the evaluation. In total, 594 treatment and 476 comparison group youth were included in the pretest sample, with 78% of these youth completing a posttest survey at the end of the year. Attrition rates were similar across both groups. Pretest/posttest school attendance data were received for 82% of pretested youth, while pretest/posttest grade point average (GPA) data were received for 88% of pretested youth. Of 13 measures on the pretests, two showed differential attrition between the treatment and comparison group. First, dropouts in the comparison group were found to have lower levels of commitment to education than dropouts in the treatment group, who were found to be no different in their level of commitment than treatment stayers (which would lead to a bias in favor of the comparison group). Second, dropouts in the treatment group scored lower on belief in conventional rules than drop outs in the comparison group (which would lead to a bias in favor of the treatment group). Because these two sources of bias were in opposite directions and because no differential attrition was found for the other 11 measures, the evaluators concluded that differential attrition was not a major biasing factor. Analysis revealed a number of significant differences between the treatment and control groups. Treatment youth were more likely to be non-White (p < .01) and to spend time in self-care after school (p < .10); they were less likely to be involved in constructive activities (p < .05), engage in rebellious behavior (p < .05), and to have been arrested (p < .05); treatment youth also had lower GPAs (p < .05) than comparison youth. For middle school youth only, treatment group youth were less likely to have peer drug models (p < .05) and have engaged in delinquent behavior (p < .01) than comparison youth. Overall, the evaluators concluded that the comparison youth were more at risk than treatment youth at pretest. Pretest differences were controlled for in all analyses. The evaluators also cautioned that 50% of the comparison group reported participating in an after school program during the school year, indicating that nonprogram participants were substantially involved in alternative after school activities. The treatment group reported spending an average of 3.5 days per week in after school programs, as compared to 1 day per week for comparison youth (p < .01). |
| Data Collection Methods | Document Review: Logs and forms completed by program staff provided data on daily facets of program administration and components (academic, recreation, and social skills). Observation: Site visits measured program characteristics, including the structure of program components, levels of youth engagement, social climate, and behavior management. Secondary Sources/Data Review: Youth's school records (grade point average [GPA] and attendance data) were collected from the prior (2000–2001) and present (2001–2002) school years. Surveys/Questionnaires: Youth completed the What About You? survey (Gottfredson, 1991) at the beginning and end of the school year. The survey measures: youth's attachment to school, rebellious and delinquent behavior, drug use, attitudes about drug use, peer relationships, parental supervision, commitment to education, belief in rules, and unsupervised after school time and involvement in constructive activities. Tests/Assessments: Additional items were added to the What About You? survey from the Social Skills Rating System (SSRS) Elementary Level Student form (Gresham & Elliott, 1990), which measured youth's social skills, and from the Communities That Care Youth Survey (Arthur, Pollard, Hawkins & Catalano, 1995), which measured youth's attachment to prosocial adults. References Arthur, M., Pollard, J., Hawkins, J. D., & Catalano, R. (1995). Communities that care youth survey. Seattle, WA: Developmental Research and Programs. Gottfredson, G. (1991). What about you? Ellicott City, MD: Gottfredson Associates. Gresham, F., & Elliott, S. (1990). Social skills rating system. Circle Pines, MN: American Guidance Service. |
| Data Collection Timeframe | Data were collected during the 2001–2002 school year. |
Findings:
Formative/Process Findings
| Activity Implementation | On average, youth received 3.1 hours of academic assistance activities per week, such as tutoring, homework assistance, and computer skills training. On average, youth received 2.1 hours of structured social skills per week. The percentage of programs failing to provide the minimum social skills instruction hours of 1.5 per week fell from 58% in 2000–2001 to 27% in 2001–2002. Observations revealed that the delivery of social skills components was fairly well structured across the programs (as measured by the presence of a lesson plan, specific order of content delivery, and advance planning). |
| Program Context/Infrastructure | On average, the programs met 56% of overall implementation standards (an improvement over the 47% that were met the previous year). This ranged from one program site that met only 29% of the standards to another that met 87%. In small programs (under 30 youth), there were no significant differences between treatment and comparison group youth in regards to whether youth spent time unsupervised after school. In large programs, however, treatment group youth were significantly more likely than comparison group youth to spend time unsupervised after school (p < .01). Observations indicated that programs used 88% of positive youth discipline management practices specified by evaluators prior to the evaluation, such as clear and communicated expectations for behavior, consistency in managing behavior, effective monitoring of behavior, and immediate responses to misbehavior. Overall, 18 of 22 programs operated the required 90 days during the school year. The programs averaged 110 days of operation over the school year. |
| Recruitment/Participation | Every program recruited at least 20 participants, but four programs failed to serve 20 youth each month. Only two programs met the attendance standard of having all youth attend 80% of all program days. The percentages of youth meeting the 80% standard ranged from 38% to 100% across sites. Overall, 17% of youth dropped out of the program between November and March. Only two programs failed to meet the standard that no more than 33% of youth would withdraw during this period. Approximately 25% of youth said they usually went to the program fewer days than they are expected to go. |
| Staffing/Training | Only one program met the standard of having all direct-service staff attend youth development training. However, many sites had staff who worked in the educational field and may have had previous training in youth development. Twenty of the 22 programs met the staff-to-youth ratio standard of 1:10. The average ratio was 1:5. |
Summative/Outcome Findings
| Academic | While the comparison group began the school year with a significantly higher GPA than the treatment group (p < .05) and both groups declined slightly in GPA from the 2000–2001 school year to the 2001–2002 school year, the treatment group experienced a significantly greater decline in GPA than the comparison group (p < .01). At the posttest, more comparison than treatment youth reported often paying attention in class (71% vs. 57%, p < .01). This gap was smaller in programs that served up to 30 youth as compared to programs that served more than 30 youth (p < .01). Despite similar levels of commitment to education at the pretest, the comparison group increased in commitment while the treatment group decreased in commitment over the course of the year, resulting in significant differences at the posttest (p < .01). At the posttest, more comparison than treatment youth reported turning homework in on time (61% vs. 53%, p < .05). |
| Prevention | While the comparison group reported significantly more rebellious behavior at pretest (p < .05), the treatment group increased in their rebellious behavior at a rate that significantly surpassed the comparison group as measured at posttest (p < .01). While the treatment group reported higher (although not significantly) intentions to use drugs at pretest than the comparison group, the treatment group had significantly lower intentions to use drugs than the comparison group at posttest (p < .10). At the posttest, more comparison than treatment youth reported that they would try marijuana or other drugs (18% vs.12%, p < .05). In programs with more structured social skills activities, significantly more comparison than treatment youth reported intentions to smoke cigarettes at the posttest (19% vs. 12%, p < .05). There were no significant differences between these groups on this measure in programs with less structured social skills activities. Treatment youth in large programs (over 30 youth) reported that they had significantly more friends who sold drugs in the last year than comparison youth (p < .01), while there were no significant differences for the two groups in small programs (up to 30 youth). There were no significant differences between treatment and comparison groups for actual drug use overall and frequency of drug use in the last month, nor differences in the last year for variety of drug use, use of cigarettes, alcohol, or marijuana, or of selling marijuana. |
| Youth Development | While both treatment and comparison group youth reported similar levels of social skills on the pretest, significantly more comparison than treatment youth reported that they often followed their teacher's directions at the time of the posttest (70% vs. 55%, p < .01). There were no significant differences between the treatment and comparison groups for: attachment to prosocial adults, involvement in constructive activities, belief in rules, or positive peer associations. This lack of significant youth outcomes held regardless of whether programs were categorized as using efficient procedures, high level of behavior management, high levels of overall structure, more time in academics, or higher levels of youth attendance. |
Evaluation 4: Attrition from After School Programs: Characteristics of Students Who Drop Out
Evaluation Description
| Evaluation Purpose | To examine differences between youth who stayed in MASCGP programs and youth who withdrew prior to the end of the school year. |
| Evaluation Design | Non-Experimental: Data were collected from all 234 youth in 8 MASCGP programs (19–47 youth per program). On average, youth were 11.5 years old. Nearly two thirds (61%) were male; 80% were non-White, with 96% of non-White participants being Black. According to parent reports on registration forms, 40% of youth were classified as latchkey (i.e., they would have no supervision after school if not for the program). The following data were collected on all registered youth at the 8 programs: demographic and program attendance (100% of youth, n = 234), school attendance (62% of youth, n = 146), school class grades (65% of youth, n = 153), census data (91% of youth addresses, n = 214), and youth surveys (100% of youth, n = 234). Comparisons were made between youth who attended (stayers, n = 157) and youth who withdrew from the program (dropouts, n = 77). Comparisons were also made between youth with varying levels of program attendance. Comparisons were made on 12 outcomes, ranging from school attendance and grades to drug use, delinquency, and neighborhood disadvantage. Of the drop outs, 63 completed short withdrawal interviews. |
| Data Collection Methods | Interviews/Focus Groups: Withdrawal interviews asked youth why they left the program. Secondary Sources/Data Review: Demographic and program attendance data on youth participants were provided by all 8 programs. School attendance and grade data were obtained from participants' school records. Block group data from the 1990 census were used to compute a measure of neighborhood social disorganization. This measure was made up of indicators for the proportion of youth in the block group displaying the following 12 family characteristics: welfare receipt, poverty, divorce, female unemployment, male unemployment, males not in the labor force, females not in the labor force, low professional/managerial employment, low family income, low educational attainment, low-level nonpublic school enrollment, and female-headed households. Surveys/Questionnaires: A survey, What About You? (Gottfredson, 1991), was administered to youth at the beginning of the program year. The survey provided eight indicators of youth at-risk status: rebellious behavior, delinquent behavior, last year variety of drug use (i.e., usage of more types of drugs), last month frequency of drug use, attachment to school, commitment to education, peer drug models (i.e., how many of their friends use drugs), and parental supervision. In addition, a measure of youth's social skills was embedded in the survey. Tests/Assessments: The social skills assessment in the survey was drawn from the Social Skills Rating System (SSRS) Elementary Level Student form (Gresham & Elliott, 1990). References Gottfredson, G. (1991). What about you? Ellicott City, MD: Gottfredson Associates. Gresham, F., & Elliott, S. (1990). Social skills rating system. Circle Pines, MN: American Guidance Service. |
| Data Collection Timeframe | Data were collected in the 1998–1999 program year. |
Findings:
Formative/Process Findings
| Recruitment/Participation | Dropout rates ranged from 11% to 53% per program. Eleven out of the 12 comparisons between dropouts and stayers indicated that higher risk youth were more likely to drop out, though only three of these relationships were significant. More specifically, dropouts were more likely to have more peer drug models (p < .01), higher school absence rates (p < .01), and come from more socially disorganized neighborhoods (p < .05) than stayers. Youth who reported more parental supervision had significantly higher levels of program attendance than youth who reported less parental supervision (p < .05). Youth with more peer drug models (p < .01), days absent from school (p < .01), frequent drug use in the last month (p < .05), and variety of drug use in the last year (p < .05) had lower program attendance than youth less at-risk according to these characteristics. Youth from more highly disorganized neighborhoods had lower levels of program attendance than youth from less highly disorganized neighborhoods (p < .01). Withdrawal interviews indicated that the main reasons youth withdrew were because they found the program boring (33%), relocated (19%), and had transportation problems (14%). Transportation was mainly an issue in programs that did not provide transportation home after the program. |
Evaluation 5: Do After School Programs Reduce Deliquency?
Evaluation Description
| Evaluation Purpose | To examine the effects of MASCGP on delinquent behavior and the mechanisms through which such programs may affect delinquent behavior. |
| Evaluation Design | Experimental, Quasi-Experimental, and Non-Experimental: Outcome data were collected on 14 programs, which were given the choice of using a randomized control or comparison group design. Three programs chose a randomized control group design, in which youth interested in MASCGP were either assigned to participate in the program immediately, were placed in a comparison/control group (ineligible to participate), or were placed on a waiting list during the 1999–2000 school year. Youth on the waiting list were selected randomly to fill vacancies as youth withdrew from the program, while the remainder of this group was added to the comparison/control group. Eleven programs selected comparison group members from schools or areas outside the population served by their program. Evaluators examined the results to see if they differed by evaluation design; they concluded that programs' use of nonrandomized designs did not bias estimates of the effects of program participation. Thus, data from both types of sites were collapsed together in the analysis. A survey was administered to all participants and control/comparison youth. In total, 97% of both the program and control/comparison groups completed pretests, and 89% of the program group and 87% of the control/comparison group completed a posttest. The main outcome analyzed was a “delinquent behavior” variable, which included delinquent behavior, rebellious behavior, and substance use. Results are presented separately for younger (grades 4 and 5) and older (grades 6 through 8) youth. In total, 417 youth were in the treatment group (which included those who dropped out of the program, had inconsistent attendance, or participated later in the year after being taken off the waiting list). A total of 408 youth were in the control/comparison group, including waiting list youth who were never recruited to participate. Younger participants and nonparticipants were very similar at pretest. Among older youth, comparison youth reported lower levels of rebellious behavior (p < .01), higher levels of social skills (p < .05), and fewer unsupervised hours per week (p < .01) than did participating youth. All analyses statistically controlled for demographic differences. There were no differences between the groups in attrition bias. To examine possible mechanisms through which participation produced estimated effects on reduced delinquency, evaluators estimated a “path model,” testing links between program participation and hypothesized mechanisms (i.e., involvement in constructive activities, intentions not to use drugs, social skills, positive peer associations, and unsupervised time) and between these hypothesized mechanisms and delinquency outcomes. |
| Data Collection Methods | Document Review: Logs and forms completed by program staff provided data on daily facets of program administration and components (e.g., number of hours spent on social skills and character development training). Surveys/Questionnaires: The youth survey, What About You? (Gottfredson, 1991), measured youth's attachment to school, rebellious and delinquent behavior, drug use, attitudes about drug use, peer relationships, parental supervision, commitment to education, belief in rules, unsupervised after school time, and involvement in constructive activities. In addition, a measure of youth's social skills was embedded in the survey. Tests/Assessments: The social skills assessment in the survey was drawn from the Social Skills Rating System (SSRS) Elementary Level Student form (Gresham & Elliott, 1990). References Gottfredson, G. (1991). What about you? Ellicott City, MD: Gottfredson Associates. Gresham, F., & Elliott, S. (1990). Social skills rating system. Circle Pines, MN: American Guidance Service. |
| Data Collection Timeframe | Data were collected in the 1999–2000 program year. |
Findings:
Formative/Process Findings
| Program Context/Infrastructure | Middle school programs were expected to meet 3 days per week for a total of 90 program days, and elementary school programs were expected to meet 4 days per week for a total of 120 program days. Middle school programs in general exceeded this standard by meeting for 116 days, while elementary school programs fell short, meeting 107 days on average. Elementary school participants were found to have received approximately 30 fewer hours of social skills and character development lessons over the course of the program than middle school participants. In the typical program, students received approximately 1.7 hours of educational services per week and 1.8 (elementary) or 2.4 (middle) hours of social skills or character development training. |
| Recruitment/Participation | On average, youth attended 71% (for middle school youth) and 80% (for elementary school youth) of available program days. The mean number of days actually attended was 64 (middle) and 68 (elementary). Eighty-four percent of elementary and 77% of middle school youth participated for more than 30 days. |
Summative/Outcome Findings
| Prevention | Participation was significantly related to reduced delinquent behavior among older youth (p < .01). No significant relationship was found among younger youth. In the path model, the intervening (or mechanism) variables partially explained the estimated effect of after school participation on reduced delinquency, but a direct relationship remained, implying that the mediating variables were not sufficient to explain all of the estimated program effect on delinquent behavior. Taken together, the path model revealed that program participation was significantly related to reduced delinquency through two of the estimated pathways: increased intentions not to use drugs and positive peer associations. In particular, participation was related to both increases in intentions not to use drugs (p < .05) and increases in positive peer associations (p = .10). Both of these factors were then significantly related to reductions in delinquency (p < .01). Among the other hypothesized mechanisms, increased social skills were significantly related to decreased delinquency, but program participation was unrelated to changes in social skills. Program participation was related to decreased hours per week in self-care, but this relationship did not reach significance. Hours per week in self-care were not in turn associated with changes in delinquency. Program participation was positively related to involvement in constructive activities (p < .01), but involvement in constructive activities was positively related to delinquent behavior (p < .05). The effects of participation on reduced delinquency were generally larger in the five middle school programs that had a high level of emphasis on social skills and character development then in the three middle school programs with a low emphasis on social skills and character development (p < .10). In these programs, relationships between participation and delinquency continued to be mediated by attitudes pertaining to substance use and by positive peer associations. However, the positive relationship between involvement in constructive activities and reduced delinquency was not found in the programs with a high level of emphasis on social skills and character development. |
Evaluation 6: After-School Programs, Antisocial Behavior, and Positive Youth Development: An Exploration of the Relationship Between Program Implementation and Changes in Youth Behavior
Evaluation Description
| Evaluation Purpose | To assess the relationship between program implementation and youth outcomes. |
| Evaluation Design | Quasi-Experimental and Non-Experimental: Data were collected from 21 program sites. Implementation data were collected from program staff records, observations (8 site visits at each program site), and program director interviews at all 21 sites. Program outcomes were assessed from participant surveys and school records collected at the beginning and end of the program year. A total of 486 of the 625 youth who had ever registered for these programs during 2000–2001 completed pretest surveys, and 402 (83%) of these youth also completed posttest surveys. The sample was 46% male and 82% non-White, with an average age of 10.5 years. Pretest/posttest school attendance data were collected for 53% of pretested youth, and pretest/posttest grades data were received for 60% of pretested youth. |
| Data Collection Methods | Document Review: Logs and forms completed by program staff provided data on daily facets of program administration and components (amount of time spent in academic, recreation, social skills, and youth-staff ratio during the academics component). Interviews/Focus Groups: Program directors were interviewed at the end of the year about the programs' populations served, staff, and components. Observation: Site visits examined program characteristics, including the structure of program components, youth engagement levels, social climate, and behavior management. Secondary Source/Data Review: Youth's school records (grade point averages [GPA] and attendance data) were collected for the 1999–2000 and 2000–2001 school years. Surveys/Questionnaires: The youth survey, What About You? (Gottfredson, 1991), measured youth's attachment to school, rebellious and delinquent behavior, drug use, attitudes about drug use, peer relationships, parental supervision, commitment to education, belief in rules, and unsupervised after school time and involvement in constructive activities. Tests/Assessments: Additional items were added to the What About You? survey from the Social Skills Rating System (SSRS) Elementary Level Student form (Gresham & Elliott, 1990), which measured youth's social skills, and from the Communities That Care Youth Survey (Arthur, Pollard, Hawkins & Catalano, 1995), which measured youth's attachment to prosocial adults. References Arthur, M., Pollard, J., Hawkins, J. D., & Catalano, R. (1995). Communities that care youth survey. Seattle, WA: Developmental Research and Programs. Gottfredson, G. (1991). What about you? Ellicott City, MD: Gottfredson Associates. Gresham, F., & Elliott, S. (1990). Social skills rating system. Circle Pines, MN: American Guidance Service. |
| Data Collection Timeframe | Data were collected during the 2000-2001 school year. |
Findings:
Formative/Process Findings
| Program Context/Infrastructure | Middle school programs were expected to meet 3 days per week for a total of 90 program days, and elementary school programs were expected to meet 4 days per week for a total of 120 program days. Middle school programs in general exceeded this standard by meeting for 116 days, while elementary school programs fell short, meeting 107 days on average. Elementary school participants were found to have received approximately 30 fewer hours of social skills and character development lessons over the course of the program than middle school participants. In the typical program, students received approximately 1.7 hours of educational services per week and 1.8 (elementary) or 2.4 (middle) hours of social skills or character development training. |
| Recruitment/Participation | On average, youth attended 71% (for middle school youth) and 80% (for elementary school youth) of available program days. The mean number of days actually attended was 64 (middle) and 68 (elementary). Eighty-four percent of elementary and 77% of middle school youth participated for more than 30 days. |
Summative/Outcome Findings
| Academic | The number of social skills lessons completed by participants, as reported by the program directors in interviews, was related to improvements in GPA (p < .01). In particular, participation in more than 30 lessons predicted increases in GPA; all other participants showed a decrease in GPA. The number of social skills lesson hours (according to program logs) predicted increased school attendance (p < .01). In particular, providing more than 1 hour of social skills per week predicted increases in school attendance. More than 3 hours of academics per week predicted increases in elementary school participants' school attendance (p < .05), though this relationship was not found for middle school youth. |
| Prevention | Programs rated by observers as having more efficient procedures (p < .05), higher levels of behavior management (p < . 01), and higher levels of overall structure (p < .05) showed significantly smaller increases in youth's rebellious behavior from pretest to posttest. Programs rated by observers as having higher levels of overall structure were more likely than other programs to show gains in youth's intentions not to use drugs (p < .05). The number of hours youth were involved in academics per week predicted lower variety of drug use in the last year for elementary school youth (p < .01), though this relationship was not found for middle school youth. In particular, more than 2.33 hours of academics per week predicted decreases in the variety of drug use by elementary school participants. The percentage of time spent in recreation predicted increases in delinquent behavior, after school violent crime, and peer drug models for middle school participants (p < .05 for each). In particular, more than 2.6 hours per week of recreation was related to increases in these three negative outcomes. |