You are seeing this message because your web browser does not support basic web standards. Find out more about why this message is appearing and what you can do to make your experience on this site better.

www.HFRP.org

The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.

Terms of Use ▼

Program Description

Overview The Massachusetts Afterschool Research Study (MARS) examined the relationship between program characteristics, program quality, and youth outcomes in Massachusetts after school programs located in 10 school districts across Massachusetts. These programs provided school-age child care and academic enrichment programming to youth during the after school hours. MARS sought to understand when, why, and how after school programs affect positive youth outcomes.
Start Date Study completed (2003–2004)
Scope state
Type after school
Location urban, suburban, rural
Setting community-based organization, public school
Participants kindergarten through middle school students
Number of Sites/Grantees 78 sites in 10 school districts
Number Served 4,108 in 2003–2004
Research Participants MARS included a wide range of multifocus programs, the majority of which could be categorized as either school-age child care programs or academic enrichment programs. School-age child care programs, many of which date back to the 1980s, aim to provide general developmental support for children of working families. They tend to be located in community centers or schools, where they are run under the auspices of community-based agencies. School-age child care programs are usually open 5 days per week after school as well as school vacations and holidays. Academic enrichment programs, most of which are relatively new, are focused on increasing the school success of youth. They are usually located in schools and often depend on school paraprofessionals and teachers for the core of their staff. They are typically open fewer hours than school-age child care programs and typically are closed for school holidays and vacations.

The sample of programs included 58 programs located in urban areas, 14 programs located in suburban areas, and 6 programs located in rural areas or small towns. The following 10 Massachusetts public school districts collaborated with the MARS data collection: Boston, Fall River, Framingham, Greenfield, Holyoke, Lowell, Northampton, South Hadley, Turners Falls, and Worcester. To be eligible for the MARS study, programs needed to (a) have been in operation for 2 years or more, (b) serve either elementary or middle school youth or both, (c) have regularly enrolled youth who attend at least 4 days per week, and must either be (a) funded by the Massachusetts Office of Child Care Services and classified as a Tier 4 program, (b) supported by the Massachusetts Department of Education’s 21st Century Community Learning Centers, and/or (c) supported by a local United Way agency. Overall, the after school programs sampled in MARS served a youth population that was evenly divided between boys and girls and was 46% White, 26% Hispanic, 21% Black, 6% Asian, and 1% other ethnicity.

About three quarters of participating programs were best described as school-age child care (N = 62), and the remaining quarter were categorized as academic enrichment (N = 16). Sixty programs in the study served elementary school youth (most of which were school-age child care programs), 15 served middle school youth (most of which were academic enrichment programs), and three served youth ranging from grades K–8. The sample included programs operated by YMCA, Boys & Girls Clubs, Citizen Schools, faith-based organizations, and others. Most programs in the sample were administered by community-based organizations but located in schools.
Funding Level Not applicable
Funding Source The study was a collaborative effort of the Massachusetts Office of Child Care Services, Massachusetts Department of Education’s 21st Century Community Learning Centers, United Way of Massachusetts Bay, the Barr Foundation and the Nellie Mae Education Foundation.


Research

Overview MARS had two major goals: (a) to identify program features/characteristics related to high quality implementation, and (b) to explore the links between program quality and youth outcomes. The MARS team developed a conceptual framework that posits two types of program features: (a) program characteristics that are structural in character (e.g., location in school or community settings, funding source, program size), and may or may not be related to program quality; and (b) program quality, which is process-oriented, and captures the actual program as it is implemented (e.g., staff–youth interaction, engaging activities, communication with families).
Researchers Intercultural Center for Research in Education, National Institute on Out-of-School Time
Research Profiled Pathways to Success for Youth: What Counts in After-School: Massachusetts After-School Research Study (MARS) Report
Further Research Planned None
Report Availability Intercultural Center for Research in Education and the National Institute on Out-of-School Time. (2005). Pathways to success for youth: What counts in after-school: Massachusetts After-School Research Study (MARS) report. Boston, MA: United Way of Massachusetts Bay. Available at: www.uwmb.org or www.wcwonline.org/mars/MARSfull.pdf


Contacts

Researchers John Zuman
Intercultural Center for Research in Education
366 Massachusetts Avenue
Arlington, MA 02474
Tel: 781-643-2142
Fax: 781-643-1315
Email: jzuman@incre.org

Beth M. Miller, Ph.D.
Senior Research Advisor
National Institute on Out-Of-School Time
Wellesley Centers for Women, Wellesley College
106 Central Street
Wellesley, MA 02481
Tel: 781-283-2507
Fax: 781-283-3657
Email: bmiller@wellesley.edu

Joyce Shortt
National Institute on Out of School Time
Wellesley Centers for Women, Wellesley College
106 Central Street
Wellesley, MA 02481
Tel: 781-283-2526
Email: jshortt@wellesley.edu
Program Lisa Pickard
Senior Director
United Way of Massachusetts Bay
51 Sleeper Street
Boston, MA 02210
Tel: 617-624-8124
Email: lpickard@uwmb.org
Profile Updated January 10, 2006

Research: Pathways to Success for Youth: What Counts in After-School: Massachusetts After-School Research Study (MARS) Report


Research Description

Research Purpose To answer the following questions: (a) What kinds of after school programs are youth in the study attending, (b) in what ways are program characteristics and features related to program quality, and (c) which aspects of programs are most likely to result in positive youth outcomes?
Research Design Quasi-Experimental and Non-Experimental: The study focused on 4,108 children in 78 after school programs distributed across the state of Massachusetts.

The researchers visited each program in the fall of 2003 and again in the spring of 2004, conducting interviews with program directors in the fall and interviews as well as observations in the spring.

To learn about the attitudes of youth in the after school programs, MARS included a survey that was administered in May 2004 to a sample of 467 youth from 15 of the middle school programs in MARS, an average of 27.5 surveys per program. More than half (54%) of youth survey respondents were in the 1st year in their after school program, 25% were in the 2nd year, and 21% had been in the same program for 3 or more years.

Outcome data were collected from program staff and classroom teachers in the fall and again in the spring. The researchers tested whether program characteristics and features were related to measures of program quality, and whether these features were related to changes in youth outcomes.

Approximately, 750 classroom teachers completed 2,023 pre (49% response rate) and 1,638 post outcome instruments, yielding 1,481 students with matching preoutcome and postoutcome instruments (a 73% return rate for both fall and spring surveys). In the fall, program staff completed 2,538 outcome instruments (62% response rate); 2,115 postoutcome instruments were also collected, yielding 1,909 students with matching preoutcome and postoutcome instrument data (a 75% response rate for both fall and spring surveys).
Data Collection Methods Interviews: Site coordinator interviews focused on program goals and characteristics.

Observation: Observations conducted at site visits focused on program quality.

Secondary Source/Data Review: Researchers collected data from programs and schools on school attendance, program attendance, and child and family characteristics.

Surveys/Questionnaires: To capture youth outcomes, the study utilized the Survey of Afterschool Youth Outcomes (SAYO), which collects information on a range of youth academic and nonacademic outcomes. It includes a version completed for each youth by an after school staff person (SAYO-S) as well as a teacher (SAYO-T).

To examine middle school youth’s perceptions of their after school programs, the middle school survey asked youth about their attendance, their satisfaction with the program, and recommendations for improving the program. Youth were also asked to rate the staff in their after school program along a variety of dimensions, such as knowing a lot about them, spending time with them, organizing good activities, and so on.

Test/Assessments: Assessment areas in the SAYO-T include Academic Performance, Homework Completion & Effort (e.g., “persists on task even when experiencing difficulty with homework assignment”), Behavior in the Classroom (e.g., “Is able to accept responsibility for his/her own actions”), Initiative (e.g., “Sets goals for self”), Engagement in Learning (e.g., “Stays focused on task at hand”), Analysis and Problem Solving (e.g., “When encounters problems, is able to identify and describe the problem,” and Communication Skills (e.g., “Volunteers to ask a question or answer a question in class”). SAYO-S assessment areas include Homework Completion & Effort, Behavior in the Classroom, Initiative, Relations with Adults (e.g., “Initiates interactions with adults”), and Relations with Peers (e.g., “Shows consideration for peers”).

The quality of the MARS programs was primarily collected using the Assessment of Afterschool Program Practice Tool-Research Version (APT-R). The APT-R has three sections: Overall Program Observation; Activity Observation; and Homework. Related items from the APT were summarized into five key Quality Indicators. These five Indicators are: (a) Staff Engagement with Youth (staff are actively engaged in activities with youth, appear to enjoy work, give positive cues, are encouraging, are relaxed, listen to youth, have positive and respectful interactions with youth); (b) Youth Engagement (youth are respectful of each other, responsive to staff, display positive behavior, are relaxed, listen to each other and cooperate with each other, interact positively with staff); (c) High Quality, Challenging Activities (activities are appropriate for youth, have clear instructions, are challenging, stimulate thinking, encourage critical/higher order thinking, are part of larger project, show evidence of prior preparation, are conducted with enough materials and supplies); (d) Quality Homework Time (staff provide individual help, staff focus on youth, staff are encouraging, staff help youth think through problems); and (e) Family Relationships at Pick-up Time (staff greet parents when picking up child, staff chat with parents, staff acknowledge youth when they leave, parents and staff express positive nonverbal cues). In addition, a structural indicator related to the environment, Appropriate Space, was created. It included environmental items related to comfortable levels of heat, ventilation, noise, and light; organization; cleanliness; furniture in good repair; materials in good order; and other similar items.
Data Collection Timeframe Data were collected during the 2003–2004 program year.


Findings:
Formative/Process Findings

Activity Implementation Site coordinator interviews revealed that nearly all programs (95%) offered a daily snack, and most programs (82%) allocated some time every day for homework. Other activities offered included open time/free play/socializing (78%), outside time/gym time (69%), academic enrichment (55%), fine arts/arts and crafts (42%), and tutoring/test prep (31%).

The APT-R Activity ratings suggested that activities tended to (a) not require prior staff preparation, (b) include limited direction or leadership by youth, and (c) fail to build on youth participants’ diverse cultural/ethnic backgrounds. Observers saw very few examples of “intentional” learning activities that had clear learning objectives, actively engaged youth in a process of guided learning, and included staff as facilitators of the learning process.

Of middle school survey respondents who provided recommendations for the program, the most common request was for more interesting, engaging activities. However, these youth also reported that they liked the social aspects of the program and the activities best.

Nearly one in five middle school survey respondents mentioned that the homework help they received at the program was positive, although homework help was also one of the most commonly cited least liked aspect of the program.
Parent/Community Involvement Nearly all program coordinators reported regular communication with parents. All site coordinators of programs where parents picked up their children (representing 97% of the total sample) said that program staff talked with parents at pick-up time, and 95% said that staff spoke with parents on the phone. However, staff most commonly communicated with parents when there was a problem, such as an attendance or behavioral issue. Only 10% of program coordinators said that they spoke with parents on a regular basis to provide updates.
Program Context/Infrastructure The top three program goals cited by site coordinators were to develop positive self esteem and self-image (50%), strengthen relationships with peers/social skills (46%), and improve academic performance (28%).

Programs with smaller group size in activities were more likely to have higher scores on the Staff Engagement Indicator and the Engaging, Challenging Activities Indicator (p < .05).

Programs where site coordinators indicated a higher percentage of time in structured activities had lower scores on the Family Relations Indicator and higher scores on the Engaging, Challenging Activities Indicator (p < .05).

Programs where site coordinators indicated more days per week of homework time had lower scores on the Engaging, Challenging Activities Indicator and lower scores on the High Quality Homework Time Indicator (p < .05).

Programs where site coordinators indicated stronger connections with schools had higher scores on the Staff Engagement, Engaging, Challenging Activities, and High Quality Homework Time Indicators (p < .05).

Programs where site coordinators indicated stronger connections with parents and the community had higher scores on the Family Relations Indicator and lower scores on the Challenging, Engaging Activities Indicator (p < .05).

Programs with larger enrollments had lower scores on the Family Relations Indicator and higher scores on the Challenging, Engaging Activities Indicator (p < .05).

Programs serving elementary youth scored lower on the Staff Engagement and Engaging, Challenging Activities Indicators and higher on the Family Relations Indicator than other programs (p < .05).

Programs where site coordinators indicated offering more project-based learning activities scored higher on the Engaging, Challenging Activities and High Quality Homework Time Indicators (p < .05).

Programs rated as well paced scored higher on the Staff Engagement, Youth Engagement, and Family Relations Indicators (p < .05).

Programs rated as well organized with clear routines scored higher on the Staff Engagement, Youth Engagement, and High Quality Homework Time Indicators (p < .05).

Programs serving middle school youth scored higher on the Staff Engagement and Engaging, Challenging Activities Indicators and lower on the Family Relations Indicator than other programs (p < .05).

Programs with more highly educated staff, at both the program director and direct service levels, were rated higher on program quality, including Staff Engagement, Youth Engagement, Activities, and Homework Time (p < .05). In addition, programs that utilized certified teachers and other school staff tended to be rated higher on these Quality Indicators (p < .05).

Programs with higher staff wages had higher quality in all areas except the Family Relations Indicator (p < .05).

Programs offering more staff training scored higher on the Staff Engagement Indicator (p < .05).

Programs with higher staff turnover (more staff leaving during the school year) had lower quality ratings in both Youth Engagement and Homework Time (p < .05) Indicators.

Programs with low staff-to-youth ratios scored higher on the Staff Engagement, Youth Engagement, Activities, and Homework indicators (p < .05).
Program–School Linkages Programs located in schools reported much more communication and coordination with schools than community-based programs. Thirty-nine percent of site coordinators described their communication with school as “extensive.” All of these programs were school-based. On the other end of the spectrum, 25% of coordinators reported that they had minimal or no communication with schools, and two thirds of these programs (69%) were community-based.
Recruitment/Participation The after school programs in the MARS sample reported an average enrollment of 63 youth in October 2003, with a range from 14 to over 300. The average enrollment for the 60 programs serving children in grades K–5 was 53, whereas the average enrollment for the 18 programs serving middle school youth was 96.

The average daily attendance for all programs on the day of the spring 2004 visit was 74% of total enrollment. The average attendance rate was 78% for the school-based child care programs and 59% for the academic enrichment programs.

The average dosage (i.e., number of hours youth attended the program across the program year) across all programs was 191 hours, or approximately 5.6 hours per week. The dosage varied considerably between the school-age child care programs and the academic enrichment programs. For the school-age child care programs, the average dosage was 287 hours during the year, or approximately 8.0 hours per week. For the academic enrichment programs, the average dosage was 96 hours for the year, or an average of 3.2 hours per week. However, researchers caution that these numbers also may reflect differences in how programs tracked and reported attendance, the quality of which varied within the sample programs.

Program attendance of middle school youth who responded to the MARS survey was relatively high; nearly half reported that they attended 4 or 5 days per week, and on average, they reported spending slightly more than 8 hours per week in their after school program.
Satisfaction Nearly two thirds (63%) of middle school survey respondents reported that they liked the program “a lot,” and 32% reported that they liked it “somewhat.” Only 5% reported that they did not like the program at all. When asked what they liked least, one third of youth reported that they liked everything.
Staffing/Training After school programs utilized a range of paid and unpaid staff, including community volunteers, high school student tutors, school day teachers, and various specialists. Across the sample of programs, there was a total of 675 paid staff, for an average of 8.7 paid staff per program. Forty-eight percent of paid staff had a bachelor’s degree or higher degree, and an additional 7% of paid staff had a 2-year associate’s degree.

There was a wide range of pay for staff of the MARS sample programs, reflecting the variations in program models, staffing, and resources. Senior group leaders/lead teachers earned an average of $15 per hour, with a range between $7.75 and $35 per hour. Group leaders/teachers earned an average of $12.60 per hour, with a range between $7 and $30 per hour. Assistant group leaders/teachers and aides earned an average of $9.25 per hour, with a range between $6.75 and $17 per hour.

Only 27% of programs reported providing health care benefits.

Sixty percent of the site coordinators in the sample reported having a bachelor’s or advanced degree, and 26% were in college during 2003–04. Sixty percent of these program coordinators earned less than $30,000 per year. Two thirds (69%) worked full-time and the remaining 31% held second jobs. About one third (36%) had other responsibilities at the program in addition to program coordination (including classroom teacher, parent/community liaison, guidance counselor, school administrator, and curriculum coordinator).

Only 15% of the programs retained all of their paid staff between September 2003 and May 2004. Fifty-seven percent of the 66 programs who hired new staff members during the year hired one or two new staff, and the remaining 43% hired more than two new staff members during the year. The overall staff turnover rate during 2003–2004 was 22%.

The vast majority of program staff received some type of staff development during 2003–2004. The most common staff development topics for program staff included first aid/CPR, behavior management, and child development. Among site coordinators, the most common training topics included courses in child development, program administration, behavior management, curriculum, and leadership. Eighty-seven percent of site coordinators and 85% of staff reported participating in training for an average of 30 hours for coordinators and 14 hours for staff members.

According to site coordinator interviews, the average paid staff-to-child ratio across the programs in the sample was 8.4 to 1. This ratio included all paid staff, including site coordinators, if they worked directly with the youth in the program. The average ratio observed in spring 2004 site visits was 6.5 to 1 (this number included volunteers and specialists). For programs serving elementary-school-age children, this ratio was 7.2 to 1, while for programs serving middle-school-age youth, it was 4.3 to 1.

Across the sample, the highest APT-R ratings were on items related to a general sense of ease and friendliness between staff and youth. Specifically, most interactions between staff and youth were positive, youth seemed relaxed and friendly with each other, activities were appropriate for the age group, and enough staff were available for youth. There were very few conflicts between youth during the afternoons that researchers visited programs.

The APT-R staff ratings revealed that while staff tended to effectively control activities and maintain calm, they tended not to engage with youth to elicit reflection, spark interest, and facilitate learning; ask open-ended questions to stimulate conversations; or make extensive use of positive reinforcement. The Homework ratings suggested a similar pattern. While staff tended to effectively manage homework time, they tended not to engage extensively with youth to encourage them or to engage with them in learning.

About half of the youth surveyed seemed to have strong positive feelings about staff, with the most highly rated items including: “staff treat me with respect,” “staff are friendly with me,” “staff care about what happens to me,” “staff talk with me,” and “staff help me improve my learning.” The lowest rated items included “staff talk with my parent/guardians,” “staff know a lot about me,” “staff involve me in planning activities,” and “staff involve me in leading activities.”

Of middle school survey respondents, 15% mentioned negative feelings about the staff in response to what they liked least about the program.


Summative/Outcome Findings

Academic Academic SAYO-T scores did not vary by program characteristics or program quality.

Programs with better relationships with school teachers and principals had greater gains on the SAYO-S Homework Completion and Effort Indicator (p < .05). Other factors (program size, staff-to-child ratio, elementary vs. middle school, and school-age child care vs. academic enrichment program) had no relationship to the level of change in youth academic outcomes on the SAYO-S.

Programs with better educated staff had greater gains on the SAYO-S Homework Completion and Effort Indicator, while programs with higher staff turnover had greater losses on this Indicator (p < .05 for each). The percentage of staff who were certified teachers was not linked to changes in this indicator.

A path model for the SAYO-S Homework Completion and Effort Indicator indicated that appropriate space led to positive staff engagement, which in turn led to positive youth engagement, challenging activities, and high quality homework time (p < .05 for all relationships). Youth engagement was then strongly linked to the Homework Completion and Effort Indicator. Challenging activities and quality homework time were not directly linked to positive changes in the Homework outcome, though the researchers note that this result may have been due to the lack of variation in the data in these areas. Family Relations, while not related to the other quality indicators, were positively but not significantly related to changes in Homework outcomes.
Youth Development Youth Development SAYO-T scores did not vary by program characteristics or program quality.

Youth in programs with better relationships with school teachers and principals showed greater gains on the SAYO-S Initiative Indicator (p < .05).

Youth in programs with better relationships with school principals showed greater gains on the SAYO-S Behavior in the Program and Relations with Peers Indicators (p < .05).

Youth in programs located in schools showed greater gains than those located in community-based sites on the SAYO-S Initiative and Relations with Peers Indicators (p < .05).

Other factors (the size of the program, staff-to-child ratio, elementary vs. middle school, and school-age child care vs. academic enrichment program) showed no relationship to the level of change in youth developmental outcomes on the SAYO-S.

Staff turnover, percentage of staff with a bachelor’s degree, director education level, and percentage of staff that were certified teachers generally had no relationship with changes in SAYO-S Initiative, Behavior in Program, or Relations with Adults indicators. The one exception was that youth in programs with a higher percentage of certified teachers as staff showed greater gains on the Relations with Peers Indicator (p < .05).

A path model for each of the four developmental SAYO-S outcomes (Initiative, Behavior in Program, Relations with Adults, and Relations with Peers) indicated that appropriate space led to positive staff engagement, which in turn led to positive youth engagement, challenging activities, and high quality homework time. In every case, youth engagement was then strongly linked to the SAYO outcomes. Challenging activities and quality homework time were not directly linked to positive changes in these youth outcomes, though the researchers note that this result may have been due to the lack of variation in the data in these areas. Family Relations, while not related to the other quality indicators, was positively but not significantly related to changes in three of the four SAYO-S developmental outcomes and had a significantly positive relationship with changes in Relations with Adults (p < .05).

 

© 2016 Presidents and Fellows of Harvard College
Published by Harvard Family Research Project