You are seeing this message because your web browser does not support basic web standards. Find out more about why this message is appearing and what you can do to make your experience on this site better.

www.HFRP.org

The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.

Terms of Use ▼

Program Description

Overview The New Jersey (NJ) After 3 initiative seeks to increase the number of after school programs for youth in NJ that provide a safe environment during after school hours, offer enriching academic activities and homework assistance, and expose youth to nurturing adults and meaningful experiences that promote intellectual, physical, social, and artistic development.
Start Date Fall 2005
Scope state
Type after school
Location urban, suburban
Setting public schools
Participants kindergarten through middle school students
Number of Sites/Grantees 66 programs in 2005–2006
Number Served 11,108 in 2005–2006
Components NJ After 3 programs offer an array of activities intended to build youth’s skills and maximize their exposure to new activities. Community-based, non-profit organizations operate these programs in collaboration with public schools. Programs are offered between 3:00 a.m. and 6:00 p.m. every day school is in session, and youth are expected to attend every day. Programs, which are open to all youth attending the school in grades K–8, offer a comprehensive and balanced set of activities, including academic support and enrichment, arts, sports, and community service. Activities are aligned with the school’s academic standards, in order to complement but not duplicate school-day instruction. Programs are staffed by a full-time, year-round site coordinator who works at the school, and a mix of other staff including parents, college students, teaching artists, youth workers, certified teachers, and other school staff. Programs should have a 1:10 staff–youth ratio for grades K–6 and 1:12 ratio for grades 7–8.
Funding Level Approximately $12 million in grants are available during FY 2007.
Funding Sources State of New Jersey; Robert Wood Johnson Foundation; assorted sponsors and partners (including businesses, foundations, and nonprofit organizations)


Evaluation

Overview A 3-year longitudinal evaluation is being conducted. The 1st year provides baseline data. The future phases of the evaluation (Years 2 and 3) will provide data on the extent to which programs and participants were able to meet the initiative’s goals.
Evaluator Policy Studies Associates
Evaluations Profiled Evaluation of New Jersey After 3: First-Year Report on Programs and Participants
Evaluations Planned Data will be collected through the 2007–2008 program year.
Report Availability Kim, J. H., Miller, T. D., Reisner, E. R., & Walking Eagle, K. P. (2006). Evaluation of New Jersey After 3: First-year report on programs and participants. Washington, DC: Policy Studies Associates, Inc. Available at: www.policystudies.com/studies/youth/NJ%20After%203%20Year%201%20Report.pdf.


Contacts

Evaluation Elizabeth Reisner
Founder and Principal
Policy Studies Associates, Inc.
1718 Connecticut Avenue, NW, Suite 400
Washington, DC 20009
Tel: 202-939-5323
Fax: 202-939-5732
Email: ereisner@policystudies.com

Program Mayra Ramírez
Senior Program Officer
New Jersey After 3, Inc.
391 George Street, 2nd Fl.
New Brunswick, New Jersey 08901
Tel. 732-246-7941
Fax. 732-246-7993
Email: mramirez@njafter3.org
Profile Updated July 13, 2007

Evaluation: Evaluation of New Jersey After 3: First-Year Report on Programs and Participants


Evaluation Description

Evaluation Purpose To address the following questions: (a) What are the characteristics of programs supported by NJ After 3? (b) What are the characteristics and attendance patterns of the youth served by NJ After 3? (c) What did NJ After 3 accomplish with respect to enhancing the quality of after school services? (d) What did NJ After 3 accomplish with respect to expanding the availability of after school services? (e) How successful was NJ After 3 in creating sustainable systems for funding and program quality? and (f) What were the initiative’s effects on participating children and families?
Evaluation Design Non-Experimental: Data were collected from 19 of 21 programs initially funded by NJ After 3 in 2004 (Round I programs) and from 36 of 45 programs funded for the first time in 2005 (Round II programs). Evaluators collected data on participant characteristics and attendance patterns, and site director and executive director surveys from all Round I and II programs. Survey data were also collected from third- to fifth-grade participants in all Round I programs. Ten Round I programs were then randomly selected for site visits to conduct interviews and observations. School-day reading/language arts and homeroom teachers in the 10 Round I programs also completed surveys of youth in grades 3–6. The final samples included 27 executive directors (69% of the directors of 39 provider organizations); 55 site coordinators (83% of the coordinators of the 66 programs); 11,108 youth participants in the overall K–8 sample from the youth tracking system (all youth); 671 youth in the grade 3–8 youth survey sample (73% of eligible participants); 303 youth assessed by the grade 3–6 teacher survey (57% of eligible participants); 57 program staff, 63 youth, and 23 parents in the interview sample; and 179 different after school classes or activities during site visits to the 10 in-depth sites.

Boys and girls were evenly represented among participants, and most were African American or Latino. About half of all participants came from low-income families (i.e., were eligible to receive free or reduced-price lunch). Twelve percent were “English Language Learners” or received special education services. Participants were distributed across grade levels, with 36% in grades K–2, 37% in grades 3–5, and 27% in grades 6–8. Youth were categorized into one of three participation levels, based on the number of days they attended the NJ After 3 program: (a) “highly active” (attended at least 80 days and 80% of the days that they were enrolled in the program during the school year); (b) “active” (attended at least 60 days and 60% of the days enrolled); and (c) “non-active” (attended fewer than 60 days or less than 60% of the days enrolled).
Data Collection Methods Interviews/Focus Groups: Interview with site coordinators, program staff, youth participants, and parents focused on perceptions of program implementation and outcomes.

Observation: Evaluators used a detailed observation guide to record observations of activities and classes to assess youth engagement, instruction, activity content, and staff–youth relationships.

Secondary Source/Data Review: YouthServices.net, the vendor for the NJ After 3 management information system, provided data on the characteristics and program attendance patterns of all participants.

Surveys/Questionnaires: Site coordinators surveys provided data on program goals, activities, and schedules; staff recruitment and qualifications; participant outreach, recruitment, needs, and preferences; and efforts to connect participants’ schools, communities, and families. Surveys of executive directors of provider organizations provided data on NJ After 3 influences on the organizations in fulfilling their core missions, NJ After 3 programs’ links to other services delivered by provider organizations, and the cost and funding of specified elements of NJ After 3 programs. Youth surveys provided data on participants’ behavior, attitudes, and skills. Teachers who could address youth’s academic and reading/language arts skills were targeted for surveys, which provided data on NJ After 3 youth participants’ behavior, attitudes, and skills.

Test/Assessments: Reading/language arts and homeroom teachers completed a brief report on each NJ After 3 participant in grades 3¬–6 who attended one of the 10 programs in the evaluation’s in-depth sample. Teachers assessed the skills and behaviors that after school participants exhibited during the regular school day.
Data Collection Timeframe Data were collected in 2005–2006 (Year 1 of the initiative).


Findings:
Formative/Process Findings

Activity Implementation Site coordinators reported a fairly consistent set of academic program offerings across the sites. Nearly all programs offered opportunities for youth to do homework, participate in math and other learning games, and practice reading and writing. Similarly, almost every program offered arts and athletic activities such as visual arts, dance, free time for recreation, and organized team sports. More than half of site coordinators said that youth participated in service projects, discussed current events, and learned about different careers during the program. A few sites reported that they provided opportunities for civic engagement, community service, and career exploration.

Observations revealed that while opportunities aimed at promoting learning and well-rounded development—including opportunities for thinking analytically, developing relationships with staff, contributing to discussions, making meaningful decisions, assuming leadership roles, and collaborating with peers—may be available within a program, these experiences were not all available within a single class or activity. Only through exposure to a variety of instructors and activities, each with a different focus or goal, did youth get exposed to all of these opportunities.

Variation was observed in youth engagement measures across certain activities. For example, youth scored lower on engagement measures in homework activities than in other academic and enrichment activities. In arts activities in particular, youth tended to communicate and listen to each other more often and were typically allowed to make more meaningful choices within the context of the activity (e.g., assisting with choreography during a dance class).

Based on assessments of key aspects of an activity such as content and structure—including organization, challenge level, analytic thinking, and skill involvement—activities that targeted decision-making or interpersonal communication skills (e.g., peer discussion groups) received higher scores compared to other kinds of activities. Highly rated activities typically were well organized, challenged youth, and required analytic thinking. Academic activities (other than homework) received the highest score on the activity scale and also demonstrated strong staff–youth relationships and varied instructional strategies.

The presence of a certified teacher leading an activity had no significant relationship with the type of instructional methods that were employed during the activity or on any other components of the activity scale.

When the goal of an activity was to build or practice skills, staff were more likely to employ clear-cut and varied instructional methods than were used during homework or other activities.

The majority of youth participants indicated that the program gave them a chance to do new things, work on tasks that made them think, and participate in activities that held their interest. At least half agreed that the program allowed them to do things that they usually did not get to do elsewhere, gave them many activities to choose from, and provided an opportunity for them to become involved in community service. Agreement was higher amongst youth in grades 3–5 than youth in grades 6–8.

Parents and youth reported a desire for programs to provide a variety of rich content-based activities. They reported wanting programs to provide information and learning opportunities that were not available at school and or through family resources. Programs recognized this need and tried to offer such opportunities and to hire appropriate staff for the activities that youth and parents requested.

Programs confronted obstacles such as limited space, unqualified staff, or inadequate financial resources as they tried to introduce new classes and activities.

Observation data indicated that youth were particularly engaged by learning- and mastery-oriented activities that were structured and focused on the achievement of clear goals. Long-term dance and drama projects, for example, typically satisfied these criteria and were immensely popular among youth.
Costs/Revenues On average, almost three quarters of each program’s budget in 2005–2006 was provided by NJ After 3. The remaining budget resources came from general organizational funds, other state funding sources, fees charged to families, allocations from other municipal sources, federal funding sources, and other organizations.
Parent/Community Involvement Almost all site coordinators said that they reached out to parents and the community at least once a month by calling parents, arranging opportunities for communication between parents and representatives from local agencies, and meeting with one or more parents. Much of the interaction between staff and parents occurred informally, such as during pick-up at the end of the program day or over the phone.

Parent involvement in program activities was limited because work and family responsibilities made it difficult for them to attend the program during after school hours. Those who said they were involved often chaperoned occasional field trips or prepared and served food at program events and celebrations in the evenings or on weekends.

Community agencies typically got involved in program activities through the provision of resources that supported program goals. For example, they offered special programs and services to youth, provided funding, and referred youth to NJ After 3 programs.

Program staff used their expertise, experience, and personal contacts in the fields of social services and education to identify resources and information for participants and their families. Many programs therefore served as conduits between families and community resources, helping to identify needs, raise awareness, and provide necessary services.

Survey data revealed that the amount of program and school communication with parents varied significantly among parents. Specifically, school-day teachers reported that they communicated with African American and White parents more often than they did with Latino and Asian parents. Most site coordinators reported that a major challenge for them was finding sufficient staff with skills to work with English Language Learners.
Program Context/Infrastructure In surveys and interviews, site coordinators said that they served youth populations characterized by diverse needs. Accordingly, local program goals reflected a desire to respond to as many of these needs as possible. Most site coordinators indicated on surveys that their main goals were to provide positive adult guidance, a safe environment, and opportunities for social development. Other goals included improving academic, health, and other life skills; providing opportunities for recreation and cultural enrichment; stimulating youth’s interests in learning; and helping youth learn in nontraditional ways through exposure to positive new experiences. Several site directors also reported a desire to raise awareness among parents of the need for them to support their children’s social, physical, and cognitive development.

Parents at each program visited reported that the NJ After 3 program was the only after school option available in their community that satisfied their dual needs for safety and affordability (since it was free). They also appreciated that it was conveniently located. Similarly, youth indicated that NJ After 3 filled a vacuum that would not otherwise be filled with safe, positive, and meaningful activities.

The majority of site coordinators reported that they had provided after school services at their current location prior to receiving the NJ After 3 grant, and several indicated that they had been doing so for more than 5 years. They all reported that earlier program services had been less extensive and served fewer youth, compared to current programs.
Program–School Linkages The quality of the relationships between programs and schools varied considerably. At over half of the programs visited, staff reported that they did not have very good relationships with school-day teachers. Tension focused on the use of shared space and materials. However, some after school staff reported that they frequently spoke with school-day staff to understand what youth were doing in the classroom, so that they could provide targeted assistance or ensure that homework was completed. Several site coordinators also described efforts to align program content with the school-day program.
Recruitment/Participation In total, 43% of K–8 participants met the evaluation’s criteria for highly active program participation, and another 20% met the criteria for active participation. On average, participants attended 73% of the program sessions available to them. Attendance was generally higher among younger children than among older youth.

According to school-day teachers, most of the youth who participated in NJ After 3 programs “always” or “often” demonstrated behaviors that indicated they were academically engaged and possessed important interpersonal, technological, and study skills. Most also met or exceeded the school’s grade-level expectations in reading and language arts skills. However, teachers indicated that fewer than half “always” or “often” demonstrated academic motivation (eagerness to learn, willingness to take on challenges, ability to stay on target, and responsibility for their own learning).

In many skill and knowledge areas, participants from low-income families were less likely than other participants to be rated by teachers as possessing the capacities deemed desirable for academic success. This was also true for younger versus older youth, with fewer younger children rated as possessing needed skills.

Fewer than half of the youth surveyed indicated that they engaged in sustained physical activity for 3 or more hours each week. Youth also reported high levels of time spent watching television and playing video games (e.g., approximately 40% reported watching television more than 4 hours per school day). Three quarters of youth reported eating food from a fast-food restaurant at least 1–2 times a week, a quarter of which reported doing so at least 3 times a week.
Staffing/Training Almost half of all program staff had bachelors degrees or higher, and one quarter were certified teachers.

Among site coordinators, nearly all had a 4-year college degree and at least 1 year’s experience as an after school program director or staff member.

Most program employees worked fewer than 19 hours per week at the program, and most staff worked directly with youth.

Across programs, the youth–staff ratio was 6:1, which was lower than the 10:1 ratio required by NJ After 3. Most site coordinators said that groups were small enough to allow staff to meet participants’ individual needs.

Almost all of the executive directors of provider organizations reported that their organization delivered some type of training to site coordinators or other staff. Other data suggest, however, that there was uneven access to professional development and training opportunities and that some individuals received more hours of training than others; for example, 41% of site coordinators reported limited professional development as a major challenge.

When asked to identify the professional development that had been most valuable to their staff, the majority of site coordinators cited classroom management, youth development training, and training in academic enrichment and literacy instruction strategies.

In interviews, parents expressed satisfaction with program staff, often describing them as trustworthy, committed, capable, caring, and fair. Many attributed their child’s social development and improvements in confidence and self-esteem to the family-like atmosphere they said was fostered by program staff.

Youth participants expressed satisfaction with program staff; the majority reported that they had positive interactions with staff, felt that staff cared about them, and could talk to staff about things that were bothering them.

Evaluators found that a critical program feature was staff support of the development of positive relationships between staff and youth. The presence of “youthful” staff with whom youth felt they could relate and the programs’ low youth¬–staff ratios supported these positive relationships. Positive relationships flourished within activities that encouraged communication between staff and youth and that interested and engaged youth physically and mentally.


Summative/Outcome Findings

Academic Most youth agreed with statements designed to assess perceptions of whether the program helped them with their schoolwork and promoted academic achievement. Youth who attended the program at the “highly active” level scored significantly (p < .05) higher on the academic benefits scale than their peers who did not attend as often. Similarly, on average, youth from low-income families were significantly (p < .05) more likely than other youth to agree with statements about the academic benefits of the program.
Family Many parents expressed a sense of relief and gratitude that the NJ After 3 program was available to their children. In addition to experiencing peace of mind, parents reported practical economic and personal benefits, such as the ability to focus on their jobs and the freedom and flexibility to attend to necessary personal responsibilities.
Systemic Sites reported that support from NJ After 3 had enabled programs to expand and improve classes and activities, train staff, and increase capacity so that more youth were served. A few sites also reported that the additional resources facilitated the establishment of new partnerships with public schools and cultural organizations. A smaller number of sites reported that the NJ After 3 resources increased their ability to leverage corporate, private, or municipal funds.
Youth Development A significant (p < .05) majority of participants indicated that the NJ After 3 program gave them a chance to do new things, work on tasks that made them think, and participate in activities that held their interest. At least half agreed that the program allowed them to do things that they usually did not get to do elsewhere, gave them many activities to choose from, and provided an opportunity for them to become involved in community service. Youth in grades 3–5 expressed higher levels of agreement than did youth in grades 6–8.

© 2016 Presidents and Fellows of Harvard College
Published by Harvard Family Research Project