You are seeing this message because your web browser does not support basic web standards. Find out more about why this message is appearing and what you can do to make your experience on this site better.

www.HFRP.org

The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.

Terms of Use ▼

Program Description

Overview The Teen REACH (Responsibility, Education, Achievement, Caring, and Hope) program is an effort to provide positive youth activities during nonschool hours for Illinois youth. Teen REACH program providers are funded by the Illinois Department of Human Services (DHS). Funding goes to community-based agencies across Illinois such as local health departments, social service agencies, youth agencies, faith-based organizations, and community coalitions to implement the program. Teen REACH seeks to expand the range of choices and opportunities that enable, empower, and encourage youth to achieve positive growth and development, improve expectations and capacities for future success, and avoid or reduce negative risk-taking behavior. Its goals are to improve youth’s academic performance; provide opportunities for learning positive social skills, demonstrating positive social interactions, and building positive social relationships; encourage the adoption of positive decision-making skills that discourage harmful risk-taking behaviors; and strengthen parent-child bonds and community involvement.
Start Date October 1998
Scope state
Type after school, comprehensive services, weekend
Location urban, suburban, rural
Setting community-based organization, religious institution, recreation center, public school
Participants kindergarten through high school students (ages 6 through 17)
Number of Sites/Grantees 37 organizations with 73 program sites in 1998; 109 organizations with 250 program sites in 2004
Number Served over 50,000 from 1998 to 2004, approximately 30,000 annually
Components Programs are required to provide the following core services: (1) academic assistance, including homework time, basic skills tutoring, and enrichment programs to encourage creativity; (2) recreation, sports, cultural, and artistic activities to provide safe outlets for trying new skills and interests, building friendships and sense of belonging, and gaining developmentally relevant experiences; (3) positive adult mentors who maintain positive, sustained relationships with youth participants through one-on-one interactions; (4) life skills education that promotes abstinence from risky behaviors such as substance use, crime, violence, and sexual activity; and (5) parental involvement activities aimed at fostering parent/staff discussion and relationships, parent/child bonds, and community involvement. In addition, the Illinois DHS suggests that program participants should be given the opportunity to participate in at least one community service activity each year.
Funding Level $8.4 million in 1999; $18.4 million in 2000; over $19 million annually from 2001 to 2004
Funding Sources Illinois Department of Human Services (DHS)


Evaluation

Overview DHS’s Bureau of Community and Youth Services contracted with the Center for Prevention Research and Development at the University of Illinois to conduct an outcome evaluation for Teen REACH. During the pilot phase, the evaluation sought to understand youth outcomes at a small subset of program sites, in order to improve the overall evaluation process and provide data for continuous improvement. Second- and third-year evaluations have built on this pilot evaluation, adding a staff survey and including more programs in the evaluation sample. In the third year, work began on the development of program benchmarks and parent and teacher surveys that ultimately will be used to assess Teen REACH programs.
Evaluators Center for Prevention Research and Development at the University of Illinois
Evaluations Profiled Teen REACH: A Summary of the Pilot Evaluation

Teen REACH: Annual Evaluation Report 2002

Teen REACH: Annual Evaluation Report 2003
Evaluations Planned Teen REACH evaluations will continue to be conducted annually.
Report Availability University of Illinois, The Center for Prevention Research and Development. (2001). Teen REACH: A summary of the pilot evaluation. Champaign: Author. Available at www.cprd.uiuc.edu/trdocs.html.

University of Illinois, The Center for Prevention Research and Development. (2002). Teen REACH: Annual evaluation. Champaign: Author.

University of Illinois, The Center for Prevention Research and Development. (2004). Teen REACH: Annual evaluation report. Champaign: Author. Available at www.cprd.uiuc.edu/trdocs.html.


Contacts

Evaluation Kay Erwin Mulhall, M.Ed.
Coordinator of Research Programs
Institute of Government and Public Affairs
Center for Prevention Research and Development
510 Devonshire Drive
Champaign, IL 61820
Tel: 217-333-3231
Fax: 217-244-0214
Email: kemulhal@uiuc.edu
Program Karrie Rueter
Illinois Department of Human Services
Office of Prevention
535 W. Jefferson, 3rd Floor
Springfield, IL 62702-5058
Tel: 217-557-2943
Email: dhspab@dhs.state.il.us
Profile Updated December 1, 2004

Evaluation 1: Teen REACH: A Summary of the Pilot Evaluation


Evaluation Description

Evaluation Purpose To assess the relationship between program participation and youth educational and socio-behavioral outcomes, to develop local understanding and capacity for program sites to effectively participate in the evaluation, and to provide opportunities for grantees to understand, utilize, and continuously improve their programs through data-based decision making.
Evaluation Design Quasi-Experimental and Non-Experimental: The Illinois Department of Human Services (DHS) selected a sample of five agencies administering programs DHS judged to be well functioning and likely to cooperate with the evaluation process. The sample was selected to represent four of the five DHS regions in the state, include a range of types of provider agencies (e.g., community-based organizations, health departments), represent both rural and urban communities, and assure cultural and racial diversity among participants. Providers from each selected agency were asked to select one site based on sample size and on strength of the program. Across the five agencies, seven program sites were asked to participate in the evaluation. At the completion of data collection, evaluators had pretest and posttest data from five sites (two sites did not submit complete data due to the absence of a consistent staff member).

Evaluators used a single-group pretest/posttest design to assess changes in youth outcomes. Data were examined by comparing pretest and posttest data and by exploring the relationship between posttest data and program attendance levels (i.e., high, medium, and low attendance) or dosage (defined in this evaluation as the total number of days youth attended). Attendance levels were measured as follows: high (241 programs days or more attended over the past two years), medium (145–240 programs days), and low (1–144 programs days).

Of the 349 youth initially attending the selected program sites, 275 completed pretest surveys (response rate of 79%). At posttest, 283 youth completed surveys, and a total of 197 youth completed both surveys.

Of those who completed both, the average age at pretest was 13, with 41% in Grades 4–6, 46% in Grades 7–9, and 13% in Grades 10–12. This sample was evenly split between girls and boys. The majority was African American (68%), with the remainder Latino (21%), white (12%), and other (11%). The vast majority (89%) participated in the free/reduced-price lunch program at school. Over a third (35%) lived in single-parent households, 46% lived in two-parent households, and the remaining 19% lived with non-parent adults (e.g. grandparents, foster parents, etc.).

Of the 167 youth who answered questions about program participation at posttest, 48% had been involved for 2 years, 32% for 1 year, and 20% for less than 1 year, with an average of 1.28 years. Of these youth, 9% participated about 6–7 days a week, 38% participated 5 days, 18% participated 4 days, and 35% participated 1–3 days, with an average weekly attendance of 4.08 days. In addition, 47% were classified as high attenders, 26% as medium, and 27% as low. The average total dosage was 267 days, ranging from 16 to 672 days.

Program staff and youth were selected for interviews using a theoretical sampling approach, which involves selecting informants based on the likelihood that they will provide a rich source of information about the phenomena under study. Specifically, local staff selected youth on the basis of age, gender, and length of program experience.
Data Collection Methods Interviews/Focus Groups: Staff and youth interviews were conducted during visits at five of the sites. Through group interviews at each site visited, evaluators spoke with every director and coordinator and at least two other staff members, for a total of 25 staff members across sites. Staff who could not participate in the group interview were interviewed individually. All staff were asked about their program goals, whether the program was being implemented as originally planned, program changes they would like to make, implementation obstacles encountered and how they were handled, program successes, evidence that the program was working, whether the program differed in its effectiveness for particular groups, program improvement suggestions, and program strengths and weaknesses. Evaluators conducted five youth focus groups, one at each site visited. At least 7 youth participants were interviewed at each site, with 8–12 youth in most focus groups. Youth participants, ages 10–18, were asked about their best experience in the program, what they would keep in the program, what they would add or take away to improve the program, and anything else that would help explain how the program runs.

Observation: Evaluators conducted 4–6-hour site visits in March and April 2001 at the five sites that completed surveys. Evaluators observed site facilities, staff-youth interactions, program implementation, and selected Teen REACH programming and activities.

Surveys/Questionnaires: Youth in Grades 4 and above at the selected sites were surveyed in December 2000/January 2001 (pretest) and again in May/June 2001 (posttest). The survey included items to assess youth background/demographic information, perceived connections with staff, perceived program impacts (the frequency with which specific topics are addressed in Teen REACH and perceptions of program benefits), and youth outcomes. Youth outcome areas included frequency of homework completion, school attendance (days absent in the past month and reasons for these absences), academic performance (self-reported grades), academic aspirations (e.g., “how important is to you to graduate from high school?”), quality of school life (e.g., frequency with which youth are happy when they are in school), engagement in prosocial and delinquent behaviors (e.g., “in the last 6 months, how often did you tell lies or cheat?”), substance use (reports of alcohol, tobacco, and drug use during the past 30 days), leadership (e.g., “I like to set a good example for other young people”), self-concept (e.g., “I am happy with myself as a person”), adult connections (e.g., “I know people/places in my community where I can get help with a personal problem”), peer-group cohesion (e.g., “I enjoy the time I spend with my friends”), problem-solving skills (e.g., “when I face a new problem, I first try to find out what it is”), and parent involvement in child’s education and Teen REACH (e.g., the frequency with which parents visit the Teen REACH program or their child’s school).
Data Collection Timeframe Data were collected during the 2000–2001 school year.


Findings:
Formative/Process Findings

Activity Implementation According to site observations and interviews with staff and youth, program goals and objectives were often met through recreation activities, which provided opportunities for youth to learn positive social skills, demonstrate positive social interactions, and build positive social relationships. For example, staff at two sites organized basketball and softball games with other Teen REACH programs or youth serving agencies and found these activities to be valuable in providing youth with opportunities for positive interaction with youth outside their town or neighborhood and from other racial and cultural groups.

Every program had staff interacting with, supervising, and moderating many of the recreation activities. Staff continually taught about appropriate social interaction through praise, modeling, direct rule giving, or enforcing. Although at times youth were observed doing this for each other, the quality of recreational programming seemed to be stronger and the goals more clearly met when staff were engaged directly with youth in the activity.

Youth frequently expressed enjoyment of experiences involving traveling beyond their neighborhood or community (e.g., amusement parks, plays, museums, conferences, movies, overnight trips out of town). Other experiences that staff cited as successful in engaging youth were mentoring trips, programs at local universities or colleges, special interagency youth nights, running a lawn service for the elderly, and using facilities at other collaborating agencies (e.g., computers, swimming pools, bowling lanes).

Most of the energy and focus for program planning and delivery observed by evaluators or noted by staff and youth were directed toward homework help and basic skills tutoring.

One area of perceived inconsistency across sites was the structuring and scheduling for academic assistance/tutoring/homework help. Some sites thought they had to enforce a definite homework-only time every day. Other sites were very flexible about their academic component, making homework help available every day, but not required.

Staff and youth at four sites continually cited homework help as a program strength. Staff at all sites said that many of the participants’ parents work and often do not have the time or energy—and sometime even the ability or experience—to help with homework.

There was little evidence indicating more than a cursory use of best practice tools that programs were supposed to implement (e.g., project-based learning, service learning, problem-solving approaches in innovative programming).

Many staff and some youth (mainly teenagers) stated that it was difficult to offer services for youth in the entire 6–17 age range. Staff recognized that services for youth aged 6–12 needed to be different than services for youth aged 13–17, but noted that it was difficult to implement programming for both age groups simultaneously, especially with limited staff.

Many staff reported sharing their own career development decisions with youth to help youth open up to planning for their futures and to highlight the steps and skills needed to “make it” (e.g., school success, avoiding drugs and gangs). One staff member developed a weekend basketball tournament that was interspersed with career talks and information.

Staff at some sites felt that evidence-based programming (i.e., programming that is based on the results of research using scientific methods) was encouraged by DHS, but not supported with appropriate training or resources.

Some sites felt that short-term or one-shot program sessions worked best at their sites, because of the difficulty of keeping a cohesive group involved across multiple sessions.
Parent/Community Involvement Most sites saw parental involvement as the most difficult component to implement successfully. According to the evaluators, only one site seemed to actively work with families to achieve the targeted program outcomes.

Many staff felt they faced the difficulty of youth going home to practices and values that conflicted with those the program supported; staff often saw their role as providing relationships, opportunities, and environments as an alternative youth’s home experiences.
Program Context/Infrastructure Many staff and youth described Teen REACH as a safe place. Staff at some sites felt that the program offered an alternative to gangs by providing youth learning experiences in discipline and knowing right from wrong. Some sites provided youth with safe rides home at the end of the day to ensure safety and lessen exposure to the streets and gangs.

Program staff and youth noted transportation as a challenge. Where provided, both youth and staff cited transportation as a great asset. Staff said that it provided safe passage to youth in areas with gang-related problems. A related concern is that many youth and staff commented on the poor condition of vans used for transportation.

Evaluators reported feeling a sense of safety and staff supervision when onsite, especially at the three sites with staff ratios of six youth to every one staff member. At these sites, there was a sense that youth were “wrapped in a net” of adult concern and availability.

At four of the sites visited, focus group youth frequently expressed their sense of belonging in the program. At the fifth site, strong connections were made between youth and individual staff members, but were not as strong overall. A sense of belonging was reflected in some youth’s descriptions of the program (e.g., “it’s like home”) and the positive peer relationships established in the program (e.g., “the first day I came here, I met a lot of friends”). Staff also noted youth’s sense of belonging, reporting that youth often came back to check in and that those who had graduated from the program often returned to work with younger youth.

Many program staff and youth cited a need for improved availability of appropriate space. Staff and youth at one site said it was difficult to build their program because they only had one room that was exclusively theirs, and that room had to store their materials. Another site often could not use its computer room because it was not adequately air-conditioned.

Staff and youth mentioned physical site maintenance as an area in need of improvement; evaluators noted only two of the sites as being clean and well kept. Some staff said that a well-maintained site sent a message to youth that they were valued, and that a poorly maintained or unclean facility sent the message that they were not worth nice space.

When asked what needed to be added to Teen REACH to make it better, many youth asked for recreation equipment, especially more basketballs or soccer balls “that worked.”

Four sites had access to a gym, and the boys at these sites continually cited basketball as one of the parts of Teen REACH that was important to them.
Recruitment/Participation According to youth focus groups at all five sites, Teen REACH’s recreational component was a strong draw, and was often the initial and sometimes sole reason for participation.

Some youth said that without transportation, the distance to Teen REACH from their schools or homes would make program participation difficult.

Staff at every site thought that they could serve more youth with sufficient staff and resources.
Staffing/Training Youth at all sites noted staff’s care, concern, and involvement as a core program strength.

Each site visited had at least one, and often many, staff whom youth perceived as caring and important adults in their life. These staff varied in their age, style, culture, ethnicity, expertise, and interests, but all were able to connect with a segment of their site’s youth. Examples include a mother with grown children who helped the “young ones” with their homework and a young man who grew up “on the wrong side of town” who motivated African-American boys to finish their homework so they could play basketball with him.

Both youth and staff at every site identified at least one program champion, a person who was the visible inspiration and advocate for the program and those involved, as the key to their program’s success. The position the champion held varied from site to site.

The evaluators found evidence that programs may have been understaffed, and that staff may have been overtaxed or under-supported. For example, two sites were unable to complete the youth survey process because they lacked a consistent staff member to follow through. In addition, some programs were limited in providing recreational programming because they had too few staff for the number of youth served.

Part-time staff repeatedly said that they might not be able to continue in their jobs due to Teen Reach’s lack of benefits, a need to find full-time work because they were not making enough money on their salaries from Teen REACH plus one or more other part-time jobs, and an inability to see a future for themselves there (e.g., no clear way to get raises, no career path).

Youth at two sites reported frustration that certain staff members had left and had not been replaced. This was frustrating because they missed these particular staff members, certain program aspects could not function without them, and remaining staff were now spread over more youth and did not have as much time for specific relationships as before.

In addition to working with youth, some staff were expected to take on the janitorial and maintenance work at their sites. As a result, staff sometimes had to choose between having a clean and attractive site and working with youth, or doing both and risking burn out.

Some staff felt that they did not have the kind of ongoing training needed. Specifically, staff engaged in tutoring and homework assistance for learning and behavior disabled youth requested training.

Staff at some sites expressed concern over the perceived time drain in fulfilling DHS paperwork and reporting requirements. One site stated that requirements increased each year, demanding more staff time, which was seen as time taken away from direct service to the youth, and as serving no purpose for them or for DHS.
Systemic Infrastructure Staff members from all sites noted difficulties in maintaining, sustaining, and expanding their programs.

Sites that had more success in securing funding beyond DHS funds were those that most successfully implemented the full complement of core services and best practices.

Most staff felt isolated from other Teen REACH sites. Some staff expressed an interest in implementing site-linked programming in which participants from multiple sites would be brought together for certain program components.


Summative/Outcome Findings

Academic No significant differences were found from pretest to posttest on academic measures, nor were academic outcomes correlated with attendance levels or dosage.
Family No significant differences were found from pretest to posttest on parent involvement measures, nor were these outcomes significantly correlated with program dosage. However, the pattern of results indicated that the high-attendance group reported more parental involvement than the other groups.
Prevention No significant differences were found from pretest to posttest on measures of delinquent behavior, nor were these outcomes significantly correlated with program dosage. However, the pattern of results indicated that the high-attendance group reported less delinquency.

At four sites, both youth and staff said that Teen REACH was an alternative to being on the streets and involved with gang activity.
Youth Development No significant differences were found in youth survey responses from pretest to posttest on items of self-concept, leadership skills, problem-solving skills, peer-group cohesion, adult connections, or prosocial behavior.

Program dosage was positively correlated with problem-solving skills (p < .05), self-concept (p < .01), peer-group cohesion (p < .05), and adult connections (p < .01). In addition, attendance levels were positively associated with adult connections (p < .01) and self-concept (p < .05).

Youth at every site noted that the program had a positive effect on their future aspirations. For example, one youth commented that Teen REACH “changed my whole attitude. We get direction here from staff. They tell us how to be, what we’re going to be in for in life.”

Evaluation 2: Teen REACH: Annual Evaluation Report 2002


Evaluation Description

Evaluation Purpose To assess the relationship between program participation and youth educational and socio-behavioral outcomes, to develop local understanding and capacity for program sites to effectively participate in the evaluation, and to provide opportunities for grantees to understand, utilize, and continuously improve their programs through data-based decision making.
Evaluation Design Experimental and Non-Experimental: The Illinois Department of Human Services (DHS) selected 30 providers with programs they judged to be well functioning and likely to cooperate with the evaluation process. The sample was selected to provide geographic representation from all regions in the state, include a range of provider agency types (e.g., community-based organizations, health departments), represent both rural and urban communities, assure cultural and racial diversity among participants, and include newly funded and more established programs.

Qualitative data were collected from 15 of the 30 sites through site observations and staff and youth interviews. Sites visited during the previous year were not visited again. Program staff and youth were selected for interviews using a theoretical sampling approach, which involves selecting informants based on the likelihood that they will provide a rich source of information about the phenomena under study. Thus, a variety of individuals with differing roles in Teen REACH were asked to participate: program directors, program coordinators, various paid program staff, some unpaid volunteer staff, and youth between the ages of 10–18 who were currently participating in Teen REACH. Local staff selected youth on the basis of age, gender, and length of program experience.

Implementation was also assessed through a total of 192 staff surveys completed at 29 of the 30 evaluation sites (56% of the total number of staff surveys requested by each site). Of staff who responded to a survey item about their position in Teen REACH (n = 183), 55% were direct care program staff, 17% were site coordinators, 12% were project directors, 8% were volunteers, and 8% described their role as “other.”

Evaluators used a single-group pretest/posttest design to assess changes in youth outcomes. Four of the 30 sites did not submit completed posttest youth surveys and thus were not included in the data analyses and findings. Data were examined by comparing pretest and posttest data and by exploring the relationship between posttest data and program attendance levels (i.e., high, medium, and low attendance) or dosage (defined in this evaluation as the total number of days youth attended). Program attendance level was measured as high (433 days or more attended over the past three years), medium (81–432 days), and low (1–80 days).

Of the 1,648 youth attending the selected program sites at the time of the pretest, 1,046 youth at 30 sites completed the pretest (64% response rate), and 689 youth at 26 sites completed the posttest. A total of 468 participants (45%) completed both the pretest and posttest (the longitudinal sample). Findings are based on either the pre-post sample or, for program dosage analyses, the full posttest sample.

For the longitudinal sample, the average age at pretest was just over 12, with 50% in Grades 4–6, 30% in Grades 7–9, and 20% in Grades 10–12. This sample was evenly split between girls and boys. The sample was 36% African American, 33% white, 22% Latino, and 12% other. The majority (80%) participated in the free/reduced-price lunch program at school. Forty-one percent lived in single-parent households, 46% lived in two-parent households, and the remaining 13% lived with non-parent adults (e.g., grandparents, foster parents, etc.).

Of the 568 youth who answered questions about program participation at posttest, 20% had been involved for 3 years, 19% for 2 years, 27% for 1 year, and 34% for less than 1 year, with an average of 1.24 years. Of these youth, 10% participated about 6–7 days a week, 34% participated 5 days, 16% participated 4 days, and 40% participated 1–3 days, with an average of 3.9 days. In addition, 19% were classified as high attenders, 56% as medium, and 25% as low. The average total dosage was 259 days, with a range of 4 to 1,008 days.
Data Collection Methods Interviews/Focus Groups: During site visits, evaluators conducted staff interviews and youth participant focus groups to gain a deeper understanding of the program’s impact. At each site, 4–6 staff members and 5–10 youth participated in interviews. Group interviews were scheduled with the program director, program coordinator, and selected staff at each site. Staff who could not participate in the group interview were interviewed individually. All staff were asked about their program goals, whether the program was being implemented as originally planned, program changes they would like to make, implementation obstacles encountered and how they were handled, program successes, evidence that the program was working, whether the program differed in its effectiveness for particular groups, program improvement suggestions, and program strengths and weaknesses. Youth participants, ages 10–18, were asked about their best experience in the program, what they would keep in the program, what they would add or take away to improve the program, and anything else that would help explain how the program runs.

Observation: Evaluators conducted 4–6-hour site visits in May and June 2002 at 15 of the 30 evaluation sites. Evaluators observed selected programming and activities.

Surveys/Questionnaires: Youth in grades four and above at the selected sites were surveyed in October 2001/January 2002 (pretest) and again in May/June 2002 (posttest). The survey assessed youth background/demographic information, program exposure/dosage, perceived connections with staff, perceived program impacts, and youth outcomes. Youth outcome areas included frequency of homework completion, school attendance (days absent in the past month, and reasons for these absences), academic performance (self-reported grades), academic performance aspirations (e.g., “how important is it to you to graduate from high school?”), quality of school life (e.g., frequency with which youth are happy when they are in school), engagement in prosocial and delinquent behaviors (e.g., “in the last 6 months, how often did you tell lies or cheat?”), substance use (reports of alcohol, tobacco, and drug use during the past 30 days), leadership (e.g., “I like to set a good example for other young people”), self-concept (e.g., “I am happy with myself as a person”), adult connections (e.g., “I know people/places in my community where I can get help with a personal problem”), peer-group cohesion (e.g., “I enjoy the time I spend with my friends”), problem-solving skills (e.g., “when I face a new problem, I first try to find out what it is”), and parent involvement in child’s education and Teen REACH (e.g., the frequency with which parents visit the Teen REACH program or their child’s school).

Administered in April/May 2002 at 29 of the 30 sites, the staff survey assessed staff roles, characteristics, and responsibilities; organizational issues; perceived programming focus; and program impacts on youth.
Data Collection Timeframe Data were collected during the 2001–2002 school year.


Findings:
Formative/Process Findings

Activity Implementation Staff rated their programs as having the highest focus on academics (providing help with homework, stressing the importance of education, tutoring in certain subjects) and life skills (getting along with peers, conflict resolution, general problem solving). Parent- and family-focused programming was not rated as highly as a program focus.

Only 1 of the 15 sites visited had a mentoring program that (1) lasted for the whole year; (2) formally matched one mentor with one youth; (3) provided training, ongoing support, and recognition to mentors; and (4) provided space and time for the mentor program each week.

Staff and youth often cited homework help as a program strength. Staff at all sites said that many participants’ parents worked and often did not have the time, energy, or educational background to help with homework. So according to many respondents, if homework was not completed at Teen REACH, it would not get done.

Some youth noted that the program exposed them to new and different activities, such as participating in statewide youth leadership conferences, attending and producing plays, and visiting major museums. Staff and youth indicated that participants’ families rarely provided such experiences. Further, some urban youth felt they could not safely leave their neighborhood, while some rural youth had never been to a large city prior to the program.
Parent/Community Involvement Most sites saw parent involvement as the most difficult area to implement. Impediments included parents’ inability to attend events due to work schedules and childcare needs for their other children. Some families did not have phones, so staff could not easily contact them. In addition, some families did not cooperate with staff; one site staff member said that the program staff would phone parents when youth got in trouble, but parents did not address the problems.

Many sites described strong relationships with other groups in their communities, such as structured relationships with local police, 4-H, the park system, University of Illinois Extension, local hospitals, community colleges, universities, and local businesses.

In response to an open-ended question about program impacts, some staff noted improved parent involvement.
Program Context/Infrastructure A major program strength observed across the 15 sites was the adaptability to local needs and resources. The program’s core areas were similar throughout the state, but local staff seemed to embed the program in their particular communities. For example, staff at a particularly diverse inner city program used their youth’s different cultural groups as themes around which they created activities to promote appreciation and understanding of differences.

Many youth and staff reported that the program was a safe place, particularly among inner-city sites (but also mentioned at some rural sites). For most youth and staff, safety meant adult supervised activities. For others it meant keeping youth away from gangs and gang influence.

The provision of appropriate space, site maintenance, transportation, and materials varied greatly across the sites visited.
Program-School Linkages Staff reported strong relationships with schools, particularly sites that worked closely with local schools that had an enthusiastic and supportive school administration.
Staffing/Training Of staff that responded to a survey item asking them to rate the extent to which their site’s staff agreed on the overall program vision (n = 182), 45% reported complete agreement, 50% reported some agreement, 5% reported very little agreement, and one respondent indicated no agreement. Sites with the highest average agreement levels were also those with the highest staff job satisfaction levels.

The majority of surveyed staff had some postsecondary education (ranging from some college with no degree to graduate/professional degree), including 86% of program directors, 94% of site coordinators, and 72% of direct care staff.
Sixty-three percent of program directors, 80% of site coordinators, and 45% of direct care staff had at least 5 years of experience working with youth.

Of staff responding to the survey item on length of employment at Teen REACH (n = 183), 35% had worked there for more than 2 years, 22% for 1–2 years, and 43% for less than 1 year. The average was 1.52 years.

Just over one-fourth of staff responding to the survey (28%) indicated that staff turnover was a moderately serious or serious issue at their site.

Project directors viewed their roles as primarily administrative, indicating that tasks such as monitoring the numbers of youth served by the program and supervising staff were the most important parts of their roles. Also rated as highly important were liaison roles, such as educating the community about Teen REACH, and communicating with schools. Site coordinators viewed communicating with schools, working directly with youth, and generating program and activity ideas as their most important roles. Direct care staff indicated that their most important roles involved direct contact with participants, i.e., working with youth in groups and individually and establishing meaningful one-on-one relationships with youth. Staff in all three types of positions indicated that increasing their knowledge through ongoing professional development was a key responsibility.

Staff questioned their level of preparedness in several areas that they considered very important for their jobs. These areas were generating program and activity ideas (site coordinators and direct care staff), program evaluation (program directors), working with families (site coordinators), and establishing/maintaining an advisory board (program directors).

The areas of training that the majority of staff reported needing more of included working with special needs youth, increasing parent involvement, and using recreation to manage and teach behavior. Project directors, in particular, noted the need for additional training in curriculum design and research-based best practices.

Staff tended to report their colleagues and pride in their work as aspects of their jobs they liked best. Staff were least satisfied with salary, benefits, and promotion opportunities.

In response to an open-ended question asking what might make their jobs more satisfying, some staff indicated a need for increased funding and increased salary or benefits. Other themes included the need to hire additional staff (especially staff that could relate to youth), improve staff communication/coordination, increase parent involvement, increase program rules/structure/consistency, provide more staff development/training, serve more youth/expand the program, serve different groups of youth, decrease paperwork, increase cooperation from schools, increase the length of the funding cycle, and increase community involvement.

In 14 of the 15 sites, there was a clear “cornerstone” staff member for that site. These staff had a passion for reaching out to youth and had dedicated themselves to making the program work. Specifically, they worked to obtain additional program resources, create new and interesting projects for participants, and meet with school personnel or parents.

Staff repeatedly mentioned the increasing amounts of time needed to complete paperwork required by their various funding sources. They felt that these administrative requirements took away valuable staff time with youth. Staff also commented that the staff who were more effective with youth tended to also be the ones best qualified to complete paperwork.
Systemic Infrastructure Staff at all sites noted difficulties in maintaining, sustaining, and expanding their initiatives. In particular, program directors and coordinators noted the amount of time and creativity they spent to find enough resources to grow or even just maintain the program.

Sites that had more success in securing funding beyond the DHS grant were those that most successfully implemented the full complement of core services and best practices.


Summative/Outcome Findings

Academic Program dosage and attendance level were significantly correlated with both youth perceptions of the amount learned from program participation and with school absences (p < .01 each for perceived learning; p < .01 and p < .05, respectively for school absences). For amount learned from the program, correlations were in the predicted (positive) directions. However, for school absences, the relationship was in the opposite direction expected, with increases in program dosage and attendance levels relating to increases in absences. There were no other significant correlations between program dosage or attendance levels with other academic outcomes assessed (i.e., homework completion, self-reported grades, academic aspirations, and perceived school environment).

In response to an open-ended question about their experiences in Teen REACH, some youth mentioned the benefits they got from help with homework/better grades (e.g., “I’ve been getting higher grades because they help me”).

In response to an open-ended question about program impacts, many staff focused on improved academic performance (e.g., “I have watched and assisted several of our participants increase their grades from C’s, D’s, and F’s to A’s and B’s and making the school honor roll in the year or two [that they were] in our program”).
Family Both program dosage and program attendance level were significantly positively correlated with parent involvement in their child’s education (p < .01 and p < .05, respectively).
Prevention Neither program dosage nor program attendance level was significantly correlated with negative/delinquent behavior.
Youth Development Program dosage was significantly positively correlated with leadership skills (p < .05) and with adult connections (p < .01). There were no other significant correlations between dosage and youth development outcomes assessed (i.e., self-concept, problem-solving skills, and peer-group cohesion).

Program attendance level was positively correlated with leadership skills, peer-group cohesion, and adult connections (p < .05 for each). There were no other significant correlations between attendance levels and youth development outcomes assessed (i.e., self-concept and problem-solving skills).

In response to an open-ended question about program impacts, many staff noted improved participant attitudes and behavior, e.g., “One of our students was new to our school this year; she was very unhappy, unmotivated, and a discipline problem. By the end of the year she was happy, had made a lot of friends, and was absolutely not a discipline problem.” Staff also noted that youth had improved self-concept, e.g., “We have a girl in the sixth grade, who was very shy; she now has more self-esteem, because she is singing in our yearly talent show by herself and without music. She has really come out of her shell.”

Evaluation 3: Teen REACH: Annual Evaluation Report 2003


Evaluation Description

Evaluation Purpose To assess the relationship between program participation and youth educational and socio-behavioral outcomes, to develop local understanding and capacity for program sites to effectively participate in the evaluation, and to provide opportunities for grantees to understand, utilize, and continuously improve their programs through data-based decision making.
Evaluation Design Quasi-Experimental: The Illinois Department of Human Services (DHS) selected 30 program providers that had programs they judged to be well functioning and likely to cooperate with the evaluation process. The sample was selected to provide geographic representation from all regions in the state, include a range of types of provider agencies (e.g., community-based organizations, health departments), represent both rural and urban communities, assure cultural and racial diversity among participants, and include newly funded and more established programs. Each provider selected one program site to participate in the evaluation. These 30 sites included all 7 who participated in the pilot evaluation. Two of these 30 sites did not submit completed pretest and/or posttest youth surveys and thus were not included in the data analyses.

Evaluators used a single-group pretest/posttest design to assess changes in youth outcomes. Data were examined by comparing pretest and posttest data and by exploring the relationship between posttest data and program attendance levels or dosage (the total number of days youth attended). Pretest data were collected from 594 youth participants (out of 829 on program rosters for a response rate of 72%) and at posttest by 496 youth participants (out of 939 on program rosters for a response rate of 54%). The posttest response rate may be a low estimate because many youth on the program roster likely dropped out of the program by the time surveys were administered. A total of 332 youth completed both the pretest and posttest (the longitudinal sample). The analyses are based on either the longitudinal sample or, for some analyses, the full posttest sample.

Of the longitudinal youth sample, the average age was just under 13 years old, with 46% in Grades 4–6, 42% in Grades 7–9, and 13% in Grades 10–12. More than half were female (57%). The majority was African American (48%); the remainder were white (23%), Latino (20%), and other (11%). Eighty percent participated in the free/reduced-price lunch program at school. In terms of family composition at posttest, 57% were in two-parent families, 33% were in single-parent families, and 10% lived with non-parent adults (e.g., grandparents, foster parents, etc.).

Of participants who completed posttest surveys, 14% had been involved with Teen REACH for 3 or more years, an additional 14% for at least 2 years, 23% for at least 1 year, and 49% for less than 1 year, with an overall average of 1.03 years. At posttest, 54% of youth reported attending Teen REACH an average of 5 or more days a week, 20% for 4 days, and 26% for 1–3 days. The number of years in the program and the typical number of attendance days per week were used to create an estimate of total program dosage. This dosage is not just for the current program year, but takes into account the duration, in representing time since youth first started attending. Some analyses examined the relationship between dosage and change over time in the key outcome areas. Only the statistically significant correlations of dosage with outcomes are reported in the findings.

Staff surveys were collected in March 2003 from staff at 23 sites that were new to the statewide evaluation in the current year; the 7 sites continuing in the evaluation from the previous year were not asked to submit staff surveys again. Including the previous year’s data, surveys were collected from 135 staff at 29 of the 30 evaluation sites. (One site did not submit surveys.) Based on the number of staff sampled by each site, this represents a response rate of approximately 49%. Of staff who answered a question about their role within Teen REACH (n = 130), 53% were direct care staff, 18% were site coordinators, 16% were project directors, 2% were volunteers, and 11% classified themselves as “other.”
Data Collection Methods Surveys/Questionnaires: Youth in Grades 4 and above at the selected sites were surveyed in October/November 2002 (pretest) and again in April through June 2003 (posttest). The survey instrument included items and scales to assess: youth background/demographic information, program exposure/dosage, perceived connections with staff, perceived program impacts (the frequency with which specific topics are addressed in Teen REACH and perceptions of program benefits), and youth outcomes. Youth outcome areas included frequency of homework completion, school attendance (days absent in the past month, and reasons for these absences), academic performance (self-reported grades), academic aspirations (e.g., “how important is it to you to graduate from high school?”), quality of school life (e.g., frequency with which youth are happy when they are in school), engagement in prosocial and delinquent behaviors (e.g., “in the last 6 months, how often did you tell lies or cheat?”), substance use (reports of alcohol, tobacco, and drug use during the past 30 days), leadership (e.g., “I like to set a good example for other young people”), self-concept (e.g., “I am happy with myself as a person”), adult connections (e.g., “I know people/places in my community where I can get help with a personal problem”), peer-group cohesion (e.g., “I enjoy the time I spend with my friends”), problem-solving skills (e.g., “when I face a new problem, I first try to find out what it is”), and parent involvement in child’s education and Teen REACH (e.g., the frequency with which parents visit the program or their school).

Administered in March 2003, the staff survey assessed staff characteristics, roles, and responsibilities; organizational issues; programming focus; and program impacts on youth. To assess program focus, staff were asked to rate potential areas in which their Teen REACH programs tended to focus: academics, youth development, sports/recreation, and parent involvement/family programming. Each area was rated on a 4-point scale from not at all a focus to a major focus of their program. These data were aggregated at the site level to produce an indicator of how much the site focused on that area. This information was linked to youth survey data at the same site to assess the relationship between program focus and youth outcomes.
Data Collection Timeframe Data were collected during the 2002–2003 school year.


Findings:
Formative/Process Findings

Activity Implementation Program focus areas that staff rated the highest were academics (providing homework help, stressing the importance of education, tutoring in certain subjects) and life skills (getting along with peers, conflict resolution, general problem solving). Also rated highly were recreational activities and opportunities to establish positive caring relationships with adults. Parent- and family-focused programming areas were not rated as highly as a program focus.

At programs with a stronger academic focus, youth reported significantly higher posttest survey ratings on how much they learned academically, feelings about school, and academic aspirations (p < .05 for each) than youth in programs with less of an academic focus. No significant differences were found for self-reported grades or homework completion.

Youth in programs with a stronger youth development focus reported significantly higher posttest survey ratings on how much they learned in this area (p < .05) than youth in programs with less of a focus in this area. There were no significant differences for ratings of other youth development-related outcomes (self-concept, leadership skills, problem-solving skills, peer-group cohesion, adult connections).

No significant differences were found between programs with a high or low sports/recreation focus on any of the areas assessed on the posttest youth survey that evaluators felt might affect sports/recreation-related outcomes (self-concept, leadership skills, problem-solving skills, peer-group cohesion).

Youth in programs with a stronger parent involvement/family focus reported significantly higher posttest survey ratings on parent involvement in youth’s education (p < .05) than youth in programs with less of a focus in this area.

Of surveyed staff, 37% said that most youth and families received the services they needed.


Summative/Outcome Findings

Academic Significant increases were found from pretest to posttest in ninth to twelfth grade participants’ academic aspirations and the quality of school life (p < .05 for each). No significant differences were found in any academic outcome areas for fourth to eighth grade participants.

Over half of participants’ outcomes increased or stayed the same from pretest to posttest in self-reported grades (71%), academic aspirations (72%), and school life quality (51%).

For fourth to eighth graders, higher dosage levels were significantly correlated with increases over time in academic aspirations (p < .01) and positive feelings about school (p < .05).

For the fourth to eighth graders in their first year of Teen REACH, higher dosage levels were significantly correlated with more positive changes over time in homework completion and academic aspirations (p < .01 for each).

For ninth to twelfth graders in their first year of Teen REACH, higher dosage levels were negatively correlated with homework completion (p < .05).

When asked about program benefits, a majority of youth reported that they learned a lot about completing homework (62%), getting help with certain subjects in school (52%), and studying for tests (51%). Similarly, some youth provided open-ended survey responses that indicated that the program helped them with their homework and to get better grades.

A number of the surveyed staff reported that most participants had the following outcomes: completed homework more often (58%), had better school attendance (44%), did better in school/made better grades (37%), and improved their attitudes toward school (27%).

In response to an open-ended question about program impacts, many staff reported improved academic performance (e.g., “when I started with him, he was not reading chapter books, and now five months later, [he] can”).
Family For the total sample of youth, 51% of parents maintained or improved their levels of school involvement from pretest to posttest. Significant decreases were found from pretest to posttest in fourth to eighth grade participants’ parent involvement in education (p < .01). The differences for ninth to twelfth graders in this area were not significant.

For ninth to twelfth grade participants in Teen REACH for the first time, higher dosage levels were negatively correlated with parent involvement (p < .10).

Twenty-five percent of surveyed staff indicated that most youth involved in the program increased their positive interactions with their parents.

Of surveyed staff, 22% reported that most parents involved in the program knew more about the resources available in the community, and 21% indicated that most of these parents were more involved in their child’s education and knew more about how to help their child do well in school.
Prevention For fourth to eighth graders and ninth to twelfth graders in their first year of Teen REACH, higher dosage levels were significantly correlated with reduced negative/delinquent behavior (p < .05 for fourth to eighth graders and p < .10 for ninth to twelfth graders).

When asked about program benefits, a small majority of youth reported that they learned a lot about the dangers of drugs (53%) and the skills needed to avoid drugs (51%).

Of the staff surveyed, 36% reported that most participants were less likely to become involved with drugs, engaged in less negative behavior/fewer discipline problems, and had stronger anti-drug attitudes. In addition 33% of staff indicated that most youth decreased their drug use, and 16% reported that most youth increased their knowledge of reproductive and sexual health issues.

In response to an open-ended survey question asking about youth’s program experiences, some youth said that it gave them somewhere to go, or something to do (e.g., “it keeps me off the streets and out of trouble”).
Youth Development Significant increases were found from pretest to posttest for ninth to twelfth grade participants in peer group cohesion and adult connections (p < .05 for each). No other significant differences in youth development outcomes were found for ninth to twelfth grade participants.

Significant increases were found from pretest to posttest in fourth to eighth grade participants’ delinquent behavior (p < .01). No other significant differences in youth development outcomes were found for fourth to eighth grade participants.

From pretest to posttest, a number of participants’ outcomes either improved or stayed the same, including adult connections (59%), self-concept (55%), peer-group cohesion (55%), leadership (53%), and negative/delinquent behavior (47%).

For participants in Grades 9–12, a significant positive correlation was found between program dosage and the development of leadership skills (p < .01).

For fourth to eighth graders in their 1st year of Teen REACH, higher dosage levels were significantly correlated with more positive changes over time in the areas of problem solving and peer-group cohesion (p < .10 for each).

For ninth to twelfth grade participants in Teen REACH for the first time, higher dosage levels were negatively correlated with self-concept (p < .05) and adult connections (p < .10).

In response to an open-ended survey question about youth’s program experiences, some youth indicated that the program helped them further their education/plans for the future/goals (e.g., “Teen REACH has helped me see that I am highly intelligent and can be something in life”). Some youth also indicated that the program helped improve their relationships with peers and friends. In addition, some youth indicated that the program exposed them to new activities (e.g., “I have a good time at trying other things”).

Of surveyed staff, the following percentages reported that most program youth had positive outcomes: exposure to new activities/places/events (61%), increased happiness (56%), increased prosocial/recreational interests (51%), established positive relationships with caring adults (44%), more self-confidence/improved self-esteem (42%), getting along better with others (38%), improved physical/sports skills (37%), improved problem-solving and decision-making skills (35%), increased leadership skills (33%), improved health-related habits (exercise, hygiene; 28%), and set goals for the future (19%).

When asked about program benefits, a number of youth reported that they learned a lot about the following areas: setting goals for the future (51%), solving problems and making good decisions (51%), getting along with others (48%), preparing for a future job or career (46%), how to be a better leader (45%), dealing with conflicts with others (42%), the importance of exercise and hygiene (40%), and learning about other cultures (30%).

In response to an open-ended question about program impacts, many staff focused on improved participant attitudes and behavior, e.g., “There is one young man that I have observed who experiences an amazing transformation. A couple of years ago I considered him aloof and bordering on disrespectful. Today he is in a position of leadership teaching younger children. He is positive and polite.” Some staff also commented on youth’s improved self-concept, e.g., “There was a child who starting Teen REACH was very shy. He now is able to open up, share his thoughts and ask others to play with him. His attitude towards others is happy and welcoming.”

© 2016 Presidents and Fellows of Harvard College
Published by Harvard Family Research Project