The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.
Volume IX, Number 1, Spring 2003
Issue Topic: Evaluating Out-of-School Time
Ask the Expert
How can research and evaluation improve practice regarding access and equity in out-of-school time programs?
The growing diversity of out-of-school time program participants nationwide has generated increased interest in understanding the issue of access and equity in youth programming. These issues refer to questions of who has access to and is attending out-of-school time programs, and whether different subgroups of participants (e.g., different ethnic groups, girls, youth with disabilities, etc.) are getting the services they need to be successful. Five experts in the field of youth development and out-of-school time programming offer their perspectives on how research and evaluation can improve access and equity in out-of-school time programming.
Yolanda George, American Association for the Advancement of Science
Researchers and evaluators should examine the impact of out-of-school time programs on different subgroups of students to find out what works for whom and in what context. This requires the collection of both demographic data on student achievement and data on process variables, such as curriculum, teaching practices, or types of intervention strategies that might impact achievement. Many studies provide achievement information on groups, dividing students along ethnic and racial lines and breaking down data by gender, but most do not examine process variables to determine whether specific strategies or practices are equally effective for all students.1
In addition, few research studies include state test scores for American Indians, students with disabilities, those with limited English proficiency, or students from migrant or homeless families. Students in these groups are excluded from state tests because they do not attend school often enough or they do not reside in the same district for a full academic year. In many studies data about these subgroups is not reported at all, or if reported, may not be reliable because the sample size of these populations is usually too small to report statistically meaningful data.2 Before educators can examine what out-of-school time strategies work for these subgroups, they will need to find new ways to track and examine their academic progress.
Laurie Olsen, California Tomorrow
Diversity has become the norm in after school program enrollment nationwide. Young people’s cultural, ethnic, racial, and other identities, and their family, language, and community background are core to who they are and how they learn, as well as to what support they may need. Yet many out-of-school time program staff members are now working with some youth with whom they do not share a background or identity. To develop quality programming, research is key.
First, it is imperative that program staff have access to research on how culture and language enter into the ways in which young people experience and process life, and to the findings and frameworks being created by the research community working on these issues. Second, in order to provide meaningful research that can inform programs on access, equity, and diversity, researchers need to incorporate questions, frameworks, and quality indicators that recognize the unique access and equity challenges in diverse communities. Third, program staff need the skills and practices to engage in research of their own—to structure inquiries and action research to better understand the young people and the community they serve. This includes systems for data collection and disaggregation of data to examine the different experiences of different groups of young people, a familiarity with how to structure and use various youth voice formats, and how to collect and make sense of data about the participation and experiences of young people in the program.
Jane Quinn, The Children’s Aid Society
Research has already been helpful in improving both practice and policy regarding access and equity, and new researchers can learn from this work and build on it. Over 10 years ago, when I worked on the Carnegie Corporation of New York’s study on youth development and community programs (A Matter of Time: Risk and Opportunity in the Nonschool Hours3), I found several research studies that documented important problems related to equity and access, and these findings helped us make the case for expanding programs and services for low-income and minority youth.
For example, the federally sponsored National Education Longitudinal Study clearly established the fact that low-income and minority youth had much less access to youth development programs during nonschool hours than their more affluent peers.4 A Chapin Hall study comparing the availability of youth programs in suburban and inner-city neighborhoods showed similar patterns of fewer programs with less substantive content.5 More recently, Milbrey McLaughlin’s research in several urban neighborhoods addressed a different part of the equity agenda. In Community Counts, McLaughlin noted a significant and consistent pattern of under-service to girls.6 These studies show that serious, well-regarded researchers with an equity and access perspective to their investigations can provide reliable information to youth advocates to promote quality youth programming.
Related ResourcesBhattacharya, J., Jaramillo, A., Lopez, L., Olsen, L., Scharf, A., & Shah, M. (2002, October 1). Our roots, our future: Affirming culture and language in after school and youth programs. Oakland, CA: California Tomorrow. www.californiatomorrow.org/ Fancsali, C. (2002). What we know about girls, STEM, and afterschool programs. New York: Educational Equity Concepts. Fink, D. B. (2001). Making a place for kids with disabilities. Westport, CT: Bergin & Garvey. www.includeallkids.com/mkpd James, D. W., Jurich, S., & Estes, S. (2001). Raising minority academic achievement: A compendium of education programs and practices. Washington, DC: American Youth Policy Forum. www.aypf.org/publications/ Sanders, J., & Tescione, S. M. (2002). Gender equity and technology. In J. Koch & B. Irby (Eds.), Defining and redefining gender equity in education. Greenwich, CT: Information Age Publishing. |
Liz Reisner, Policy Studies Associates
The evaluation of the After-School Corporation (TASC) programming has yielded findings that have helped the program better target and deliver its school-based, community-linked after school services.7 The evaluation, which is assessing program implementation and student results in 96 TASC projects in New York City, has determined that TASC services are delivered in the schools whose students exhibit particularly serious educational needs. The evaluation has found that the projects are serving representative cross-sections of each school’s population, in terms of prior achievement, family income, race/ethnicity, and gender and that projects do so without screening or using ability grouping.
TASC-related educational benefits include improvements in school attendance and math achievement among participants, especially among those who attend on a frequent, extended basis. Gains have been greatest among low-income students, African-American and Hispanic students, students with disabilities, and English language learners. Although students with disabilities were initially underrepresented, their program enrollment has increased over time as projects have expanded their outreach to these students and made arrangements in some instances for specialized staffing.
The TASC program is using these results in both its training and technical assistance activities, and to inform school and project staff about the benefits of regular, multi-year participation, especially for students with the greatest educational needs. Evaluation results have given school principals and local nonprofit organizations the confidence to keep projects open to all students and to expand the model to additional schools, including the Chancellor’s District schools, which have students with the most pressing educational needs.
Donna Walker James, American Youth Policy Forum
Over the past six years, the American Youth Policy Forum has collected youth program evaluations and produced three- to five-page summaries in readily accessible language. These summaries have been compiled in four compendia to date, with volumes in production on out-of-school time programs and family involvement.8
Our 2001 compendium, Raising Minority Academic Achievement, yielded important findings related to access and equity.9 Practices that were effective in leading to improved academic achievement outcomes for African-American, Latino, and Native-American youth included quality leadership and implementation, academically demanding curriculum, professional development, family and community involvement, reduced student-to-teacher ratios (small classes, small schools, small learning communities, and extra help from tutors and mentors), individualized support, extended learning time, and long-term supports.
Our findings also indicate that research is critically important to quality programming and that access and equity cannot be ensured unless data are collected and made available to others. Only by disaggregating and sharing data on academic achievement and other youth outcomes can we be sure that programs are accessed by all population groups and that all groups are successful within the programs. As gaps are identified, long-term, individualized supports can be specifically targeted to ensure that access, equity, and academic achievement goals are met.
Priscilla Little, Project Manager, HFRP
Hayley Yaffe, Student Intern, HFRP
1 B. C. Clewell, P. B. Campbell, Y. S. George, and E. Jolly, National Science Foundation (personal communications, June 2000).
2 Office of the Deputy Secretary, Planning and Performance Management Service, U.S. Department of Education. (2002, March). U.S. Department of Education strategic plan. Washington, DC: Author.
3 Carnegie Corporation of New York. (1992). A matter of time: Risk and opportunity in the nonschool hours. New York: Author.
4 National Center for Education Statistics, Office of Educational Research and Improvement, U.S. Department of Education. (1990). National Education Longitudinal Study of 1988: A profile of the American eighth grader (pp. 50–54). Washington, DC: U.S. Government Printing Office.
5 Littell, J., & Wynn, J. (1989). The availability and use of community resources for young adolescents in an inner-city and suburban community. Chicago: Chapin Hall Center for Children, University of Chicago.
6 McLaughlin, M. W. (2000). Community counts: How community organizations matter for youth development. Washington, DC: Public Education Network.
7 Reisner, E. R., White, R. N., Birmingham, J., & Welsh, M. (2001). Building quality and supporting expansion of after-school projects: Evaluation results from the TASC after-school program’s second year. Washington, DC: Author. A summary of this evaluation is included in the HFRP Out-of-School Time Program Evaluation Database.
8 The AYPF compendia are available in Acrobat format at www.aypf.org.
9 Out-of-school time programs included in this compendium are Sacramento START and Boys and Girls Clubs of America; programs with out-of-school time components are Gateway to Higher Education, I Have a Dream, KIPP Academies, Sponsor-a-Scholar, and Upward Bound.