Volume XII, Number 1 & 2, Fall 2006

Issue Topic: Building and Evaluating Out-of-School Time Connections

Questions & Answers

A Conversation With Audrey Hutchinson

Audrey Hutchinson

Audrey Hutchinson is the program director of education and after school initiatives at the National League of Cities (NLC) Institute for Youth, Education, and Families, a national resource that helps municipal leaders take action on behalf of children, youth, and families in five areas—education and after school, youth development, early childhood, child safety, and family economic success. The Institute assists leaders by conducting research on key challenges, disseminating promising practices, and building networks of local officials working on similar issues. Prior to joining NLC, Hutchinson held several senior positions at the White House under the Clinton administration and at the U.S. Department of Education. She has also held positions at the City University of New York, where she focused on strengthening partnerships between colleges, communities, and city and state governments, and was special assistant and policy analyst for the president of the New York City Council. She holds master's degrees in both social work and public health from Columbia University.

In the after school arena, do cities embrace the concept of complementary learning and link institutions to collectively improve outcomes for children and youth?

The Institute for Youth, Education, and Families recognizes the unique and influential roles that mayors, city council members, and other local leaders can play in strengthening families and improving outcomes for children and youth. We help municipal officials bring organizations together across the community to partner on common issues such as after school. We find that city officials welcome the notion at the heart of our work and at the heart of complementary learning—that partnering will result in better after school systems and outcomes.

What challenges are there in evaluating linked after school services? What solutions have cities used to address them?

At least six challenges come to mind:

1. Financing the evaluation. Often cities allocate dollars for after school programming and staff but not for evaluation. To address this, they need to partner with institutions like schools and universities, which have resources that can be used for evaluation. In cases where cities support evaluation, resources could range from Community Development Block Grants (CDBG), to general funds, to in-kind support to help leverage funds to promote and support evaluation efforts. We support pooled funding for after school systems in general, and evaluation is an important piece of that pie.

In the first year of the evaluation of the Fort Worth After School program, foundation funding helped to pay for a researcher from Texas A&M University, while in subsequent years, the local school district paid for continued evaluation. In Grand Rapids, the Office of Children, Youth and Families has played a key role in orchestrating the evaluation efforts, securing grants, writing the request for proposals (RFP), ensuring a city commission resolution on how the money is being spent. That city was successful in using CDBG, general funds, other funding sources to support the staffing of the Expanded Learning Opportunities (ELO) Network, which includes the evaluation efforts.

2. Access to data. The Privacy Act makes it difficult for nonschool evaluators to access student data. Because schools play such an important role in providing after school services, those data are key to understanding the complete picture of citywide after school services and outcomes. There are solutions to this challenge. For example, in Indianapolis, the After School Coalition of 30 organizations hired interns from the University of Indiana who were able to access student data for the evaluation. In Fort Worth, the evaluation team included an individual from the school district who had permission to access student data. Louisville, Kentucky, bought the Kid Tracks software to give school districts and after school programs joint access to data.

3. Accountability for outcomes. Organizations are accountable for different outcomes, and therefore they measure different things. Even when partners create joint programs, they may have different ideas about outcomes. In Fort Worth, city and school district elected officials established a joint after school program called Fort Worth After School. They agreed on a common set of outcomes that included building developmental assets in children, increased attendance, and crime reduction. Their aim was to keep kids in structured enrichment programs, which could provide a safe and enriching environment for children in the after school hours. They felt the program should not be accountable for increasing state test scores, and so did not measure them. (They did, however, ask teachers and principals whether participating students were doing better in schools.) Their agreement about accountability meant that after school evaluation within the 52 programs focused on the same outcomes.

4. Data collection. Different stakeholders have different data needs. A mayor may want data on crime; schools may want data on academic achievement; parks and recreation may want data on obesity. So, they come to the table collecting data in different ways. To address this, we encourage partners to think about their data collection as part of a larger system and to coordinate and link data collection efforts. For example, parks and recreation departments traditionally do not see their work as connected to academic outcomes and therefore do not track demographic data on students. Consequently, student participation in parks and recreation programs cannot be linked to student participation in other programs and services that as a whole likely impact achievement. When viewed as part of a larger system, partners such as parks and recreation or faith-based organizations become more willing to collect data that can be linked and used to assess system-wide outcomes.

Confidentiality and privacy issues complicate data collection. The City of Grand Rapids ELO Network formed an Evaluation Advisory Team representing organizations such as United Way, the county, the police department, the school district, the university, and the city to develop a shared understanding of how each organization would contribute time and resources with the goal of studying the impact of after school on community indicators. This team also developed the RFP for an evaluator and will supervise and monitor the evaluation design for the Network.

5. Ensuring objectivity. This is critical for organizations that partner. Hiring a third-party evaluator to create data collection tools that include everyone's perspectives and balance questions to get true results is a solution to this challenge—especially if trust is an issue, which often happens when diverse partners come together. Evaluators who use a participatory approach and gather input, feedback, and buy-in help build trust and avoid the potential tension involved when an “outsider” is brought in.

6. Institutional cultures and turf battles. These conflicts prevent a shared vision of success. The National League of Cities spends a lot of time making sure that partners develop a common vision for what after school or complementary learning activities will deliver. We also encourage key stakeholders who are part of a citywide effort to build after school systems to define clear roles and responsibility for each partner. For example, mayors or prominent city leaders can function as neutral third parties to convene stakeholders and help open up the conversation. In this way, municipal leaders can play a role in both building the after school system and setting the stage for evaluation.

What evaluation findings are needed to convince city officials that after school efforts pay off?

The evidence has to be linked to what city leaders care about—outcomes like improved quality of life, public safety, economic development, and workforce productivity. While municipal officials care about academic outcomes, it helps to link after school benefits to their broader citywide agendas. For example, city leaders are concerned about public safety. They need to know that after school programs keep kids safe—off the streets and away from crime.

City officials also need cost data. For example, Indianapolis found that the city spends $66 per day per youth on juvenile delinquency costs. After school proponents made the case that it would cost less to provide a quality after school program that kept kids safe while also supporting their learning and enrichment. Data that make the case for an improved workforce also are important. We need to address the issue of absenteeism and lower productivity among working parents who worry that their kids are not safe. If kids are not in environments like after school programs, where parents know they are occupied and off the streets, what happens to productivity after school lets out?

At the same time that we link to leaders' broader agendas, we need to set realistic expectations. We have to be careful not to set up after school programs as the sole solution to all social problems. For example, after school programs in and of themselves do not eliminate crime, but they certainly help with factors that contribute to crime such as truancy.

Finally, city leaders need to know that organizations that run after school programs have a track record of success. If city officials are going to invest in these programs, there has to be evidence that the programs and the organizations that run them produce results.

What recommendations do you have for building a complementary learning evaluation agenda?

We need to know that partnering for results gets more bang for the buck than going it alone. Policymakers need to know that collaboration and linking resources reduce duplication and waste. They need to know that kids are better off when institutions partner and leverage their resources around a complementary learning agenda.

In addition, numbers are great but they are not sufficient. For example, one third of any audience will read a full evaluation report, but two thirds will want to hear the stories of how children and families are affected by after school or complementary learning efforts. To make sure everyone listens, both quantitative and qualitative data are important and need to be conveyed in different and creative ways that have broad-based appeal.

Finally, public awareness is important. We need to use evaluation to help the public understand after school's value and importance, so that they will demand it from their leaders. We also need to show the public that their leaders should not carry the weight alone. The public must be a partner in supporting, owning, and linking after school efforts to other services in their communities.

HFRP Staff
Email: hfrp_pubs@gse.harvard.edu