Volume IX, Number 1, Spring 2003

Issue Topic: Evaluating Out-of-School Time

Questions & Answers

A Conversation With Michelle Gambone

Michelle Gambone

Dr. Michelle Gambone is president of Youth Development Strategies, Inc. (YDSI), a youth policy and research consulting firm. Most recently, Dr. Gambone and her colleague Dr. James Connell, president of the Institute for Research and Reform in Education,¹ applied longitudinal research to their previous community youth development work to examine the relationship between supports and opportunities and long-term developmental outcomes for youth. These results were just released in Finding out What Matters for Youth: Testing Key Links in a Community Action Framework for Youth Development.² Michelle spoke with us about her new research and its implications for quality youth programming.

Q: Research in the field of youth development is expanding rapidly. What was the impetus for your new longitudinal research on supports and opportunities for youth?

A: Much of the early work in youth development had already begun to shift the focus of youth development programs from reducing “high risk” behaviors to promoting healthy developmental outcomes for all youth (e.g., coping, decision-making skills, successful education, employment opportunities) that would subsequently lower the occurrence of negative long-term outcomes. This was a really good step, but there was still a question about whether or not you could expect any kind of a long-term payoff from taking a developmental approach, or if you would only get better short-term outcomes. Decision makers wanted to know what would happen in the long run. Would investing in strategies to provide all youth with positive developmental experiences lead to better short- and long-term outcomes?

While the field was pushing this question, Jim Connell and I were developing our Community Action Framework for Youth Development, which integrates basic knowledge about youth development and the community conditions that affect it with emerging hypotheses about what it will take to transform communities into places where all youth can thrive (see figure).³ The framework translates developmental principles into a systematic approach to planning, implementing, and evaluating activities and investments for youth. It connects shorter-term outcomes to longer-term outcomes, looking at the things that are needed to get to the developmental outcomes, like enhancing the supports and opportunities available to youth, and the community strategies that actually help build those supports and opportunities across settings. We tried to focus our framework on outcomes that were actionable—things that you could really be accountable for and take action to change.

In the course of trying to use that framework as another way to talk about a developmental approach, we kept running into the same question—We are happy to think about developmental outcomes as short-term goals, but can you show us that young people who have supports and opportunities and good developmental outcomes are better off as young adults?

To answer this question we secured W. T. Grant Foundation funding to accomplish two things: (1) conduct a literature review that focused on the outcomes in our framework to determine the strength of evidence linking supports and opportunities to short-term developmental outcomes and, in turn, to long-term outcomes, and (2) to acquire longitudinal data sets on the elements in our framework to tackle the question of long-term outcomes. The two data sets that best matched our framework were both collected by Dr. Jacque Eccles and her colleagues at the University of Michigan; both sets had data on young people from middle school through their mid-twenties.

Our study addressed three main questions:

1. How much evidence exists that links the specific outcomes in our framework to each other?
2. Are there high-quality, practical measures of the outcomes in this framework that can be used in community settings to assess how well youth are doing?
3. Can existing data on youth be analyzed in a way that shows how well youth need to be doing on early outcomes in the framework in order for us to be confident they will succeed later in life; and can we show how much difference supports and opportunities and youth development outcomes make in achieving long-term success?

Q: What were the results of your longitudinal analysis of the data sets?

A: Using the data sets, we looked at three critical supports and opportunities: (1) supportive relationships with adults, (2) challenging and engaging activities, and (3) meaningful involvement and decision making. First, we developed thresholds or “tipping points” for the critical elements in our framework that predict later success. For example, we found that youth who have at least one highly supportive relationship with an adult will do better than youth who have none. Therefore, the tipping point for later success is youth having one or more supportive adult relationships. Then we determined how much difference it makes that youth hit these tipping points. Again turning to supportive relationships for an example, we found that young people who had good supportive relationships at the beginning of high school were twice as likely as average kids to have good developmental outcomes at the end of high school. Similar results were found for the other two supports examined.

We then looked at the next step in development—is not having good developmental outcomes at the end of high school connected to problems for young people when they are in their early twenties? We found that young people with poor developmental outcomes at the end of high school were one and a half times more likely than average to have poor early adult outcomes by their early twenties. Therefore, the chain of evidence was clear and convincing—if you look at the same young people from the beginning of high school on into their early twenties we can show that kids with sufficient supports and opportunities (as determined by tipping points) are doing well developmentally as young adults.

Our next step will be to partner with other researchers and do more sophisticated developmental trajectory models to be sure that our findings hold up, even controlling for preexisting differences between individual youth. But given the size of the effects that we found, we are quite confident they are not going to go away.

Q: What are the implications of your new longitudinal research for policy, practice, and accountability?

A: We need to move past the question of whether or not providing young people with experiences that give them good supports and opportunities is a good thing to do. We have shown that this strategy will get us to our long-term goal of getting kids to good positive adult outcomes. So we now need to focus on what this means for the way in which we provide services and create activities for young people.

A second implication is that a developmental approach means you need to focus on supports and opportunities at the community level. With rare exceptions, any individual setting that works with young people will not provide enough of what a young person needs by itself to get them to good outcomes. Programs, families, other adults in young peoples lives, schools, and after school programs contribute to, but do not determine, how young people will turn out as young adults. This has implications for accountability. The very first step in accountability ought to be to ask any of these settings, “Can you show that you are providing young people with these supports and opportunities? Are you contributing your fair share to youth development?”

Q: Have any youth-serving organizations used the Community Action Framework for program improvement and accountability?

A: Yes. We created a process with the Community Network for Youth Development (CNYD) in San Francisco where we took study data on the supports and opportunities and after school settings from youth development agencies in San Francisco and used it as the starting point for an organizational improvement process with agencies. First, we examined youth’s program experiences through our survey of supports and opportunities based on the framework. Next, we provided data back to the agencies about these experiences and used the data to engage staff in a self-assessment process. Staff then developed an assessment and action plan that articulated what practices needed to be incorporated into their work or strengthened, and an implementation plan for improvement. Then agencies used these action plans as the basis for proposals to a core group of funders in California. This enabled the agencies to spend a full year implementing their action plans.

At the end of the year we resurveyed the young people to see whether or not there had been any change in the levels of supports and opportunities they were experiencing, and, in fact, there had been. There was some variation, but every agency improved in some area. It was very clear when we went back and looked at where agencies improved and where they did not that improvement was linked with the strategies they implemented.

Agencies and funders alike learned that you can reliably, and in compelling and meaningful ways, measure supports and opportunities for youth, and if you intentionally implement improvement strategies, you will get positive outcomes.

Q: How can you bring this kind of organizational learning to scale?

A: YDSI is creating a web-based assessment process that allows us to work with single agencies or groups of agencies in different places. Using our website we can orient agencies to the framework. We can do virtual training on the supports and opportunities survey, and we have created a low-cost way to collect and analyze data and report it back to programs. In Santa Cruz, for example, we are working with 14 agencies that are part of a youth development alliance to conduct an organizational assessment process. We will conduct, on-site, an introductory workshop and two other workshops over the course of eight months, but these will be the only times that we will have face-to-face contact. The rest of the work will be done by phone and using the website. This provides a low-cost alternative to hiring an evaluator and it addresses the unfairness of asking youth workers to add evaluation skills to all the other demands made on them.

We are also planning to use our web-based data collection system to compile a national data set on supports and opportunities that can link these outcomes with program activities. We are collecting data everywhere the survey is used so that, over time, YDSI can create field-building reports that can help prioritize youth development policies and practices that are tied to the youth experiences our research shows really matters.

Q: What is the most important message for our readership to take away from your research?

A: That supports and opportunities do matter. They matter because they are things young people have the right to expect. And they matter because they get us to the community goals that we have for young people. The field now needs to determine how we deliver those supports and opportunities and make sure young people get more of those kinds of experiences in more settings.

Priscilla Little, Project Manger, HFRP

¹ Website: irre.org
² Gambone, M., Klem, A., & Connell, J. (2003). Finding out what matters for youth: Testing key links in a community action framework for youth development. Philadelphia: Youth Development Strategies, Inc. and Institute for Research and Reform in Education.
³ Connell, J. P., Gambone, M. A., & Smith, T. J. (2000). Youth development in community settings: Challenges to our field and our approach. In P. L. Benson & K. J. Pittman (Eds.), Trends in youth development. Boston: Kluwer Academic Publishers. Also in Public/Private Ventures. (Eds.). (2000, Fall). Youth development: Issues, challenges and directions. Philadelphia: Author. Available at irre.org/pdf_files/connell.pdf (110KB Acrobat file).