Volume XI, Number 3, Fall 2005

Issue Topic: Democratic Evaluation

Promising Practices

Program Evaluation in a Democratic Society: The Vera Model

Tim Ross, Research Director at the Vera Institute of Justice, explains Vera's rigorous and multitiered data collection process and the benefits of partnerships with public programs.

A great advantage of open societies is the ability of citizens to critically evaluate government activities. The Vera Institute of Justice, a New York City-based nonprofit organization, serves as an incubator for new and innovative ideas concerning social justice. Over the last 44 years, the Vera Institute has launched dozens of demonstration programs and created 15 nonprofit organizations to serve vulnerable populations such as court-involved youth, the homeless, and children in foster care.

To evaluate its programs, Vera's research department applies rigorous social science methods using four data sources: administrative data collected by government agencies, conversations with government and nonprofit managers that implement and direct programs, discussions with frontline staff, and interviews with clients.

To gain access to agency staff and nonpublic data and facilities, Vera researchers actively work to build trusting relationships with public agency officials. Vera's evaluators frequently solicit input from government staff and ask government managers to comment on draft reports. In our experience, government officials will accept and act on the results of our research—positive or negative—as long as they understand how conclusions were made and feel that they were part of the research team. By working closely with government agencies, researchers learn more about the systems under evaluation and in turn produce better research.

Vera's effort to overcome the bias against foster children in juvenile detention decisions provides a good example of a successful collaboration with government agencies. Family court judges, lawyers, and child welfare officials long suspected that foster children who were arrested were more likely to be sent to juvenile detention facilities than their non-foster counterparts. Vera conducted a small-scale study that suggested a bias did exist—not because foster children committed more serious offenses but because the juvenile justice system relied on parents to advocate for their children. Foster youth, in contrast to youth living with their parents, must rely on child welfare caseworkers who are unfamiliar with the juvenile justice system and often fail to appear in court.

To remedy this problem, Vera—with the support of juvenile justice, family court, child welfare, and other officials—launched Project Confirm. Project Confirm screened newly arrested youth for those in foster care, contacted the caseworkers of foster youth, and escorted them through the court process. To evaluate Project Confirm, Vera researchers used administrative data from the child welfare and juvenile justice systems, interviewed senior managers and frontline staff, and observed court hearings. The research showed mixed results. Project Confirm eliminated the foster care bias for first-time, low-level offenders after controlling for the effects of age, race, gender, and other characteristics. However, for repeat and higher level offenders, the program in fact exacerbated the bias. In these cases, caseworkers brought information—including instances of running away from foster care—that led court officials to detain youth at a higher rates than before the program.

The evaluations of Project Confirm produced three key findings.¹ First, the evaluations showed that with the proper training and support, child welfare caseworkers could reduce and even eliminate the foster care bias. This led to better outcomes for youth—namely, they kept their liberty, did not have to suffer through school interruptions while in detention, and did not lose their foster care placements. Second, reducing the foster care bias led to cost savings for child welfare, juvenile justice, and family court budgets by reducing emergency placements, juvenile detention bed days, and court hearings. Third, the foster care bias against higher level and repeat offenders occurred primarily because court officials interpreted instances of foster children leaving care without permission as an indicator of flight risk. No government databases routinely record runaway events by children who live with their parents.

Project Confirm used the results of Vera's evaluation to modify its training of court advocates, child welfare caseworkers, and family court judges. Following the evaluation, the program's demonstration period ended, and Project Confirm became part of the city's child welfare agency. Since then, Maryland, Chicago, Los Angeles, and other jurisdictions have conducted separate studies of foster care bias and in some cases have started programmatic initiatives around the same issue.

Although Vera's evaluation model has succeeded—in the case of Project Confirm and in many other instances—in improving the conditions of vulnerable populations, ours is not the only or always the best method. In democratic societies, there exist many evaluation paths to the same end: finding out what works best for citizens.

¹ See Ross, T., Conger, D., & Armstrong, M. (2002). Bridging child welfare and juvenile justice: Preventing unnecessary detention of foster children. Child Welfare, 81(3), 471-494; and Conger, D., & Ross, T. (in press). Project Confirm: An outcome evaluation of a program for children in the child welfare and juvenile justice systems. Youth and Family Services Review.

Tim Ross
Research Director
Vera Institute of Justice
233 Broadway, 12th Floor
New York, NY 10279
Tel: 212-376-3139
Email: tross@vera.org