The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.
Volume VIII, Number 2, Fall 2002
Issue Topic: Evaluation for Continuous Improvement
Evaluations to Watch
ProDES: A Continuous Improvement System for Juvenile Justice
Philip Harris and Lori Grubstein of the Crime and Justice Research Center describe the “bottom-up” development of ProDES, an outcome-based information system that tracks youth in the juvenile justice system.
ProDES (the Program Development and Evaluation System) is an information system that tracks every court-committed Philadelphia delinquent in the juvenile justice system and measures program outcomes. It is designed to support both the simultaneous continuous improvement of individual programs and the larger system of services to which each program is linked.¹ ProDES provides programs and the system as a whole with a continuous flow of intermediate (changes during the program) and ultimate (recidivism and community adjustment²) outcome information. This information is then used for program development, matching youths to programs, and assessment of the program resources available to system-level administrators.
The system comprises data collection, analysis, and feedback. ProDES provides four types of information that allow program administrators to make informed changes to program designs, and allow system administrators to assess the value of different programs for different categories of youths and to identify the need for new or expanded program resources. The information consists of:
- Case-specific reports at intake, discharge, and follow-up
- Semi-annual, program-specific reports summarizing aggregated outcome data
- Annual, system-wide reports that describe trends for the entire system
- Topical reports that explore policy issues such as “the female delinquent” and “substance-abusing delinquents”
With this information, programs can study the reasons why some youths leave programs early, explore changes in their incoming client population over time, examine client change on dimensions relevant to future behavior, and develop testable theories on the impact of program improvements.
A Focus on Utilization and Continuous Improvement
We began developing ProDES with a clear bias—that whatever kind of system we created, the information had to be useful to stakeholders for purposes of program and system development. At the same time, we recognized that the information would eventually be used to hold programs accountable for their outcomes. We also recognized the risks inherent in building a data collection system across nearly 100 programs, most of which are operated by private agencies. We could not depend on coercion to build the system; it had to appeal to its user groups.
In planning ProDES, the developmental process had to be “bottom-up” with the private agencies having direct input to both the structure and content of the system. Accordingly, the first phase of the system involved research with 15 agencies that were selected to represent a broad range of programs. Within each program, we conducted evaluability assessments that sought to articulate the goals and objectives of the various programs, their measures of success or failure, and the range of information they routinely collect and use.
The first phase demonstrated that agencies commonly agree on what type of information is important for their programs and the system as a whole. They also agreed about the ideal structure of an information system—one which went beyond simply describing program clients at one point in time and included multiple measures to demonstrate change over time.
Once we identified our key outcome dimensions, we devoted several months to testing standardized scales that would be used to measure change on these key variables. The 15 programs supported the effort by testing the measures on their clients and providing feedback. These same programs served as the core of ProDES when it went into full operation.
A significant part of our job involves facilitated discussions about utilization, helping users develop their own research questions and supporting their use of the results of our analyses. In our utilization design (see box ), each major decision is linked to the appropriate decision maker and to the information that best supports that decision.
Goals and Intended Utilization of ProDES |
|||
| Goal | Information | Intended User Groups | Intended Use |
| Program Development To facilitate program development through the use of continuous outcome information |
• Trends in intake population characteristics • Trends in specific program outcomes • Comparisons to similar programs serving similar clients • Predictive analyses of program outcomes |
• Program administrators | • Revision of target population characteristics • Better screening of referrals • Program design modification • Marketing |
| System Planning To facilitate planning of services through information about program strengths and weaknesses |
• Trends in program outcomes • Trends in intake population • Comparisons of similar programs with similar clients • Cost benefit analysis |
• Court and Community Services Planning Group – reviews and renews contracts annually • Human Services Monitors – monitor programs regarding compliance with regulations |
• Renewal of contracts • Development of requests for proposals • Program accountability for improving poor outcomes • Adjustment of per diem payments |
| Client Matching To facilitate rational matching of youths to programs |
• Program outcomes subdivided by risk of recidivism and case type • Detailed information on program design and structure |
• Probation officers • Special Needs Unit Staff • Judges |
• Consideration of predictions of outcomes when making program recommendations and decisions |
Through surveys and an independent evaluation we know that stakeholders’ use of ProDES has grown over time. Programs use ProDES to:
- Refine their target population characteristics or screening criteria
- Modify their designs and document their need for additional resources
- Better market their services to the larger system
- Help new managers better understand their strengths and weaknesses
The services system uses ProDES to:
- Review program contracts
- Explore problems with specific programs
- Answer policy-relevant questions (e.g., Why are so many first time offenders being sent to residential facilities? Why are some youths not receiving aftercare services?)
- Plan for future resource demands
The Court and the Probation Department use ProDES to:
- Learn about trends in the delinquent population
- Improve their matching of youths to programs
- Track program use
- Develop plans for future resource needs
Utilization of data proceeded most rapidly among programs, although about three years passed before the use of ProDES data became institutionalized. This adaptation has been facilitated by changes in human services, where there is a growing respect for program evaluation, but working to make evaluation a collaborative venture clearly made a difference.
Developing consistent use in the Court has been more challenging. We developed a working group of Probation Department staff that meets monthly, began conducting training sessions within the department and with the judges, and secured separate funding to work with the Court to develop a tool that will assist judges in matching youths to programs. Support for ProDES is growing steadily as the Court has begun to view its data and staff as supports for developing meaningful goals.
Conclusion
Building an outcome-based information system for an entire juvenile justice system has been, and continues to be, a challenge, even after eight years of operation. We have learned that the strategy of setting program development as our primary goal has worked well. It is a goal that makes sense to all system stakeholders and that enabled us to build working relationships with our primary sources of data and the most critical user group.
We have also learned that face-to-face interactions with users about their outcome data have helped to build the trusting and productive relationships that are necessary to guarantee data utilization. Moreover, these relationships have empowered users to collaborate in our research process, often raising valuable research questions which we can respond to quickly.
Lastly, we have learned the value of both training and effective communication. In order to increase the capacities of users to consume research information we need to make it accessible. This means continually improving our presentations of information as well as training users to become smart consumers of research.
Philip W. Harris
Co-Director
Crime and Justice Research Center
520 North Columbus Boulevard, Suite 600
Philadelphia, PA 19123
Tel: 215-204-5766
Email: pharri01@temple.edu
Lori Grubstein
Former Senior Research Associate
Crime and Justice Research Center
Email: lgrubst@msn.com
¹ ProDES was developed as a joint venture between the city’s Department of Human Services (DHS), Family Court, and the Crime and Justice Research Center.
² “Recidivism,” which generally means “re-offending,” is defined for use in ProDES as a petition to Family Court for a new offense. By “community adjustment” we mean that the youth is attending school regularly or is employed, is getting along well with family members, is not abusing drugs or alcohol, and has not been involved in new delinquent behavior.