The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.
Volume I, Number 1, Winter 1995
Issue Topic: Systems Reform
Promising Practices
Maryland Shares Lessons in Self-Evaluation
Betsy Martin of Harvard Family Research Project reveals the lessons Maryland has learned after becoming the first state in the U.S. to pursue a self-evaluation strategy for systems reform.
When Maryland launched its Systems Reform Initiative (SRI) in 1988, it committed itself to a community-based effort to reform the delivery of services to children and families. Not only would the planning and delivery of services happen at the community level, but communities would also develop the means to monitor their own progress: they would become self-evaluating. With this decision, Maryland became the first state to pursue a self-evaluation strategy for systems reform. Six years later, it has lessons to share.
Why Self-Evaluation?
Although SRI is powered by a network of local planning entities, it comes under the auspices of the Governor’s Subcabinet for Children, Youth and Families. Part of the Subcabinet’s initial responsibility was to decide how best to evaluate SRI. Donna Stark, who was Maryland’s systems reform director at the time, explained: “A traditional approach would develop a system, invest a lot of money, and sometime down the road bring in someone to tell us whether or not it was working. We thought it was better to take a look at our work while we were doing it and make midcourse corrections.”
Self-evaluation gave them the data they needed to do so, but it also fit with the political climate. “We were selling a new system of accountability. Self-evaluation gave us a methodology to go back to the governor and the legislature with specific details about how we were doing. It gave us a lot of credibility,” Stark said.
Steps to Implementation
In keeping with the community-based focus of SRI, once the decision to use self-evaluation was made by the state, the local planning entities determined how to implement it. “In Prince George’s County (PGC), we chose our approach as an attempt to put our money where our mouth is,” said Winnifred Wilson, executive director of the Commission for Children, Youth and Families in PGC. “We created budgeted staff positions that are research and data oriented and integrated evaluation into everyone’s work. Evaluation is now a line, management, and board responsibility.”
But it wasn’t always that way, and getting there wasn’t easy. PGC’s systems reform effort initially revolved around intensive family preservation services, and when PGC first attempted to evaluate them, they discovered that they didn’t really know what the services entailed. Lynn Usher, Ph.D., from the School of Social Work at the University of North Carolina at Chapel Hill, was the lead evaluation consultant. He explained: “Our objective was to do an outcome evaluation or determine if one could be done. But you have to have an intervention whose outcome you can evaluate. We didn’t want to make the mistake of concluding that the family preservation outcome couldn’t be achieved if, in fact, the intervention hadn’t been implemented properly.”
The Goals of Maryland’s
|
So Usher suggested using self-evaluation to generate information about the family preservation services that could be evaluated. “The first step was to obtain data that described the pattern of interaction with families, and we found that they did not conform to the model of service that was intended,” Usher said. For example, they found that they weren’t serving the families they intended to serve. With this information they changed who they were serving and brought the service m line with the model in other ways, too. Once self-evaluation was used to improve the evaluability of its family preservation services, PGC could then use it to continually improve the services. Self-evaluation relies on good data. The state, working with PGC and other communities, developed and implemented an automated Activity Log and a case worker system called Home Base. It includes an assessment of the family’s strengths and needs; a family plan; goals; and measurements.
“The data is useful for supervision, managing, monitoring, and planning, all leading to decision making for the future,” explained Wilson. For example, PGC used to have five teams of family therapists and one support team. By tracking and monitoring the teams’ activities, they discovered that staffing levels weren’t always appropriate. Therapists sometimes had time on their hands. By changing the teams, they were able to divert therapists when they were needed for other programs. “It helps us determine how we can be more responsive,” Wilson added.
Emerging Opinions
The change to self-evaluation via an automated data system was not without resistance. “At first people felt they were being micro-managed, but when they found out they could use the data to see patterns, they began to see it as a useful tool,” explained Margaret Rawle, state director for SRI. It is also tied into an incentive dollar process, meaning that when communities reach their goals for families and children, they get additional funding beyond their operations budget. “Initially the data tracking felt onerous to some staff until they saw how what they were doing was tied directly into the money we would get into the community,” PGC’s Wilson added.
Further Reading
Usher, C. L.. (1995). Improving evaluability through self-evaluation. Evaluation Practice, 16(1), 55–64. Usher, C. L., Gibbs, D., & Wildfire, J. (forthcoming). A framework for planning, implementing and evaluating child welfare reforms. Child Welfare. |
This sense of broad ownership is, perhaps, the strongest selling feature of self-evaluation. In traditional evaluations, the relationship between evaluators and program staff is adversarial, based on the belief that evaluations need to be objective and program managers and staff have too much vested in the program to be unbiased. Self-evaluation, on the other hand, is based on the “alternative view that program staff sincerely want to do a good job and avoid embarrassing mistakes,” Usher explained. “As a result, it would be helpful for them to have information that would enable them to plan and manage programs more efficiently and effectively.”
In Maryland most information is in the hands of staff who use it regularly for case planning and management. “The system was built with the idea that most data would stay at the worker level. Less is needed as you go up. At the state level, we tend to aggregate and look for patterns. It is a management tool, something I as a manager and policymaker can use to make midcourse shifts to do the best with the money I have,” Rawle added.
Although there is some discussion of doing a traditional, after-the-fact evaluation of SRI, for the most part Maryland’s Governor, legislators, and advocates seem satisfied that self-evaluation is giving them the information they need to improve the state's services to families and children.
Betsy Martin, M.P.A., HFRP