Search Publications |  The Evaluation Exchange |  FINE Network |  OST Database

HFRP > Evaluation > The Evaluation Exchange > Issue Archive > Evaluating Community-Based Initiatives > Alternative Designs for Community-Based Research: Pittsburgh’s Early Childhood Initiative

Volume IX, Number 3, Fall 2003

Issue Topic: Evaluating Community-Based Initiatives

Promising Practices

Alternative Designs for Community-Based Research: Pittsburgh’s Early Childhood Initiative

Stephen Bagnato, Robert Grom, and Leon Haynes describe an evaluation design for Pittsburgh’s Early Childhood Initiative that provides scientific rigor in a community setting.

Little agreement exists about how evaluations of social intervention programs should be conducted. Traditional social scientists argue for the use of laboratory-based, control group, randomized designs as the gold standard, but this approach lacks generalizability to real-life settings. Alternative evaluation designs are necessary to document the elements of intervention programs that predict outcomes in natural community settings. Yet critics charge alternative methods with a lack of experimental rigor.1 An evaluation approach known as authentic assessment and program evaluation research meets the demand for rigor while addressing the community setting context.

Pittsburgh’s Early Childhood Initiative
In 1994, as part of the Early Childhood Initiative (ECI), the Heinz Endowments organized the business, corporate, agency, and foundation sectors in Pittsburgh to expand quality early care and education programs and options for unserved children in high-risk neighborhoods. The overarching mission of ECI is to foster preschool and school success for children of poverty, whose typical retention and special education placement rates in kindergarten have ranged between 18% and 40%.

A consortium of business, community, and foundation leaders designed the goals, approach, and expected outcomes of ECI. This design was based on seven core features of successful early childhood programs for children at developmental risk that were identified by Craig Ramey and Sharon Ramey in their article, Early Intervention and Early Experience.2 The seven core features include: (1) longitudinal interventions starting in infancy and monitored through functional benchmarks; (2) intensive, comprehensive, and individualized programs and supports; (3) integral parent participation; (4) high program quality and frequent monitoring; (5) direct child interventions; (6) community-directed programs and integrated services; and (7) follow-through of child and family supports and program evaluation into the primary grades.

Several Pittsburgh urban neighborhoods have participated in this collaboratively designed and privately funded joint venture. Braddock’s 4 Kids Early Childhood Initiative and the Wilkinsburg ECI are two of the most distinctive of these community-driven ventures. A community leadership council established in Braddock forged a relationship between Woodland Hills School District, Head Start, and various formal and informal resources in the community (e.g., churches, U.S. Department of Housing and Urban Development community councils, and local hospital networks) to link services for children and families. In Wilkinsburg, Hosanna House, a broad community service center, incorporated family support programs as central features of their early care and education programs. In fact, these communities have lead efforts to incorporate the School Readiness Group, a nonprofit early childhood consortium, in order to harness the influence of cross-community partners to advocate for government, foundation, and agency funding.

SPECS Authentic Program Evaluation Research Model
In 1996 the Heinz Endowments and the ECI Management Council, composed of business, corporate, foundation, and community members, selected an interdisciplinary research team from Children’s Hospital of Pittsburgh and the UCLID Center at the University of Pittsburgh known as SPECS (Scaling Progress in Early Childhood Settings), as the winners of a national competition to conduct ECI’s longitudinal evaluation.

SPECS’ evaluation approach—authentic assessment and program evaluation research—helps community-based programs demonstrate “how good they are at what they do.” It has been validated in the field through evidence-based research conducted through “natural experiments” in real-life community settings rather than laboratory settings.3 SPECS’ strategies are unique and effective because they:

• Use a collaborative research model with community partners for the formative and summative research phases.
• Ask whether the program works in a natural setting rather than a laboratory setting.
• Assess all children, families, and programs in the study without exclusions.
• Apply the developmentally appropriate quality guidelines of the National Association for the Education of Young Children, the Division for Early Childhood, Council for Exceptional Children, and the Head Start Performance Standards.
• Do not use traditional “tabletop testing” and remove the child, teachers, or parents from their natural situation or “developmental ecology.”
• Rely on ongoing observations from consistent caregivers in the child’s life.
• Sample skills within the preschool’s developmental curriculum that are teachable and predictive of future kindergarten success.
• Offer ongoing feedback to teachers, parents, and the community about children’s learning and needed program refinements.
• Operationalize a longitudinal repeated-measures design using HLM and path analysis strategies.4

SPECS’ research methods track progress and interrelationships among multiple factors like children’s development (e.g., basic concepts, literacy, social skills, and self-control behaviors), parenting and family strengths, the standards and “best practices” of early childhood programs, and neighborhood resources and interagency partnerships in systems reform efforts.

The Results of the Early Childhood Initiative
The SPECS evaluation team carefully tracked the progress of 1,350 enrolled children between 1997 and 2003. The team observed and profiled progress three times each year, focusing on thinking, language, early literacy, social, behavioral, and play skills. They regularly provided feedback to teachers and parents to guide their teaching and care. They also conducted program quality evaluations in 25 programs in nine Pittsburgh neighborhoods (Braddock, Wilkinsburg, Sto-Rox, East Liberty, South Side, Highlands, Hill District, Homewood, and Steel Valley).

SPECS’ research on ECI’s impact showed major outcomes in four areas (for details see the box):

• Children beat the odds and learned early skills for school success.
• Mentored programs achieved stringent quality standards.
• With teachers’ help, parents learned new ways to nurture their children’s development.
• Communities proved their leadership and made their programs successful.

To download the full SPECS report or executive summary go to www.uclid.org:8080/uclid/ech_specs.html.

How Children Benefited From the Early Childhood Initiative (ECI) Developmental Progress • On entering the program, 86% of the children were classified as “high risk” for shortcomings in overall thinking, language, and social and school-readiness skills. Fourteen percent of the students were deemed to be both high-risk and developmentally delayed, which would qualify them for early intervention or special education services in Pennsylvania. The documented national rate for developmental delays is 3% to 8%. • The longer that children participated in high quality ECI programs, the greater the developmental progress and achievement of early school success skills. • After nearly three years in the program, the high-risk group showed at least average developmental progress without the typical setbacks for children of poverty documented in national research. • The delayed group showed an accelerated rate of developmental progress into the average range that was 160% of the typical or expected rate in normal child development. Social and Behavioral Progress • ECI children in the full high-risk group achieved normal social skills and self-control behaviors compared to national peers. • 18% of the children at entry into ECI showed significant problems with social skills and self-control behaviors that would qualify them for mental health diagnosis and support; this challenging behavior problem-group achieved normal social and behavioral skills after nearly three years of ECI participation. Early School Success • 125 of the children in the ECI program transitioned to kindergarten and first grade over this period. • In the school districts from which students were recruited, an average of 23% of children are retained or “held back” in kindergarten and first grade, and 21% are referred to special education programs. After nearly three years of ECI participation, less than 2% were retained and less than 1% were referred for special education. • End-of-year “blind” follow-up assessments by kindergarten and first grade teachers on the Basic School Skills Inventory-Revised, a nationally standardized achievement test of early learning skills, demonstrated that ECI children who transitioned to school performed at an average to above-average range compared to their national peers.

1 Yoshikawa, H., Rosman, E. A., & Hsueh, J. (2002). Resolving paradoxical criteria for the expansion and replication of early childhood care and education programs. Early Childhood Research Quarterly, 17(3), 3–27.
2 Ramey, C. T., & Ramey, S. L. (1998). Early intervention and early experience. American Psychologist, 53(2), 109–120.
3 Bagnato, S. J., Suen, H. K., Brickley, D., Smith-Jones, J., & Dettore, E. (2002). Child developmental impact of Pittsburgh’s Early Childhood Initiative (ECI) in high-risk communities: First-phase authentic evaluation research. Early Childhood Research Quarterly, 17(4), 559–580.
4 HLM (Hierarchical Linear Modeling) is an analysis that estimates the effects of social units—groups, neighborhoods, schools, workplaces, organizations, communities, social networks, or whole social systems—on individuals. Path analysis refers to the method by which the path of the cause and effect relationship among variables is determined.

Stephen J. Bagnato, Ed.D.
Professor of Pediatrics & Psychology
Director, Early Childhood Partnerships
Children’s Hospital of Pittsburgh
The UCLID Center at the University of Pittsburgh
Tel: 412-692-6520
Email: steve.bagnato@chp.edu

Robert Grom
President and CEO
Heritage Health Foundation, Inc.
Greater Braddock Early Childhood Network

Leon Haynes
President and CEO
Hosanna House
Wilkinsburg Early Childhood Initiative