The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.
Volume V, Number 2 & 3, 1999
Issue Topic: Methodology
Evaluations to Watch
Cooperative Extension’s Capacity to Support Programs For Children,Youth, and Families At Risk: The Organizational Change Survey
Donna Peterson, Mary Marczak, Sherry Betts, and Erik Earthman, The University of Arizona Institute for Children, Youth and Families, write about their evaluation of the Children, Youth and Families At Risk (CYFAR) National Initiative.
Introduction
In 1991, the Cooperative State Research, Education, Extension Service (CSREES), responding to pervasive conditions in America that place children and their families at risk for not having basic needs met, introduced the Children,Youth and Families At Risk (CYFAR) National Initiative. The Cooperative Extension System is based in land-grant universities and has a mission of disseminating, through educational programs, research-based knowledge to communities in order to improve their economic, environmental, and social conditions. CSREES utilizes national initiatives to give direction and special attention to the development of educational programming within the Cooperative Extension System.
Through the CYFAR Initiative, the Cooperative Extension System made a commitment to support programs for at-risk children, youth, and families. This initiative emphasizes a holistic approach to educational prevention programs that facilitate the development and maintenance of healthy, happy environments in which those at risk are enabled to develop life skills necessary for contributing, fulfilling lives.
While Cooperative Extension had the knowledge base and the resources to make a significant difference in communities, it needed to find effective ways to expand its educational programs to at-risk audiences. Thus, the CYFAR Initiative involved a significant expansion of constituencies for Cooperative Extension nationally.
Given this organizational shift in programmatic focus, the CSREES, under contract with The University of Arizona, initiated an evaluation collaboration with state Extension systems implementing CYFAR programs. As a result, the Organizational Change Survey (OCS) was developed to assess states’ abilities to develop and sustain effective programs for children, youth, and families at risk. The survey was implemented from late 1997 to early 1998 to document the current state of Extension systems and to establish a baseline. It will be repeated in 2000-2001 to assess any changes that have been implemented. Ultimately, if the broader organizational changes were effective, Extension professionals will feel better equipped and supported in carrying out program-level goals for serving at-risk audiences.
Survey Design
OCS was loosely based on the national surveys conducted by the Search Institute in 1993 and 1996 to evaluate the National 4-H Council’s Strengthening Our Capacity to Care (SOCC) Project. The development of the OCS was also informed in part by the literature on organizational learning, organizational niche expansion, intra- and inter-organizational relations, and evaluating organizational transitions.
Ultimately, the survey needed to assess six organizational components (derived from research on effective programs for at-risk children, youth, and families) that are key to CYFAR programs:
- Develop and implement a common vision and strategic plan for programming for children, youth, and families at risk.
- Train, support, and reward Extension salaried and volunteer staff for implementing programs which accomplish the CYFAR mission.
- Recognize Extension professionals as critical resources in research and education for children, youth, families, and community issues.
- Promote diversity, inclusivity, and pluralism in Extension programs and staff.
- Promote (internal) collaborations of Extension 4-H, Family and Consumer Science, Agriculture, Community Development, and other University departments in programming for children, youth, and families at risk across the state.
- Promote joint (external) collaborations of community, county, state, and federal agencies and organizations to strengthen programs and policy for children, youth, and families.
To aid in the development of the OCS and to strengthen its validity, a 15-member work group was formed. The members of this group had diverse interests and expertise, and represented USDA-CSREES, the University of Arizona, other land-grant universities including county Extension programs, and the National 4-H Council. After receiving permission to adapt the SOCC survey, the work group members were given a copy of the 1996 SOCC version and six organizational components. The work group members were asked to: (1) categorize the items from the survey under each of the components; (2) determine which questions were essential to each component; and (3) determine gaps in addressing each component.
After a frequency distribution of survey items for each of the six components was calculated, the work group held a face-to-face meeting to come to consensus on the items which best address each component and to discuss survey methods and procedures. A draft of the OCS was then developed and sent out for review to the work group members. Based upon reviewer comments, the survey was revised and piloted with a small sample in Arizona. This copy of the survey, along with supporting documents and survey procedure, was sent to two members of the work group for a final review. After receiving additional feedback via individual interviews with these members, the survey and procedure were sent to CSREES for approval.
The final version of OCS contained 74 items that assessed each state Extension system’s status relative to the six organizational components, utilization of technology, program sustainability, and respondent characteristics. Sample items from the OCS include:
- “Our State Extension system has a strategic plan in place for expanding and strengthening CYFAR programming in our counties.”
- “We are provided the resources (time and money) necessary to engage in collaborative efforts to better serve children, youth and families at risk.”
A two-part response assessed the current and ideal status for items, using a 5-point Likert scale ranging from Strongly Agree to Strongly Disagree. Two open-ended questions assessed the factors facilitating program sustainability.
Survey Procedure and Response Rates
Evaluation collaboration members at The University of Arizona coordinated the survey process. Materials were sent to each state Extension director, and a contact person in each state was then identified as the individual responsible for implementing the survey. Each state was responsible for selecting participants from among eligible Extension professionals.
Forty-five states or territories implemented OCS from late 1997 to mid-1998. Dillman’s (1978) Total Design Method was utilized as a framework for this survey. An introductory letter alerting respondents to the coming survey was mailed one week before the survey itself. A cover letter informing respondents about the purpose of the project and providing directions on returning the completed survey accompanied each questionnaire. All responses, identified only by code numbers, were returned directly to Arizona for analysis. To ensure confidentiality,Arizona did not have a list of respondents and their matching code numbers. Arizona provided the state contact persons with a list of code numbers from returned surveys to facilitate the mailing of reminder letters. One week after the survey was mailed, all eligible respondents were sent a follow-up postcard thanking those who had responded and encouraging others to do so. Two weeks later, a reminder letter was sent to all nonrespondents. Finally, four weeks later, a reminder letter and another copy of the survey packet were sent to all individuals who still had not responded. Response rates ranged from 70% to 99% in 43 of the 45 participating states or territories. The survey procedure also enhanced the validity of the project by minimizing social desirability bias and nonresponse bias.
Data Analysis and Reporting
Initial data entry was performed with the use of a computer software program that allows a standard image scanner to read filled-in bubbles directly from a survey and convert them to numbers for statistical analysis. Each state’s data were saved in a separate file. A state-specific report was prepared for each participating state. These data files were then aggregated for analyses for a national report. In the national report, states were the unit of analysis; in other words, each state served as an individual case. The report included tables containing quartile ranges of state percentages, means, and standard deviations for various items, which allowed states to compare their own data to the national trends.
Results from the Organizational Change Survey provided valuable information about Cooperative Extension’s ability to work with at-risk children, youth, and families. Perhaps the most important result of this survey is the very positive picture of organizational learning that is taking place in Extension across all states; generally, a majority of respondents reported that their state Extension system is successfully implementing the six organizational components. Also, the discrepancies between what is currently happening and what would be ideal for all six organizational components revealed strong support for further expanding and strengthening this system in working with children, youth, and families at risk.
In-depth interviews are currently being conducted with a subsample of randomly selected respondents to gain additional insight into some of the issues raised by the survey responses. For example, while most respondents reported that one of their roles in Extension is to educate policymakers and other community leaders on children, youth, and family issues, they feel they lack knowledge on policy and legislation affecting children, youth, and families at risk. Thus, one interview question examined ways to reduce this gap. Additionally, five evaluation bulletins covering the organizational components (one theme per bulletin) are being published on the Internet. By presenting questions that can be addressed, lessons learned, and additional resources, these bulletins encourage states to further examine their survey data.
It is important to reiterate that the intent of this survey was to examine the “system” rather than how the “individuals” were doing within the system. Thus, the lessons learned from this process will inform the Extension system as it strives to expand and better support its efforts around at-risk issues. The assumption upon which the survey and report are based is that Extension as a system will better support community-based programs for children, youth, and families at risk when the following conditions exist: personnel have a vision and plan for programming; staff and volunteers are trained, supported, and rewarded appropriately; Extension professionals are viewed as critical resources in research and education; diversity, inclusivity and pluralism are valued; and staff collaborate with their colleagues in Extension and the university, and also with others in the community, county, state, and nation.
Donna J. Peterson
Mary S. Marczak
Sherry C. Betts
Erik Earthman
The University of Arizona Institute for Children, Youth and Families