The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.
Volume X, Number 3, Fall 2004
Issue Topic: Harnessing Technology for Evaluation
Ask the Expert
Evaluating a Literacy Program Using Web Surveys
Frontier College, a 105-year-old nonprofit organization, provides tutoring in literacy skills to children, youth, and adults in a range of settings across Canada. The College has over 3,500 volunteers working in both English and French at over 250 sites. In 2001 the College undertook an extensive evaluation to examine four key program components: learners, volunteers, community organizers, and community partners.
Traditional evaluation methods such as focus groups and interviews were used, as well as an extensive web survey. The survey examined three questions: How can volunteers be better supported and trained? What specific learning outcomes did volunteers observe? How does this experience impact volunteers?
A link to the survey was sent in both English and French to 2,941 volunteers; the return rate was 28.6%. Although this percentage was higher than expected, response rates were inconsistent nationwide. Our findings indicated, broadly, that we needed to refine our tutor training by providing more concrete material, introducing evaluative components into training, and improving volunteer retention. Each of our 25 regional staffs received a detailed breakdown of their own data and the aggregated national data—information invaluable in developing program plans.
This survey taught us that we tried to collect too much information and that applying good survey design principles was crucial. With these lessons in mind we launched our second web survey in January 2004. Because of our initial experience, we made this survey shorter and more focused, put a link to it on our website, and printed flyers with the website address for people who did not receive email.
Our return rate for this second survey was 36.7%, with 1,264 responses. The design changes resulted in a more evenly distributed response pattern, which allowed us to compare data more effectively. Key findings included the fact that initial volunteer management steps, such as interviewing and screening, are being consistently used, and that volunteers wanted specific aspects of tutoring activities to be refined further.
The benefits of using web surveys for ongoing evaluation include cost-effectiveness, the ability to reach a highly mobile volunteer base, ease of data analysis, and the ability to track volunteers over time for trend analysis. However, this technology presents important challenges, including limited ability to work in multiple languages, accessibility barriers, and high volumes of email that may compete for respondents' attention.
We will continue our web survey in spring 2005, with a more detailed outcome matrix to track learner progress from a volunteer perspective. Details will be available in summer 2005.
The author wishes to acknowledge the Canada Volunteerism Initiative for funding the online evaluation for 2003–2004, and Melanie Valcin of Frontier College for her tireless work on the project.
Daniel Khimasia
National Program Manager
Frontier College
c/o Heartwood House
153 Chapel St.
Ottawa, ON K1N 1H5
Canada
Tel: 613-241-9525
Email: dkhimasia@frontiercollege.ca