Measuring Program Success in the Field

One of the biggest hopes for participants in environmental education (EE) programs is that, afterward, they undertake actions that are aligned with sustainability goals, in both the short and long term. Yet, it can be difficult to measure the long-term impact of environmental programs on environmentally related attitudes, identity, and, especially, behavior. Because of this, scholars have identified indicators, or intermediary outcomes, that have been correlated with longer term outcomes, such as environmental behavior change. Those measures include environmental awareness, attitudes toward nature, critical thinking skills, feelings of connection to nature, and environmental knowledge, among others.

How are intermediary outcomes being measured in the field, and what are challenges to evaluating programs that may be interested in measuring these kinds of variables? Researchers investigated this question by examining a range of EE programs and their evaluations, with a focus on residential programs. Using Internet searches, the researchers looked for information on EE programs with: functioning websites; descriptions of the EE provider's mission statement and specific program goals; at least one overnight “on site”; and intermediary outcomes such as knowledge, attitudes, and skills. Based on those criteria, the researchers identified 206 programs; of these, 37 had previously conducted evaluations.

The most commonly reported types of evaluation instruments used by the organizations in this sample were surveys, interviews, and participant observation forms. Researchers found that, in general, the programs' stated goals, mission statements, and philosophies matched the outcomes described in the evaluations. In other words, the programs appeared to be accomplishing the goals they set out to achieve. The objectives that the programs described commonly measuring were environmental awareness, attitudes, behaviors, skills, citizen participation, personal development, social skills, and community development.

As a follow-up to the finding that a small number of programs actually measure their outcomes, researchers interviewed 12 environmental educators working in a residential field science-based program to identify some of the challenges to conducting evaluation in the field. The main barrier that the educators described was the time required for evaluation; their perception was that evaluations detract time from regular programming.

Educators did suggest several ways that evaluation might be more fully integrated into programming. Educators, in particular, identified student-generated material (such as journal entries), easy-to-use observation protocols, mapping exercises that demonstrate student knowledge connections, and student art projects (such as photographs) as potentially useful evaluation tools. An educator might teach an activity on snow pack levels and the importance of conserving water, for example, and, as an evaluation of its effectiveness, she might ask students to journal about their impact on the water cycle, how they could conserve water, and why that would be important. Another example of evaluation shared by the authors relates to a lesson on tide pool ecosystems: before and after the lesson, educators might ask students to draw a group map of a tide pool. The drawing could be evaluated using a rubric to examine for complexity and completeness of the ecological interactions depicted. An educator might also ask students to take photographs during the course of an environmental education program and use those to facilitate a discussion about what they learned throughout the week.