Identifying areas and approaches for improving evaluation processes in environmental education in the United States of America

Many environmental educators evaluate programs because a funder asks them to. The result can be a report that satisfies the funder but tells staff little about how to make the program better. This study asked whether the purpose and style of an evaluation affect how useful educators find it.

In June and July 2020, the researchers invited members of NAAEE and the Association of Nature Center Administrators to take an online survey, and also shared it on social media. In all, 410 educators whose organizations run programs in the United States described how they evaluate programs, what they want evaluation to achieve and how satisfied they are. The researchers compared two kinds of evaluation. Formal evaluation means collecting and analyzing data in a planned, systematic way. Informal evaluation means staff observations, discussions and reading feedback, without a planned way of collecting data.

What they found:

1. Educators were not very satisfied with their evaluations. On a five-point scale running from "not at all" to "extremely" satisfied, the average fell between "slightly" and "somewhat".
2. Formal evaluation went with greater satisfaction. Nearly four in ten organizations used only informal evaluation. Educators whose organizations did formal evaluation were more satisfied, whether staff ran it alone or with an outside evaluator. They also said their evaluations met most of their goals better, such as finding out whether programs worked and using results to make changes.
3. Evaluations that help improve programs left educators most satisfied. The researchers asked how well evaluations met seven goals. Satisfaction was most closely linked to three improvement goals: knowing whether a program met its goals, understanding why, and using results to change the program. Knowing whether participants enjoyed the program also mattered.
4. The goals educators valued most were met least. Educators rated improvement goals as more important than reporting goals, such as meeting grant requirements. But they said reporting goals were met more often. Only four in ten said their evaluations mostly or fully explained why programs succeed or fall short.
5. Few evaluations showed how programs work for different groups. Fewer than three in ten said their evaluations mostly or fully showed how programs affect different groups of participants. This was the goal met least often.
6. Outside evaluators helped most with reporting. Organizations working with an outside evaluator did best at meeting grant requirements and showing success to funders and other supporters.

What this means for your practice:

• Decide what your evaluation is for. Evaluations that supported improvement left educators most satisfied. The researchers suggest shifting from evaluating to prove success toward evaluating to find what can be improved. A less-than-perfect result then becomes useful information rather than a threat.
• Collect at least some data systematically. Formal evaluation went with greater satisfaction and better results on most goals, including when staff ran it alone. Giving every group the same short survey, in the same way, is a first step beyond impressions.
• Record what happened, not just the results. Understanding why programs work was among the goals met least often. To address it, the researchers suggest recording details of each session alongside outcomes, such as who taught it, the order of activities, the setting and how students were prepared. Comparing sessions can then show which elements go with better results. This is the researchers' suggestion; the study didn't test it.
• Use results, adjust and check again. The researchers describe this as adaptive management: evaluate, change the program, then evaluate again to see whether the change helped.
• Look beyond the average. Educators rarely knew how programs affect different groups. The researchers suggest collecting basic information about who takes part, such as age group or school, and comparing results between groups. An average score can hide who gains most and who gains least.
• Get outside help for funder reporting. In this study, organizations working with an outside evaluator were best at meeting grant requirements and showing success to funders.
• Talk with funders about learning. The researchers suggest funders hold programs accountable for learning and improving over time, rather than for hitting set numbers.

RETURN TO EVALUATING ENVIRONMENTAL EDUCATION EERESEARCH COLLECTION