Evaluating the impact of a high school program on environmental leadership and environmental behavior

Environmental education (EE) continues to grapple with how best to develop youth leaders in sustainability. Previous research indicates that assumptions around a direct relationship between attitudes and behavior change is too simplistic. Instead, more recent literature finds that learning is complex and social. This study explored whether and how a program for high school students promoted transformational learning and action competence through a participatory evaluation process.

Two theories underpin this study: transformational learning and action competence. Transformational learning refers to the idea that an intense experience, or set of experiences, may promote a deep reassessment of assumptions. In EE, transformational learning can foster a new way of understanding and living. Action competence is when an individual continually reflects on their actions; when applied in EE, individuals think critically and act to address the cause of environmental issues. Together, these concepts describe what may happen when an individual learns new information that leads to an internal change (transformational learning) that is reflected in their actions and choices (action competence).

The authors of this study used participatory evaluation processes, which involves program staff and participants in data collection, analysis, and interpretation. Specifically, they used logic modeling to articulate how activities within the program are tied to desired outcomes. A theory of change (TOC) is a visual representation of the logic model.

This study investigated the Make a Difference (MAD) program in Aukland, New Zealand. MAD was developed by city officials with input from local youth and sought to promote both transformational learning and action competence among its participants. To become involved with MAD, high schools nominated students to apply, and roughly 30 were accepted each academic year. The program had an unusual structure, which began with a 3-day intensive residential experience known as a hui (Maori for “social gathering”) followed by two years of support. At the hui, students were encouraged to focus on environmental issues of personal interest and to develop potential solutions. During the long-term engagement, coordinators were in touch with students through email and social media, and provided support such as mentoring and funding opportunities for their projects. This study was undertaken in four stages and the methods and findings for each stage are described below.

Stage 1 – Workshops to develop the TOC: During this phase, the researchers held two workshops. The first included four program coordinators (Auckland Council staff who delivered MAD) and asked about the content of MAD, changes they expected to occur, and why those changes were expected. In addition, the coordinators listed activities undertaken in the program and were asked to identify short-, intermediate- and longer-term outcomes of the program. The second workshop included nine recent MAD participants who volunteered to participate, had completed the hui, and were in the follow-up phase of the program. These participants were asked to explain how they felt about MAD, describe their experiences in the program and how they were impacted, discuss their current environmental actions, and consider their behavior in the future. The participants were also asked to consider outcomes at various timepoints.

To produce the TOC, the authors summarized the data from the workshops, listed activities and the rationale for each, as well as listed outcomes at each timepoint. They presented the resultant TOC to program coordinators to confirm the accuracy of their interpretation of the data. The researchers found that coordinators and participants discussed a variety of developmental outcomes (e.g., knowledge gain, self-confidence) as well as action outcomes (e.g., actions at the personal, school, and community levels). Furthermore, the data they collected suggested that participants experienced both transformational learning and gains in action competence through participation in MAD.

Stage 2 - Verifying the TOC using existing literature: To determine whether the findings from the workshop were plausible, the authors compared these findings with findings from the academic literature. They confirmed that the Theory of Change aligned previous research.

Stage 3 - A survey of MAD graduates to validate TOC: To understand whether the perspectives shared by the small group in the workshop were shared with the larger group, the authors conducted a survey of MAD alumni. The survey was based on TOC and included statements related to outcomes described in the program. Participants were asked to gauge their level of agreement with each statement (1=strongly disagree; 7=strongly agree), as well as to describe current environmental actions in open-ended questions. The authors received 31 responses out of 108 surveys distributed. They then calculated the average score for each scaled question and analyzed the open-ended questions for categories of action.

The authors found that there was general agreement with the outcomes identified in the workshops, as well as that MAD inspired transformation and intentional action. Participants rated hui residential experience as particularly impactful. Specifically, that it raised awareness of environmental issues, motivated participants to make changes, and provided an opportunity for participants to engage socially and form relationships. Survey participants indicated they agreed most with the statement that MAD provided leadership opportunities.

The most frequently mentioned categories of action among participants included individual decisions around waste reduction and sustainable transportation. Participants also noted engaging in environmental issues and groups at school, such as leadership positions or becoming active in environmental student groups. In addition, respondents reported engaging in community groups or networks and participating in community meetings or restoration projects. The results of the survey were used to refine the TOC.

Stage 4 - Integrating ongoing evaluation processes into MAD's program structure: The authors felt that it was important to promote ownership of evaluation process, and therefore developed a survey to continue to evaluate the program and measure the impact on MAD participants.

Overall, the authors found that MAD was successful in promoting transformational learning and action competence. They concluded these concepts were interrelated and iterative, and that the program structure supported both. Furthermore, they felt the participatory evaluation process was successful and that it promoted the inclusion of evaluation as part of MAD in an ongoing manner.

The authors acknowledged a number of limitations. One being that students self-selected into the study. Students who got more out of the program may have been more motivated to participate, which means a more critical perspective may have been absent. The authors also indicated that the group nature of the workshops may have suppressed discussion of critiques or challenges. The results of this study are specific to this program and context. The authors indicated that further exploration of participants' experiences, as well as evaluations of similar programs, would better contextualize these results.

The authors recommend that those seeking to develop a program that encourages environmental leadership adopt a similar structure to MAD. Combining an intense residential experience with long-term support aided participants in becoming leaders and making personal and community-level changes. In addition, the authors recommend engaging in participatory evaluation processes, which can be particularly helpful for programs that do not have clearly articulated goals. This process can help the program better achieve stated goals and outcomes, as well as facilitate ownership over ongoing evaluation processes without the support of an external evaluator.