Correlating observed and self-reported student engagement with environmental literacy outcomes

One of the goals of environmental education (EE) is to promote environmental literacy, but this is only successful when EE programs effectively engage students. Yet student engagement is difficult to accurately measure. For example, a teacher may think a student who nods and smiles during a lesson is actively engaged, but that student may not be able to answer a simple question on the material when quizzed. Conversely, a student who may seem disengaged, perhaps by fidgeting and looking around, might perform well during assessment. The inconsistencies between observable cues and material retained highlight the difficulty of measuring cognitive engagement. Similarly, emotional and behavioral engagement are difficult to connect to learning and retention, despite being easier to observe. Some research has explored student engagement in EE, however, it has not targeted informal settings like outdoor EE programs. For this study, the researchers examined how two forms of student engagement, observed and self-reported, are associated with environmental literacy.

This study was conducted at the NorthBay Educational Foundation over the course of nine weeks from February 2015 to November 2015. NorthBay is a residential EE facility located on the Chesapeake Bay in Maryland that middle school students from around the area attend. After completing a four-week pilot to develop and refine study tools, researchers collected data from 15 sixth grade groups representing 17 different schools. These groups were hand-selected to reflect the diversity of the students who attend NorthBay. During the five-day program, students attended six, two-hour long EE lessons across the 90-acre campus. Researchers observed the students' behavioral and emotional engagement during lessons, using a five-point Likert scale to rate negative engagement, non-engagement, or positive engagement. The engagement levels were categorized in six ways: disruption, frustration, detachment, attention, effort, and enjoyment. Students also completed a short post-lesson survey about their engagement and environmental literacy, responding to statements such as “I enjoyed the lesson” or “This lesson made me appreciate nature more than I did before.” In total, the researchers analyzed 1,279 student surveys from 80 observed lessons.

The researchers found that student's self-reported engagement was the only predictor that significantly correlated with positive environmental literacy outcomes. This finding supports decades of educational psychology findings that warn educators against using observed student engagement as a reliable indicator of actual student learning. The researchers discussed three key reasons why self-reported engagement is a better way to predict student learning. First, cognitive engagement cannot be readily observed. Second, the skills or bias of the observer may cause misinterpretations in recorded behavioral or emotional engagement, especially because these can vary significantly and require correct interpretation. Third, some students may disguise emotions or limit obvious engagement due to peer influence or cultural norms. Despite these limitations, observing student engagement is still an important tool for educators to remain responsive to students' interests, learning styles, prior experiences, and competencies. Previous research has shown that adapting programs in the moment based on observable cues can help educators better support student learning. However, as the researchers reiterate, “smiles, in and of themselves, do not equal learning.”

This study had limitations. First, its single-case design of evaluating one program in one location limits its generalizability. Second, the survey contained items about both environmental literacy and self-reported engagement. This can lead to upward bias in which student responses in one category are influenced by the previous category of questions. Finally, because the self-reported survey was completed in a group setting, social desirability bias may have caused students to inflate their level of engagement.

The researchers recommend environmental educators recognize both the importance and limitations of evaluating programmatic success based on observed student engagement. Educators should continue to use observable student engagement to be more responsive while teaching, but this cannot be a substitute for systemic evaluations of desired student learning outcomes.