The Validity of Self-Report Measures of Pro-Environmental Behavior

Many environmental problems could be dramatically reduced or solved by the collective efforts of numerous individuals reducing their resource consumption and engaging in other types of pro-environmental behavior. As such, fostering pro-environmental behaviors has become one of the primary goals of many environmental education efforts. To evaluate the effectiveness of these efforts, self-report measures are commonly employed. Self-report measures ask individuals to quantify their own environmental behaviors, typically through surveys or interviews. A critical assumption of these self-report tests is that they accurately reflect individuals' actual environmental behavior. Previous research regarding the validity of this assumption, however, has suggested that self-reports are only weakly associated with actual behavior. Because these self-report tests are easier and often more feasible to gather than objective data on individuals' behavior, it is critical to understand to what degree they are valid and what factors, if any, affect their validity. This paper's authors aimed to address these questions through a detailed meta-analysis of studies that have included objective as well as self-report measures to evaluate the same environmental behavior.

First, the authors outlined several criteria for the studies they included in this meta-analysis. They focused on studies that examined associations between self-reported measures of a pro-environmental behavior and an objective measure of the same behavior. These behaviors ranged from assessments of water usage, energy consumption, and recycling behavior to heads of households' estimates of forests they had cleared on their land that year. To be included, the self-report and objective measures had to be from the same individual or household. The studies also had to include the necessary primary, quantitative data for the authors of this paper to perform their analysis. After an extensive search, the authors found 15 studies that met the criteria to be included in their analysis. These 15 studies included data from 6,260 individuals or households and examined 19 pro-environmental behaviors.

Three objective measures of environmental behavior were used by the studies included in the analysis: device measurement, trained observers, and peer ratings. Device measurement in these studies involved obtaining readings from water, gas, and oil meters in a household. This is likely the most objective measure; however, only a limited number of behaviors lend themselves easily to this kind of measurement. Trained observers are skilled in observing participants' behaviors and can detect a wider range of behaviors, such as how often certain materials are reused in a household. Trained observers, however, are limited in the behaviors they can discern, especially involving daily activities that are private and might change when someone knows they are being observed. Finally, peer ratings involve asking others who are close to the participant, such as spouses or housemates, to report on participants' environmental behavior. This method provides a solution to some of the limitations of trained observers.

Several factors were examined as potential moderators of the validity of self-report measures, including the characteristics of the participants and the methodology used in each study. The participant characteristics included in this meta-analysis were: percent of male participants and mean age of participants. The methodological characteristics examined included: the number of items in the self-report measure (i.e., how many questions were in the survey or interview for measuring the behavior); the number of response options (e.g., two options for “Yes/No” questions versus seven options on a scale from “Never” to “Always”); whether the participants were individuals or households; the nature of the objective measure (device measurement, trained observers, or peer ratings); type of behavior (e.g., recycling or energy usage); year of publication; and study location.

The authors describe a number of features of the meta-analysis data. The average age of the 6,260 participants was 38.2 years old; 32% of the participants were male. Year of study publication ranged from 1984 to 2011. The mean number of items in the self-report measure was eight (range = 1–65), and the mean number of response options was 4.5 (range = 2–7). For the objective measure, 8 of the 15 studies used device measures, 9 studies used trained observers, and 2 studies used peer ratings. In terms of the 19 behaviors studied, 6 studies measured waste behaviors (recycling or reuse of materials), 5 measured energy usage (electricity, gas, or oil), 2 measured water usage, and 1 each measured food consumption, deforestation, and transportation. Finally, three measures included a combination of several different pro-environmental behaviors.

In terms of the overall validity of the self-report measures, the analysis revealed a significant and large effect size (r = .46) of self-reports predicting objective measures of pro-environmental behavior. In addition, 21% of the variance in the self-report measures was linearly associated with the variance in the objective measures. Both of these statistical findings show that the self-reports have some degree of validity for predicting the actual differences in pro-environmental behavior between individuals. The authors, however, caution against thinking that these findings prove self-reports are highly valid. They emphasize that 79% of the variation between self-reports and objective measures of behavior remains unexplained. The authors argue that, while self-reports have some predictive power of actual behavior, it would be false to assume that they are a measure of the actual behavior itself.

In addition to these overall findings, there were significant differences in the predictive power of self-reports between studies: some were very accurate while others were not at all accurate. To help understand these differences, the next round of analysis considered various potential moderators. One of the factors that emerged as significant was the percentage of male participants in the study. Specifically, the greater the percentage of males in a study, the more likely the self-report was to be accurate; study analysis shows, on average, female participants tended to overreport their environmental behavior. The type of behavior examined in the study also had a significant effect: specifically, the validity of self-reports was greatest for deforestation-related behavior (r=.73), followed by energy usage (r=.61), mixed-behavior measures (r=.55), food consumption (r=.31), transportation (r= 30), water usage (r = .29), and waste (r = .28). Another factor that was found to be significant was location of the study: specifically, the 10 studies performed in non-Latin American countries had a larger predictive power (r = .45), compared to the 5 studies performed in Latin American countries (r = .36).

The authors add, however, that several limitations to this study make these significant moderating factors worthy of further study. They determined many factors had no significant moderating effect, including: average participant age, the number of items on the self-report measure, the number of response options, whether participants were individuals or households, the nature of the objective measurement (device measurement, trained observer, or peer rating), and the year of study publication.

In their discussion, the authors emphasize several important implications of this study and suggestions for future research. Given that self-reports are not always highly valid, the authors note that researchers may inadvertently be drawing skewed conclusions about the utility of certain predictive models for behavior. In addition, they argue that it is possible that certain models may overestimate the influence of a particular education intervention on behavior change. By questioning these assumptions and better understanding the factors that skew the validity of self-report measures, researchers may be able to develop better models and educational strategies for addressing environmental behavior.