Environmental educators often have a clear idea of what best teaching practices are, yet it is difficult to critically evaluate one's own practice while also teaching. External evaluators, such as supervisors, can help, but the process of conducting evaluations often disrupts and changes the teaching day. This makes the evaluation results less helpful for improving practice. In this study, the researchers implemented an evaluation method traditionally used in retail service industries to evaluate the teaching practices of scientists at a science festival expo, a conference gathering that includes a large exposition hall with booths where scientists can highlight their research and interact with other researchers and conference participants. The researchers found that the method was informative, more accurate than other methods, and minimally disruptive.
The evaluation method, called the mystery shopper protocol, was developed in the 1940s as a way of evaluating customer service. In this protocol, a trained shopper completes a transaction as a typical customer while paying attention to predetermined areas of focus, such as setting, products, and employees. After the interaction, the trained shopper immediately evaluates his or her experience against the predetermined protocol. Although the employee often knows that he or she is going to be evaluated, he or she doesn't know how or exactly when the evaluation will happen. This mystery element ensures a more genuine interaction and evaluation.
Although the protocol was originally used for investigating the transaction quality between a customer and a service provider, the researchers adapted it to assess educational aspects of interactions between scientists and their audiences. To do this, the researchers used a research-based approach to best science teaching practices developed by researchers Davies (2008) and Nisbet and Scheufele (2009) as their evaluation protocol. The protocol included 22 items divided into three sections; the items focused on describing the scientist's booth, the interaction of the scientist and the public, and an overall summary of the experience. The trained mystery shoppers participated in extensive training regarding how to use the protocol and conduct a mystery shopping experience. Overall, the mystery shoppers observed 192 booths.
To compare the effectiveness of the mystery shopper evaluation method, the trained researchers also conducted a 14-item evaluation survey with conference attendees. The protocol was similar to that used by the mystery shopper; the researchers collected data from 186 attendees.
To analyze the results, the researchers coded best practices as either occurring (a score of 1) or not occurring (a score of 0). Those results were compared to the overall booth rating, the time spent at the booth, and whether the message of the booth was clear. In comparing the two methods of evaluation (mystery shopper with surveys), the mystery shopper protocol was more in-depth: 93% of mystery shoppers provided rationale for their ratings, while only 54% of survey participants did so. Also, mystery shoppers were more critical in their evaluations, giving an average overall score of 3.11 (as opposed to 4.44 by the survey participants). The researchers concluded that these results indicated that the mystery shopping method was not only an effective way to observe scientists' best teaching practices, but the mystery shopper protocol also offered a more in-depth and honest evaluation.
The results of implementing this evaluation method suggest that it could offer a more informative, and possibly accurate, way of evaluating teaching practices. Similarly, environmental educators might consider using a mystery shopping evaluation method to help critically examine and improve their teaching practices.
The Bottom Line
<p>Evaluating and reflecting on one's own teaching practice is challenging, because it is difficult to become grounded in an unbiased, outside perspective. Evaluations conducted by outside observers are also often difficult, because they can disrupt the teaching day and are sometimes conducted by people unfamiliar with the program, topic, or setting. The mystery shopping method of evaluation, in which a highly trained mystery participant, or “shopper,” observes and evaluates based on effective teaching practices, provides an alternative evaluation option. In comparison to survey methods, the method may provide more in-depth, honest, and less-biased feedback. Using this method may help educators reflect on and improve their teaching practices.</p>
Research Partner
Research Category