The Hawthorne Effect or Observer Bias in User Research

Definition: The observer bias or the Hawthorne effect refers to the fact that people behave differently when they know they are observed. 

The Hawthorne Effect in UX and How to Mitigate It

“How can I observe people’s natural behavior during a field study if they know I am there?”

According to this Forbes opinion piece, there are multiple ways in which very productive employees get punished at work, so why should anyone want to stand out or show their vulnerabilities?

Mitigation Strategies

Do not make people feel judged.  

Consider longer sessions to allow participants to build a rapport with the facilitator. 

User Testing

Mitigation Strategies

Design natural task scenarios 

Ease people into testing with small talk and giving subtle feedback while avoiding being overly friendly.

Assure participants that their performance is not shared outside of the study and that their feedback would be very helpful.

Explain to participants that you are interested in their honest feedback and behavior.

Explain that the study is meant to test the design, not the participants. 

Surveys

Mitigation Strategies

Prompt participants to upload additional materials.

Use direct and indirect questioning in your survey design. Examples of a direct question (DQ) compared to the indirect questions (IQ) used in the study:

DQ: “How private do you consider this information?”

IQ: “How frequently do you check [content type]?”

IQ: “How frequently do you forward or otherwise share (e.g., by printing and giving the printed copy) [content type] with your close friends or close family members?”

Diary Studies

Mitigation Strategies

Keep diary-study questionnaires short. ... narrowing down the questions by asking yourself:

1. What do you want to know?
2. Why do you want to know it?
3. What decision will you make based on those answers? 
4. What number do you need to make a decision? 

Explain what you want from your participants and why their honest responses matter. 

Provide reasonable accommodations.

Conclusion

Why do we still conduct studies if validity might be an issue? Because data is better than guessing! However, it is important to critically assess the findings and recommendations based on diversified studies and secondary research. We need to be aware that our study design and the wording of the questions can affect users' behavior. Yes, the Hawthorne effect will primarily be associated with people's tendency to alter their behavior when they are being observed. However, it can also lead to benign applications, prompting accountability, awareness, or reflection on certain behaviors (e.g., spending time on a mobile device) so people can manage them and make informed decisions.

In user experience, the observer bias can serve as a foundation for more empathy for the users and for better study design –– would you want somebody to watch you when you work or live your life? How can you make that experience less intrusive, more enjoyable, and a safe space for people to share?