From time to time, The Prospectory has to design and conduct surveys as part of a research project. Surveys can be useful for collecting data on behaviour from a larger sample of people than is possible with free-form interviews or group discussions. But, wherever possible, I would always favour the latter. Free-form techniques are more likely to yield useful and suprising insights and they are also a lot more reliable.

We try to keep survey questions as concrete as we possibly can having learned from experience that the less concrete the question, the less reliable the answers you get. Even when asking about concrete behaviours, it’s  more reliable to ask what people actually did yesterday or to describe the last time they did X or Y than to ask them to judge how often they do it in general.

It’s even worse when asking people in surveys to rate their attitudes to any topic. How reliable are their answers? Research suggests not very! People will happily generate and argue for any viewpoint if that is neccesary to justify something they do or have done or a position they believe they hold.

In a recent study, participants completed a survey concerning their moral attitudes. When asked to read through their responses and explain a few of them, a trick was used on 2 of the questions to display the opposite attitude rating to the one they had actually recorded whilst completing the survey.

The experiment then recorded whether particpants would detect the change or whether they would justify and argue for the opposite view of what they had stated on the survey only moments earlier. The sobering result was that 69% of participants failed to detect at least one of two changes. And they often constructed coherent and unequivocal arguments supporting the opposite of their original position. As the authors point out, the results suggest that there is plenty of flexibility in our attitudes and self-attribution and post-hoc rationalisation play a critical part in the views we hold at any point in time.

So, design and interpret attitudinal surveys with care – our brains are highly tuned at generating on the fly reasons why what we just did or said (or think we just did or said) makes perfect sense.