When It Comes to Scale Questions, It’s Not What You Ask But How You Ask It

A client came to me a few months back asking some questions about questionnaire design best practices. We had a few conversations about how she and her team could improve some of their existing questionnaires and guidelines for designing mobile-friendly research. Of course, being that guy, I had to tell her sorry, but for the most part her beloved grids have got to go – at least their current incarnations. As we continued down this path we began to work through some of the different ways that her team uses scales and the critical place they occupy in most of their surveys. While the advice I was giving, based on experience and familiarity with academic work on the subject, was sound (or at least I’d like to believe), it occurred to me that it had been a while since my colleagues and I at Research Now SSI had dug into the topic.

Serendipitously, at the time this was rolling around my head, Ted Saunders of MaritzCX and Zoe Dowling of FocusVision and I were discussing the future of our industry and some of the challenges our clients are having – just chatting about the nerdy stuff researchers talk about when given the chance. For some reason we kept coming back to scales. With today’s mobile survey UX, are 10-point scales off the table for smartphone users? Will participants hold their phones in landscape and if so, will that positively impact data quality with longer scales? How impactful is the assumed primacy effect? Why did we have a growing list of questions without clear answers? Ultimately, we agreed that while the terrain around how to use scales in survey research is fairly well trodden, solutions and best practices for presenting such questions deserved additional attention. And thus, an idea was born! While we kept mobile in mind as we planned the study, we hatched a plan for exploring various scale question presentations across all (or almost all) devices: smartphones, tablets, and computers.

We’re looking forward to sharing the results at NEXT, April 30 – May 1 in New York City – and hope you’ll join us there. Without spoiling the surprise, we can at least say that we’ve found some compelling differences between usage of scales of different lengths (10 point vs. 5 point) as well as with the screen orientation in which a scale is displayed (portrait/vertical vs. horizontal/landscape). Additionally, you may be surprised at how factual and opinion-based questions fared in our test.

Ultimately, we hope to remind you that when writing questionnaires and designing them to be presented on mobile devices, it’s not just what you ask, but how you ask it. And specifically, when it comes to crafting scale questions, attention to difference matters.

NewsTechniqueRoddy Knowles – Research Now – SSI