Today I received an email from Columbus City Schools asking for feedback on the School Choice system. They produced a web survey, http://www.columbus.k12.oh.us/
First:
Issues:
- If the response to the main item is Yes, the respondent can only provide one reason. Maybe it’s implicit that “select one” means “If there was more than one reason, please select what you think was the principal reason.”
- Worse, however, is that there’s no “don’t know” option (yes, you can list it under ‘other’). Asking the respondent to proxy for someone’s attitude or reasoning is probably even more of a challenge than asking to proxy for someone’s actions.
And then:
Issues:
- A “1-5 scale” doesn’t indicate what’s being scaled. Is it agreement with the statement? Amount of benefit? For example, I could totally agree that the lottery system is beneficial, but only that it’s marginally beneficial.
- There’s no Don’t Know or No Opinion option. Worse yet, you’re not allowed to leave the item blank. So, in order to submit my response I have to pick something — perhaps ‘3’ because that’s the closest I can get to neutrality?
And:
Issues:
- Ranking isn’t the problem here; it’s the inclusion of an Other option that’s outside the ranking. If I specify something in the Other Suggestion box, will be it analyzed as as high or low rank?
Throughout, if I tried to respond out of order, even specifying the ranked options out of order (for example, where answering the ranking items), I got this popup… every single time:
Some of these issues are easy to miss when designing a survey instrument. Assuming that all respondents are very knowledgable, and/or can “read between the lines” when you leave out details, is an easy mistake to make. It takes training to identify, and avoid, the sorts of assumptions that alienate respondents who don’t share the designers’ and researchers’ knowledge. Some of the other problems, such as the lack of an ‘Other’ field or the inclusion of an ‘Other’ option without a way to put it into the context of the main item, are not only part of basic survey design knowledge but are actually prevented in many free web survey services. You want to specify an open-ended ‘Other’ option? Great — the software includes it and automatically gives it the rating/ranking/whatever format you’ve given for the item that includes it.