Since panel respondents are offered reward points for their participation, some may be tempted to rush through the questions without regard for truthfulness or validity of their responses.
Web panel providers and market research firms work hand in hand to minimize such behaviour.
The following common control measures may be applied either during completion of the survey to terminate it in real time, or retroactively once results become available.
- Response time: Questionnaires or specific questions which have been completed unrealistically quickly can be ignored;
- Quality: Statements may be inserted in attribute matrices and require a predetermined answer (i.e. Please score 8 on 10 for this statement);
- Internal consistency: Validation formulae can be integrated into the programming such that certain answers must correspond to one another to be plausible (i.e. I bought a 2,500 square foot house for $10,000).
- Origin control: Web panels can also block the ISP of hosting companies which are known to be used by « avid respondents ».
The advantages of such practices is to control for those respondents who will do a poor job answering survey questions, and make those who respect them more engaged and pay greater attention to questions. In both cases, the goal is to increase the validity of results.
There is also a potential for undesirable consequences. Although most respondents who fail to see trap questions tend to do so systematically (they don’t read through the questions fully or punch in the same answer for all matrix attribute questions), some respondents who miss one trap question might have simply misread it or had a mouse-click error.
As such, it may be preferable to filter out at the end of the survey, so that closed-ended responses can be compared with open-ended questions, which may contain more thoughtful answers.