Illuminas Presents Paper Entitled Survey Quality & MaxDiff
An Assessment of Who Fails, and Why at the 2009 Sawtooth Software conference in Delray Beach, Florida.
Andrew Elder, Vice President of Marketing Sciences for Illuminas in Austin, TX presented a paper that he co-authored with Illuminas MS Manager Terry Pan entitled Survey Quality and MaxDiff: An Assessment of Who Fails, and Why at the 2009 Sawtooth Software conference in Delray Beach, Florida. Andrew and Terry propose using MaxDiff (a derivative of conjoint analysis) as a tool to assess respondent quality, since it implicitly measures the consistency across multiple comparisons. Since this consistency can be standardized across topics and audiences, a meta-analysis of 8 MaxDiff studies helps to paint a more specific picture of the low-quality respondent.
Based on this analysis, “speeding” through a survey and “straightlining” ratings questions are shown to have the most significant negative relationship with MaxDiff performance. When poor performers in each task are overlaid, it becomes clear that there is a hierarchy of quality in which people who fail at multiple question types are objectively providing the lowest data quality. However, individuals who fail at an individual task (e.g., those who “speed” without “straightlining” or poor MaxDiff consistency) perform relatively well in the remaining survey components. These insights serve as a cautionary tale to the overzealous rejection of questionable respondents, as this risks biasing results toward those who answer questions in a particular way. By using a variety of question types, researchers are less likely to exclude individuals who favor one response style over another. Overall, we find between 1-4% of individuals justify rejection, varying by study.
For more information please contact us at [email protected].