← All topics

Methods, methodology & rigor

Data quality (general)

41 indexed claims · 14 guests · peaked 2026-Q1

Data quality (general)

Overview

Data quality sits at the center of market research's most contested debates: how severe the problem actually is, who owns it, and whether the industry's reflex to clean data after collection is fundamentally misguided. These conversations bring together fraud-detection vendors, panel operators, academic researchers, and platform executives — all circling the same uncomfortable truth that decades of optimizing for speed and cost have left the industry struggling to trust its own numbers. The dominant tension is between remediation and discipline: a filter mindset that accepts contamination as a given and cleans after collection, versus a discipline mindset that builds quality in from survey design through sample sourcing. A parallel thread asks who bears accountability — and whether market dynamics rather than moral persuasion will ultimately force the reckoning that professional exhortation alone has not.

From the corpus

  • Counterintuitively, LLMs excel at querying unstructured data (PDFs, transcripts, open ends) but struggle with structured data; properly querying structured data on the fly is a tremendously harder technical challenge than reading verbatim, and remains the holy grail for the next generation of systems. — Shanon Adams, Ep. 49 · cl-50th-004
  • Detecting low-quality respondents requires building smart models that triangulate over many signals (the 'sum of all factors' or digital body language) rather than reaching for simple binary rules; data quality is a perpetually multifaceted problem and no blunt instrument will durably solve it. — Lev Mazin, Ep. 24 · cl-lev-solo-014
  • The data quality problem resembles the antivirus industry's problem: the work is never done because new threats and bad actors emerge continuously, and AI is now both the attack vector and a source of defenses; the same technology that erodes data quality can be used to spot and improve it. — Shanon Adams, Ep. 49 · cl-50th-006
  • Researchers should apply the same empathy to survey respondents that they apply to the consumers they are studying — translating questions into the respondent's language and generation, and remembering that some bad actors are economically desperate humans rather than fraud-bots. — Shanon Adams, Ep. 49 · cl-50th-007
  • Crude defenses against AI-generated survey responses (e.g., blocking copy-paste) harm honest, careful respondents — non-native English speakers checking grammar in Word — more than they catch fraud; the right response is multi-signal triangulation, not blunt instruments. — Lev Mazin, Ep. 24 · cl-lev-solo-013
  • Data quality problems aren't unique to primary research; social listening data carries an even larger and less-quantified bad-data load (estimates of 15-68% bot/fake content on Twitter), and method choice should follow business problem, not data-source preference. — Dave Ritter, Ep. 16 · cl-dave-ritter-008
  • Fraud prevention requires a full-stack approach beginning with sample source selection and targeting — technology gates like dtect occupy only one layer; poor sourcing and targeting produce bad data that is not technically fraud but functions the same way. — Roddy Knowles, Ep. 8 · cl-roddy-knowles-009
  • Synthetic data is only as reliable as the data used to build the underlying models — if the training corpus contains fraud-contaminated surveys, the models will replicate those errors, making data quality a prerequisite for synthetic respondent validity. — Roddy Knowles, Ep. 8 · cl-roddy-knowles-023

Contributing guests

Episodes in Data quality (general)

See Methods, methodology & rigor mapped across all quarters →