← All topics

Methods, methodology & rigor

Survey methodology, design & limits

52 indexed claims · 22 guests · peaked 2026-Q1

Survey methodology, design & limits

Overview

Survey methodology surfaces across these podcast episodes not as a solved problem but as an ongoing crisis of craft. The dominant tension runs in two directions simultaneously: outward, toward what surveys fundamentally cannot measure — the 20–30% correlation between stated and actual behavior stands as the industry's most uncomfortable open secret — and inward, toward the self-inflicted damage of poorly designed instruments. DIY platforms, kitchen-sink questionnaires, and researchers who pretest with colleagues rather than civilians have industrialized low-quality data. Connecting both currents is a deeper argument about the respondent's status in the research ecosystem: multiple guests contend that treating survey participants as a commodity to be extracted and churned is depleting human capital the industry cannot replenish, and that better methodology begins not with better technology but with more basic professional honesty about what surveys can and cannot do.

From the corpus

  • AI can draft a starter questionnaire but tends to generic output without specific prompting, and it does not yet detect bias in the questions it generates — including order effects (asking how healthy fries are before asking how much you like them inflates the healthiness weight on liking) — so survey-design training is still required. — Ernest Baskin, Ep. 48 · cl-ernest-baskin-010
  • The three most common respondent complaints about panel companies — being lied to about incentives (especially being screened out after long surveys with no compensation), enduring overlong convoluted surveys, and privacy breaches — are predictable, addressable, and were systematically designed against in PaidViewpoint. — Lev Mazin, Ep. 24 · cl-lev-solo-009
  • Post-purchase 'how did you hear about us?' questions are among the worst survey use cases: respondents remember only the most salient touchpoint from a multi-channel journey, and investing based on those answers systematically overweights already-working channels while creating a permanent blind spot for new ones. — Rand Fishkin, Ep. 19 · cl-rand-fishkin-007
  • Synthetic or simulated respondent data in pharma is not yet trustworthy for primary research decisions; the bar for adoption should be a properly controlled side-by-side comparison — same survey design, same questions, real vs. synthetic participants, with the training provenance of the synthetic sample disclosed. — Shawn McKenna, Ep. 9 · cl-shawn-mckenna-010
  • The industry has a long way to go on inclusivity in research design because it still thinks about it in black-and-white terms; true inclusivity means recruiting for diverse cognitive styles — expressive people for emotional mapping, journalers for diary studies — not just demographic checkboxes. — Louisea Hudson, Ep. 15 · cl-louisea-hudson-012
  • Surveys and interviews are genuinely poor instruments for behavioral sequence questions — 'what websites do you visit when planning X?' — because people cannot accurately self-report their own browsing paths; they are not lying, they are simply cognitively unable to reconstruct the sequence. — Rand Fishkin, Ep. 19 · cl-rand-fishkin-006
  • Asking a colleague to review a survey is how jargon survives; the right pretest is outside the company — a parent, a grandparent, a regular customer — and watching them take the survey while they report what words mean and which questions confuse them is how surveys actually get sharper. — Ernest Baskin, Ep. 48 · cl-ernest-baskin-023
  • AI has not produced compelling use cases in pharma primary research and survey design; NLP-based open-end coding dates to the 1970s, AI-designed surveys miss clinical nuance, and the productivity gains in AI-assisted data categorization still require human supervision of the output. — Shawn McKenna, Ep. 9 · cl-shawn-mckenna-008

Contributing guests

Episodes in Survey methodology, design & limits

See Methods, methodology & rigor mapped across all quarters →