← All topics

Methods, methodology & rigor

Sample integrity & panel quality

24 indexed claims · 10 guests · peaked 2026-Q1

Sample integrity & panel quality

Overview

Sample integrity and panel quality sits at the methodological heart of quantitative research — and these podcast episodes reveal it to be under pressure from multiple directions simultaneously. The shift from in-person to online surveying traded away structural quality controls for speed, leaving the door open to bot infiltration, professional respondents, and fraud now industrialized by AI. Meanwhile, the industry's dominant accountability vocabulary — 'our quality is good' — has rarely been backed by auditable evidence, a gap the guests explore with unusual candor. The central tension running through these conversations is not technical but ethical: whether respondents are partners in a knowledge-building enterprise or a commodity to be extracted from until they churn.

From the corpus

  • The decennial census and the American Community Survey supply the frame that fuels essentially every piece of US quantitative research — and even shapes qualitative approaches — across both the private sector and government, which is why researchers should care about a complete and accurate 2030 census even if they never think about it day to day. — Howard Fienberg, Ep. 57 · cl-howard-fienberg-023
  • The three most common respondent complaints about panel companies — being lied to about incentives (especially being screened out after long surveys with no compensation), enduring overlong convoluted surveys, and privacy breaches — are predictable, addressable, and were systematically designed against in PaidViewpoint. — Lev Mazin, Ep. 24 · cl-lev-solo-009
  • Fraud prevention requires a full-stack approach beginning with sample source selection and targeting — technology gates like dtect occupy only one layer; poor sourcing and targeting produce bad data that is not technically fraud but functions the same way. — Roddy Knowles, Ep. 8 · cl-roddy-knowles-009
  • aytm's core insight was reframing market research as a two-sided marketplace — connecting people with questions to people with answers — rather than as a survey-tooling business; this required building the panel community before launching the product. — Lev Mazin, Ep. 24 · cl-lev-solo-003
  • There is a meaningful difference between being a panel provider and a panel company: panel companies treat respondents as a commodity to extract ROI from before churn, while a panel provider treats respondents as partners in an ecosystem of trust. — Lev Mazin, Ep. 24 · cl-lev-solo-008
  • The most precious commodity for any human being is the life of their time; respondents who spend minutes or hours on a survey in exchange for nothing are being demeaned, and panel platforms that allow this are eroding the ecosystem they depend on. — Lev Mazin, Ep. 24 · cl-lev-solo-010
  • Market research can be done cheaply and in a single day on cheap survey-and-panel platforms — as long as you keep the expertise in-house and don't offload it to a consultant; the expertise, not the platform, is the load-bearing input. — Ernest Baskin, Ep. 48 · cl-ernest-baskin-008
  • Mall intercept and phone-era research had quality protections — a real interviewer never interviewed a bot, never let one person take the survey six times, never let a speeder through — that the industry traded away for online speed. — Charlie Grossman, Ep. 34 · cl-charlie-grossman-002

Contributing guests

Episodes in Sample integrity & panel quality

See Methods, methodology & rigor mapped across all quarters →