Dear Younger Me: Lessons in Sampling, Panels, and Data Quality from 1994 to Today
- Renato Silvestre
- Apr 7, 2024
- 4 min read
Updated: Jul 22, 2025
As I boarded the plane home from SampleCon, the largest conference dedicated to the marketing research sample industry, I reminisced about my early days in market research. Thirty years ago, I began my journey from sampling and data collection management to market insights, innovations, and artificial intelligence.
With the airport speakers blaring, a quirky thought arose: What if I could give my younger self some advice? Knowing it's fruitless to say, "Be a doctor...," what sampling wisdom would I pass onto my "younger me" THEN and NOW?
Sample Recruitment and Representativeness
1994: When recruiting and interviewing via telephone random digit dialing (RDD), it was critical to build a robust, representative sample list for telephone interviewers to use. A well-maintained database ensured a more reliable and unbiased reach across target populations.
For in-person street intercept interviews, best practices included selecting a diverse mix of locations, times, and days of the week. Every location carries its own inherent population bias, so spreading interviews across multiple environments helped achieve a more cross-sectional and representative sample.
Today: Sampling from online research panels offers a fast and cost-effective way to collect insights, but it comes with critical limitations related to self-selection bias, panel recruitment practices, and administration methods.
Each online sample panel or research community carries inherent biases based on how participants are recruited, managed, and activated. From panel sourcing to respondent engagement strategies, these factors directly influence sample quality and the overall integrity of your market research data.
Is data quality keeping you awake? Book a complimentary data quality strategy session.
Training and Fieldwork Execution
1994: Interviewers were the frontline of data collection. It was essential to train them to ask questions and record responses in a standardized, unbiased manner. They needed coaching to avoid leading respondents, apply screeners consistently, and follow the survey script precisely.
Proper field execution also meant allocating enough time for sample quotas, manual open-ended coding, and data processing, including managing those infamous punched data cards.
Today: With interviews now self-administered through online surveys, the burden of quality shifts to sample partner selection and fieldwork controls. Be intentional when choosing online panel providers, as their recruitment and validation methods directly affect data quality. Sample sources currently include double-opt-in panels, internet publishers, social networks, marketplaces, and exchanges. But beware the "all of the above" approach—blending sample sources without rigorous vetting can seriously compromise data integrity.
To mitigate biases from professional or fraudulent respondents, early completions, and non-representative timing, it's essential to stagger survey completions across at least seven business days. Doing so ensures a cross sectional mix of weekday and weekend participants. Based on our ongoing research-on-research (RoR), extending field duration leads to more representative samples and significantly enhances overall data quality and reliability.
Data Quality and Controls
1994: "Cleaning" the data was relatively straightforward. Common quality issues included duplicate responses, unqualified participants, or fraudulent entries. If your interviewers were well-trained, that covered 80% of the battle. The rest relied on judgment, basic data management, and treatment heuristics to resolve inconsistencies.
It was equally important to document every step of the process to ensure that quality checks were replicable and standardized across studies, especially when multiple field teams were involved.
Today: Data quality is one of the most critical and complex challenges in online market research. Over the last two decades, the industry has shifted from treating panels as long-term, managed assets to a transactional, short-term sample arbitrage model.
Completed interviews are now bought, sold, and resold across a patchwork of exchanges, marketplaces, and programmatic networks, often with limited transparency around source, duplication, or validation.
So yes, Toto! We're no longer in Kansas. To protect your data, you must lift the curtain and insist on transparency at every step of the sampling process. Know where your sample is coming from, how it was recruited, and what controls are in place before it reaches your survey.
In response to data quality issues, STRATEGENCE has launched PANELYTICS, a next-generation sample integrity solution powered by pre-survey and in-survey metadata, as well as advanced machine learning and hybrid AI. PANELYTICS offers a holistic, real-time approach to fraud detection and respondent validation, dramatically improving confidence in your data.
See PANELYTICS in action below.
Parting Thoughts
Well, younger me. I'll wrap it up here. If I say too much, I risk altering your future; you might end up becoming a doctor instead. And then this blog post would never exist.
But don't worry. During a recent data scrub, our quality control algorithm flagged a respondent who claimed to own a quantum teleporter. If I can borrow it, I'll visit you in person. Until then, never forget the golden rule of research: "Garbage in, garbage out."
Want to learn more about how you could benefit from this exciting new service? I'm happy to share details with you and your colleagues in person. Simply book a consultation with me here or email me at rsilvestre@strategence-us.com
Comments