# Methods and techniques of data collection > Surveys, interviews, and observation compared with how organizations collect data at scale today, plus batch, cursor, and CDC ingestion. Source: https://www.erathos.com/en/blog/data-collection-methods-and-techniques Em português: https://www.erathos.com/blog/data-collection-methods-and-techniques Published: 2026-09-11 Category: Data Ingestion ![Methods and techniques of data collection](https://cms-media.erathos.com/Methods and techniques of data collection.png) Every analysis starts with the same problem. Somebody has to go get the data. A student writing a thesis runs interviews. A public health team mails a questionnaire. A data team copies rows out of a CRM into a warehouse on a schedule. All three are collecting data, and the choices they make at this step decide what the analysis can say later. This guide covers the classic research methods first (surveys, interviews, focus groups, observation, experiments, document review), with the tradeoffs from methodology sources. Then it covers how organizations collect data from their own systems today, and the separate step of moving that data somewhere you can query it. ## What are the methods and techniques of data collection? Data collection is the [systematic process of gathering observations or measurements](https://www.scribbr.com/methodology/data-collection/) from a defined source. A method is the approach (a survey, an interview, a sensor feed). A technique or instrument is the specific tool that does the capturing (the questionnaire, the interview guide, the tracking script). The sources you can collect from are [people, documents, business systems, websites, experiments, sensors, or existing data sets](https://www.simplilearn.com/what-is-data-collection-article). Methodology guides sort methods along two axes before they list any of them. The first axis is origin, whether you collect the data yourself or reuse data somebody else collected. The second is form, whether the data comes out as numbers or as words. Those two axes matter more than any fixed list. Different guides count four, five, or seven "types", and there is [no single universal list](https://www.simplilearn.com/what-is-data-collection-article). What stays constant is the pair of questions behind them. ## What are the four types of data collection techniques? The four types are primary quantitative, primary qualitative, secondary quantitative, and secondary qualitative. They come from crossing two questions: where the data comes from (you collect it, or someone already did) and what form it takes (numbers or words). Primary data collection happens when researchers get information [directly from the original sources](https://www.indeed.com/career-advice/career-development/methods-of-data-collection). Secondary collection reuses information gathered by previous research or by another system. Quantitative data is [expressed in numbers and analyzed with statistical methods](https://www.scribbr.com/methodology/data-collection/). Qualitative data is expressed in words and analyzed through interpretation and categorization. Type Origin Form Example Primary quantitative You collect it Numbers A closed-question survey you send to your customers Primary qualitative You collect it Words Recorded interviews with your users Secondary quantitative Someone else collected it Numbers Census tables, your own CRM's order history Secondary qualitative Someone else collected it Words Meeting notes, policy documents, support tickets The same method can land in more than one cell. A survey with open text boxes produces qualitative data. Observation can produce counts (quantitative) or field notes (qualitative) depending on how you design it. So the type describes the data you end up with, and the method describes how you got it. ## What are five methods for collecting data? Five common methods are surveys, interviews, focus groups, observation, and document review. Experiments and ethnography round out the seven you see in research methods courses. Experiments are the only method in the group built to [test a causal relationship](https://www.scribbr.com/methodology/data-collection/). Method Data it produces Cost and effort Main limitation Good for Survey / questionnaire Standardized answers, mostly quantitative [Easy and cost effective](https://reu.charlotte.edu/toolkit/analysis/data-collection-methods/) Self-report bias, little depth or context Perceptions, attitudes, and knowledge in a predetermined sample Interview Recorded, transcribed qualitative responses [More time and resources](https://pmc.ncbi.nlm.nih.gov/articles/PMC4857496/) than surveys, to run and to analyze Interviewers need training so answers stay comparable Individual stories and in-depth accounts Focus group Qualitative group discussion More efficient than one-to-one interviews because you hear from a small group at once A few loud participants can dominate; peer pressure can skew answers Reactions to other people's views, spotting agreement and disagreement Observation Behavior records, quantitative or qualitative Easy to collect Subjective, and what you see may not apply to all participants What people do, as opposed to what they say they do Experiment Quantitative measurements under controlled conditions Needs a manipulated variable and a control Only fits questions with a clear cause and a measurable effect Cause and effect Document / record review Secondary evidence, words or numbers [Easy and cost effective](https://reu.charlotte.edu/toolkit/analysis/data-collection-methods/) Documents were made for another purpose; the research question has to guide which ones you pick Historical practice, official positions, context for other findings Ethnography Long-form field notes and observations Long time in the field Hard to standardize or repeat Understanding a group's culture and daily context The optimal focus group has [8 to 10 people](https://pmc.ncbi.nlm.nih.gov/articles/PMC4857496/), enough for a rich discussion while every member still gets to speak. Interviews will generally yield richer data than surveys but take more time to conduct and analyze. Observation records what people do in their everyday behavior, instead of relying on what they remember or how they perceive themselves. Document review covers more than written reports. Documents include [reports, policies, records, films, photographs, meeting notes, and checklists](https://pmc.ncbi.nlm.nih.gov/articles/PMC4857496/). Reviewing them can track how an institution's views changed over time, or give context to findings from an interview or survey. The National Science Foundation [recommends a mixed-method approach](https://reu.charlotte.edu/toolkit/analysis/data-collection-methods/), and document review is a cheap way to add a second method to a study. ## How do organizations collect data at scale today? Organizations collect data automatically from business systems, websites, apps, APIs, uploaded files, and sensors. The research methods above still apply (a customer survey is still a survey), but the volume comes from systems that record events as a side effect of doing their normal work. Source What it records Typical problem Transactional databases (ERP, CRM, billing) Orders, customers, invoices, status changes Records get updated and deleted in place, so history is lost unless you capture changes Web and app event tracking Clicks, page views, signups, purchases [Tracking errors and inconsistent event definitions](https://www.simplilearn.com/what-is-data-collection-article) Application logs Errors, requests, timings High volume, loose structure Third-party APIs (ads, payments, support tools) Whatever the vendor exposes Rate limits, authentication, and schema changes on the vendor's side Forms and file uploads Manual entries, spreadsheets, CSV exports Typos, missing fields, no shared format Sensors and devices Location, temperature, machine state, medical readings Consent, calibration, and security The number of connected IoT devices [reached 18.5 billion in 2024](https://iot-analytics.com/number-connected-iot-devices/)! That count only includes active devices and gateways, so end sensors behind a gateway are left out. IoT here means "Internet of Things", physical devices that report measurements over a network. The primary versus secondary split shows up here too. Buying a data set from a vendor is secondary collection. Instrumenting your own product or robots is primary collection. Brett Adcock, the founder of Figure, described going from purchased data that was scarce and poor quality to building their own collection pipeline for robot training. The same quality rule from research methods holds for automated collection. High-quality collection is difficult, and [few AI researchers are trained in collection methods](https://arxiv.org/abs/2403.01208v2), which is why survey methodologists argue their field can improve how training data gets gathered. A tracking script with a vague event definition is the operational cousin of a badly worded survey question. ## What is the difference between data collection and data ingestion? Collection captures information at its source. Ingestion [moves that information into a database, warehouse, lake, or another system](https://www.simplilearn.com/what-is-data-collection-article) for storage and processing. A survey form collects. The pipeline that loads the responses into BigQuery ingests. Most ingestion today follows the ELT pattern: [extract, load, transform](https://www.erathos.com/en/blog/etl-vs-elt-key-differences?utm_source=blog&utm_medium=organic&utm_content=bydefault&utm_campaign=methods-and-techniques-of-data-collection). Raw data gets copied into the warehouse first, and the cleaning and modeling happen there with SQL. This keeps the raw copy around, so you can re-run a transformation later without going back to the source. A [pipeline in Erathos](https://www.erathos.com/en/pipelines?utm_source=blog&utm_medium=organic&utm_content=bydefault&utm_campaign=methods-and-techniques-of-data-collection) is the whole path between one of your tools and your warehouse: extraction, scheduling, and loading. You pick the tables, the frequency, and the update type. Each run records run time, rows processed, and errors with context, and sends a Slack or email alert when something fails. ### Batch, cursor-based, or Change Data Capture The update type decides what the pipeline can capture. A batch sync copies the whole table each run. A [cursor-based sync](https://www.erathos.com/en/blog/cursor-based-sync-vs-change-data-capture?utm_source=blog&utm_medium=organic&utm_content=bydefault&utm_campaign=methods-and-techniques-of-data-collection) looks at a column like updated\_at, stores the highest value it saw, and next run pulls only rows above it. Cheap, but a deleted row has no updated\_at to find, so deletes stay in the warehouse forever. And if a row changed three times between runs, the cursor only stores the last value. Change Data Capture (CDC) reads the database's own transaction log instead of querying the table. In PostgreSQL that log is the WAL (Write-Ahead Log), where the database records inserts, updates, and deletes in order before applying them. Because a delete is an entry in the log, CDC captures it. The delivery to the warehouse can still run as a scheduled batch. CDC changes what gets captured, and the batch schedule still decides how often it ships. CDC on PostgreSQL needs [four settings](https://docs.erathos.com/connectors/databases/postgresql) on the source database: Setting Value Why wal\_level logical Makes the WAL carry row-level detail max\_replication\_slots 10 or higher Reserves a slot for the CDC reader to track its position max\_wal\_senders 10 or higher Allows connections that stream the log max\_slot\_wal\_keep\_size Anything other than -1 (the docs example uses 5GB) Limits how much WAL piles up if the reader disconnects, so it cannot fill the disk The first three need a database restart. The last one supports hot reloading. By default, an update or delete in PostgreSQL only logs the primary key, so the tables you capture also need their replica identity set to FULL to carry the full before and after state of each row. Two snapshot modes control the first run. The initial mode takes a full snapshot of the table, then streams the changes that follow from the WAL. The no\_data mode skips the snapshot and streams only the changes already available in the WAL. For sources outside the built-in catalog, a [custom connector](https://docs.erathos.com/connectors/custom-connectors) is a GitHub repository that Erathos watches. Every push rebuilds the connector image, and it shows up in the source catalog with a Custom badge. Cursor-based sync still fits small tables with a reliable update column, no physical deletes, and a need for only the latest state. CDC pays off on large tables, tables with real deletes, and cases where you need the full history of each row. ## How do you choose the right data collection method? The question you need to answer decides the data form, the source, and the instrument that fit it. The choice depends on [the research question, the type of data needed, and the resources and time available](https://www.questionpro.com/blog/data-collection-methods/). A checklist drawn from the methodology guides above: 1. Write the question. "Why do users churn" points to interviews. "How many users churn each month" points to a database query or a survey. 2. Pick the data form. Numbers for measuring how much or how many, words for understanding why. 3. Pick the source. Check whether the data already exists (secondary) before you collect it (primary). 4. Define the population and the sample. Who can respond, and how you reach them. 5. Build and pilot the instrument. A questionnaire, an interview guide, a checklist, or a tracking plan with named events. 6. Plan for bias. Self-report bias in surveys, dominant voices in focus groups, inconsistent event names in tracking. 7. Handle consent, confidentiality, and anonymization for any data about people. 8. Decide where the data goes and how it gets there. On sampling, a high or low response rate says little about data quality by itself. Survey response rates [do not relate well to nonresponse bias](https://www.bls.gov/osmr/response-rates/), though the pattern of rates over time can tell you something about the process. US federal surveys with a unit response rate [below 80 percent must run a nonresponse bias analysis](https://statspolicy.gov/FCSM/groups/nonresponse-bias/). For scale, the American Community Survey's housing unit response rate was [82.9 percent in 2024](https://www.census.gov/acs/www/methodology/sample-size-and-data-quality/response-rates/). A low rate is a prompt to check who did not answer, and a high rate still leaves that question open. For automated sources, step 6 means [shared definitions for customers, dates, events, products, and metrics](https://www.simplilearn.com/what-is-data-collection-article) across the systems you combine. For step 8, a managed pipeline replaces hand-written scripts. The [pipeline guide](https://www.erathos.com/en/blog/how-to-build-and-manage-data-pipeline?utm_source=blog&utm_medium=organic&utm_content=bydefault&utm_campaign=methods-and-techniques-of-data-collection) covers the data quality checks and scheduling that go with it. ## Recap - Data collection methods sort along two axes: primary or secondary, quantitative or qualitative. - Surveys, interviews, focus groups, observation, experiments, and document review cover most research questions, and mixing two of them is the standard advice. - Organizations collect at scale from databases, event tracking, logs, APIs, files, and sensors. - Collection captures data at the source. Ingestion moves it into a warehouse, and the update type (batch, cursor, CDC) decides whether deletes and intermediate changes survive the trip. If the ingestion half is the part you are working on, you can [try Erathos free for 14 days](https://app.erathos.com/signup?utm_source=blog&utm_medium=organic&utm_content=bydefault&utm_campaign=methods-and-techniques-of-data-collection) and connect a PostgreSQL table with CDC to your warehouse.