Blog / Analysis and research impact
Validate AI-Assisted Coding Against Human Researchers
Use a blinded, reproducible protocol to test AI qualitative coding, review disagreements and set defensible thresholds for human intervention.
What AI coding validation must establish
To validate AI-assisted qualitative coding against human researchers, compare independent human and AI decisions on held-out material, review disagreements blind, and check whether the resulting themes remain led by the evidence.
Agreement shows whether coding decisions are consistent. It does not show that an interpretation is defensible. Human coding is a comparison baseline, not unquestioned ground truth: trained researchers can reasonably read an ambiguous passage differently. Human–LLM agreement alone is insufficient for an automation decision (arXiv).
A result applies only to the dataset, codebook, prompt, model version, settings and transcript preparation you tested. Change one of these and retest. This is the practical limit of using AI tools for qualitative research: they can apply a defined framework, but cannot establish whether that framework fits the research question.
First, check whether agreement testing fits the method
Agreement testing suits methods where a stable framework must be applied consistently. This includes structured content analysis, coding-reliability thematic analysis and deductive work with a fixed codebook.
Do not use intercoder reliability as an accuracy test for reflexive thematic analysis. Reflexive coding is situated interpretation rather than a search for one correct label. Braun and Clarke argue that, without one accurate way to code, the logic of intercoder reliability does not apply (Thematic Analysis). Use AI output as material for reflexive discussion instead and document interpretive differences.
Before selecting a measure, state the analytic approach, coding unit and AI role.
| Analytic approach | Suitable validation approach |
|---|---|
| Fixed deductive codebook | Blind human–AI comparison, per-code measures and adjudication |
| Evolving inductive codebook | Review category formation, changes and evidence trails before testing stable codes |
| Reflexive interpretation | Compare interpretations and reflexive notes; do not claim reliability proves correctness |
Our guide to approaches to thematic analysis can help you make that choice explicit.
Freeze the test and build a challenging holdout sample
Freeze the workflow before running the comparison. Record the codebook version, definitions, inclusion and exclusion rules, examples, prompt, model, settings, transcript preprocessing and coding unit.
Keep calibration material out of the validation set. A passage used to refine the prompt belongs to development, not the holdout. Build the holdout around the decisions the research needs to support. Include participant groups, clear positives, ambiguous passages, overlapping codes, negative cases, uncoded passages, rare consequential content, and relevant differences in format or data quality.
Human coders and the AI must work independently. Store original decisions and calculate metrics before revealing labels. Give each source a stable identifier so every code, excerpt and quotation can be traced to its transcript location.
This validation sits inside the full interview-analysis workflow. It does not replace it.
Compare human–human and human–AI agreement correctly
Start with a held-out subset coded independently by at least two trained human coders. Preserve their original decisions before discussion, consensus and expert resolution. Independent coding, dialogue and consensus are established parts of intercoder work (Canadian Medical Education Journal). The results provide a project baseline for assessing AI coding.
For one binary code applied to fixed segments, retain the confusion counts and calculate precision, recall and F1.
| Outcome | Meaning |
|---|---|
| True positive | Both reference decision and AI apply the code |
| False positive | AI applies the code where the reference does not |
| False negative | AI misses a reference-coded segment |
| True negative | Neither applies the code |
Precision asks whether assigned labels have support. Recall shows whether relevant material was missed. F1 balances both. Cohen’s kappa can suit two coders assigning categorical labels to the same fixed units, but only when those assumptions fit the design. For multi-label coding, report precision and recall for each code alongside a set-overlap measure such as Jaccard similarity.
Report exact agreement alongside chance-corrected or set-based measures. Break out rare and high-consequence codes. An aggregate score can conceal a consequential miss.
Review disagreements blind, then diagnose their cause
A metric shows where disagreement occurred. It cannot explain why.
Give an adjudicator the passage, research question, code definition and two anonymised label sets. Keep the source of each set hidden. Ask the adjudicator to choose the better-supported coding, accept both as reasonable, or revise the codebook.
Classify each disagreement by cause:
- Unclear codebook rule or segmentation mismatch
- Missed context or unsupported inference
- Human inconsistency or model fabrication
- Omission of a minority, negative or disconfirming case
- Legitimate alternative interpretation
Aggregate causes by code. Repeated segmentation problems call for a workflow change. Recurring ambiguity suggests a codebook revision. If unsupported inference persists, tighten the prompt, require evidence citations, or remove that code from the AI’s scope.
This test is narrower than checking insights end to end. For that broader task, see validate AI-generated research insights. A documented human–LLM workflow can retain human authority over definitions, category changes and interpretation (arXiv).
Check themes and evidence, not just code labels
Good label agreement can still produce shallow themes. One published comparison found overlap between human and ChatGPT analysis while the AI output showed weaker latent or interpretive insight (Qualitative Health Research).
Verify every quotation against its source. Confirm the speaker, surrounding context, negation and transcript location. Treat an invented or untraceable quotation as a stop condition. One published AI-assisted protocol verifies all quoted evidence and escalates review when sampled answers exceed its failure threshold (Digital Health).
Ask a blinded domain reviewer whether human- and AI-supported themes:
- Answer the research question.
- Are internally coherent and distinct from other themes.
- Have evidence across relevant participants.
- Remain useful without claiming more than the data supports.
Look deliberately for contradictory and low-frequency cases. Repeated excerpts from one participant do not establish broad support. Keep the chain from each theme to its codes, excerpts and transcript locations. Affinity mapping after qualitative coding can make that chain visible.
Use explicit gates for pass, revision and human review
Set gates before inspecting results. The filled matrix below is a hypothetical UX-research policy, not a universal standard. Tighten or relax its thresholds before testing according to the consequences of error.
| Measure | Hypothetical threshold | Observed result | Risk | Action | Owner |
|---|---|---|---|---|---|
| Quotation traceability | 100% | 100% | High | Pass evidence handling | Research lead |
| Critical-code recall | At least 0.90 | 0.92 | High | Permit sampled review | Research lead |
| Human–AI result against human baseline | No more than 0.05 below | 0.03 below | Medium | Pass for this code | Method lead |
| Low-risk code F1 | At least 0.80 | 0.76 | Low | Revise and retest | Analyst |
| Unresolved high-risk disagreements | 0 | 0 | High | Pass | Domain reviewer |
| Participant-group coverage | No systematic omission | One group under-represented | High | Expand holdout and retest | Research lead |
| Fresh holdouts | Two consecutive passes | First pass only | Medium | Do not deploy yet | Method lead |
Pass only codes that meet every applicable gate. Code-level deployment can make the approved scope explicit instead of giving an entire model one pass-or-fail verdict. Keep human review for poorly performing or high-consequence codes, ambiguous latent interpretation and outputs supporting important decisions.
Stop deployment if you find fabricated evidence, privacy violations, an unrepeatable audit trail or systematic omission of a participant group. Design the method around auditability and retained human interpretive authority from the start (University of Cambridge Apollo Repository). Approved outputs can then enter the insight-to-impact workflow with their limitations intact.
Document the decision and monitor for drift
Archive the holdout corpus, original labels, metrics, adjudication log, codebook and prompt versions, model configuration and approved scope. Record limitations, unresolved disagreements and codes that remain human-only. Retest with fresh material after any change to the model, prompt, codebook, participant population or research question.
Copy this worksheet into your research plan:
| Field | Record |
|---|---|
| Analytic approach and AI role | |
| Research question and coding unit | |
| Frozen codebook and prompt versions | |
| Model, settings and preprocessing | |
| Holdout composition and exclusions | |
| Human–human baseline by code | |
| Human–AI metrics by code | |
| Blinded disagreement causes and decisions | |
| Theme and quotation review result | |
| Pass, revise or escalate decision | |
| Approved codes, human-only codes and retest trigger |
Complete the worksheet and decision matrix before deciding which codes the AI may apply, which require human review, and what evidence supports each decision.
Frequently asked questions
What is a good human–AI coding agreement score?
There is no context-free score. Compare human–AI performance with the held-out human–human baseline, inspect every code, and set stricter requirements for consequential codes.
Should Cohen’s kappa be used for AI qualitative coding?
It can suit two coders assigning categorical labels to the same fixed units. Other designs may require per-code precision and recall, Krippendorff’s alpha or multi-label Jaccard similarity. Retain confusion counts and conduct a blinded review of disagreements.
How much qualitative data should humans double-code?
Do not rely on a universal percentage. Choose a holdout large and varied enough to cover participant groups, rare codes, ambiguity and negative cases. Justify its size against the decision and risk involved.
Can AI coding be valid when it disagrees with human researchers?
Yes. Disagreement may reveal human inconsistency, shared bias or a legitimate alternative interpretation. A blinded adjudicator should assess evidential support without knowing which source produced each coding.
When should AI-coded data receive full human review?
Use full human review for high-consequence codes, fabricated or untraceable evidence, failed project gates, systematic group omissions, latent interpretations, and material changes to the model, prompt, codebook or population.
About Glasgow Research — Glasgow Research helps B2B SaaS teams turn customer and market research into product decisions. Work with us.
Author
About Vadim Glazkov
Vadim Glazkov is the founder of Glasgow Research and a product research expert working with founders and B2B SaaS teams on customer interviews, JTBD, market validation, and decision-ready research.