Intercoder Reliability in Qualitative Research Guide

Run a defensible intercoder reliability calibration round, select suitable agreement measures, and know when reflexive analysis needs different quality checks.

Intercoder reliability: what it checks—and what it cannot prove

Intercoder reliability in qualitative research checks whether independent coders apply a shared coding frame to the same, pre-defined units consistently. It is useful when several researchers need classifications that can be applied and explained across a project.

It differs from interrater reliability, which often concerns ratings or scores, and from intracoder consistency, which asks whether one person codes consistently over time. Intercoder reliability assesses the robustness of a coding frame and its application, not the quality of every research decision. O’Connor and Joffe, 2020

Agreement does not validate recruitment, prove themes are correct, or determine a product decision. It gives evidence about whether a codebook is clear enough for its intended use. Treat it as one checkpoint in an insight-to-impact workflow, alongside sound study design, interpretation and decision-making.

Decide first: is agreement the right quality check?

Choose the quality check that fits your analytic approach before selecting a statistic. Intercoder agreement suits a structured or fixed codebook, several coders, and comparable segments that need a shared classification. For example, a product-research team may use it to classify stated navigation barriers or reasons for churn.

Independent coding and calibration test whether that frame works in practice. The aim is not to remove judgement; it is to make the judgement required by each code explicit and usable.

Reflexive thematic analysis has different assumptions. It treats interpretation, researcher subjectivity and code development as part of the analysis, rather than assuming a single correct code for each passage. In that context, imposing a kappa target can conflict with the method. ThematicAnalysis.net recommends method-coherent practices such as reflexive notes, analytic discussion, transparent researcher positioning and an audit trail.

Analysis need	Suitable quality practice	Is ICR a fit?
Fixed codebook used by several coders	Independent double-coding and calibration	Yes
Comparable responses categorised for reporting	Agreement measure plus per-code review	Yes
Reflexive thematic analysis	Reflexive memos and analytic dialogue	Usually no
Early discovery with evolving concepts	Collaborative code development	Not yet

Name your approach and purpose before choosing a measure. If you are following a structured thematic analysis process, state whether agreement is intended to test codebook clarity, support distributed coding or make a classification process communicable to stakeholders.

Set up a calibration round before anyone sees results

Decide the process before results are visible. Record:

Coders, preparation and whether they will work independently.
Codebook version, code definitions and unit of analysis.
Whether a unit can receive more than one code.
Material selected for double-coding and why it represents the wider dataset.
Agreement measure, interpretation approach and response to ambiguity.
Original files, codebook changes and decision records.

Run a small informal pilot to reveal obvious misunderstandings. Then select a justifiable subset for formal independent double-coding. There is no universal double-coding percentage: the appropriate sample depends on the dataset, codebook and consequences of the classification.

Give both coders the same clean, pre-segmented file. Record identities, dates and codebook version. They should code independently before discussing results. Reconciliation may improve the final dataset, but it is not independent agreement and should not replace the original record.

These design choices matter, including choices about coders, material, thresholds, disagreements and the audit trail. American Journal of Qualitative Research discusses these decisions. The same discipline helps when you analyze user interview data.

Use the Calibration Round Pack: a filled example

Copy this fictional pack into a project workspace. The interview excerpt is hypothetical, not client research.

1. Method-fit decision

Question	Filled decision
Analytic approach	Structured descriptive coding
Purpose	Check whether two researchers can apply a shared frame before distributed coding
Is ICR suitable?	Yes: units and categories are pre-defined
Unit of analysis	One pre-segmented interview statement
Multiple codes allowed?	Yes, where each definition is met
Alternative quality check	Reflexive notes, analytic discussion and decision log

2. Six-column codebook worksheet, version 0.1

Code	Definition	Include	Exclude / non-example	Example	Coder notes
Navigation uncertainty	A participant cannot predict where to find or complete a task	Searching or uncertainty about next steps	A slow page where the route is clear	“I was not sure where to change my delivery address.”	Code uncertainty, not the eventual outcome.
Information overload	Too much information makes a task harder	Excess detail, crowded choices or difficulty identifying relevant information	Missing information alone	“There were so many options that I stopped comparing them.”	Apply only where volume or density is stated.

Hypothetical interview excerpt

“I opened the account area to change my address. I kept moving between sections because I could not tell where it belonged. Once I found it, there were lots of delivery options and I was unsure which one applied.”

3. Two-coder coding sheet

Unit ID	Coder A code(s)	Coder B code(s)	Agreement?	Disagreement reason	Revision decision
U01: “could not tell where it belonged”	Navigation uncertainty	Navigation uncertainty	Yes	—	None
U02: “lots of delivery options”	Information overload	Navigation uncertainty	No	Overlapping boundary	Clarify whether option volume can trigger both codes.
U03: “unsure which one applied”	Navigation uncertainty	Information overload	No	Definition too broad	Add contextual rules and permit both codes where conditions are met.

4. Per-code agreement log

Code	What the comparison revealed	Action
Navigation uncertainty	Its boundary with option overload is unclear.	Define when uncertainty is caused by navigation rather than choice volume.
Information overload	Coders interpreted relevance and volume differently.	Require stated excess information or difficulty comparing options.

5. Revise/retest decision record

Field	Filled record
Version frozen	0.1
Independent coding complete	Yes
Discussion finding	Both codes could apply to a statement about many choices and uncertainty.
Revision	Version 0.2 permits both codes; information overload requires stated excess or difficulty comparing options.
Retest plan	Independently code fresh pre-segmented material.
Original results retained	Yes
Final allocation	Use version 0.2 only after the retest is documented.

Freeze version 0.1, code independently, compare by code, discuss the rationale, revise to version 0.2, retest on fresh material, then apply the final frame. Do not silently alter earlier scores after discussion. Vague boundaries, overlapping codes and mismatched segmentation are useful findings because they show what the codebook needs.

After calibration, use affinity mapping in qualitative research to examine relationships across coded material. Keep coding decisions separate from later synthesis decisions.

Calculate and interpret agreement without hiding the hard parts

Percent agreement shows the proportion of coding decisions where coders matched. Cohen’s kappa is a two-coder measure for categorical coding that adjusts observed agreement for agreement expected by chance. PubMed Central explains why chance correction distinguishes kappa from simple agreement.

Conceptually:

kappa = (observed agreement − expected chance agreement) ÷ (1 − expected chance agreement)

Krippendorff’s alpha is another option for multiple coders or different measurement levels. Select a measure because it fits the coding structure, not because it produces a familiar figure.

Report the score alongside the number and roles of coders, unit type and count, available codes, multiple-coding rule, sampled material, codebook version, treatment of missing units, and whether results are pooled or per-code. A pooled score can conceal weak agreement on an ambiguous or consequential code.

There is no universal pass/fail cut-off for qualitative coding reliability. Specify and justify the interpretation approach before reviewing results, and apply more scrutiny where classifications carry greater consequences. This record is also useful when you validate AI-assisted qualitative coding.

Turn disagreement into a better codebook

Classify disagreement before deciding what to change:

Unclear definition or overlapping codes.
Insufficient contextual rule.
Segmentation mismatch.
Coder preparation gap.
Meaningful analytic difference.

Then record a response: clarify, merge or split codes, add a rule, retain a distinction with a rationale, revise the unit rule, retrain coders or reconsider whether reliability fits the analysis. Retest meaningful revisions on new material where feasible. Negotiated consensus may be appropriate for final coding, but it is not a substitute for the original independent agreement result.

Use this methods note:

We used structured descriptive coding with a shared codebook. Two researchers independently coded a defined subset of pre-segmented interview units using codebook version 0.1. We reviewed pooled and per-code agreement, identified ambiguity between two codes, revised their boundaries in version 0.2, and retested the changes on fresh material. Agreement results assess coding-frame application and do not validate wider interpretation or sampling decisions.

Apply the same record-keeping when you validate AI-generated research insights: document what was checked, what changed and what remains uncertain.

A practical reporting checklist

Before scaling coding across a team, confirm that the record includes:

Analytic approach and rationale for intercoder reliability.
Coder roles, preparation and independent coding procedure.
Defined coding units and multiple-coding rules.
Codebook version and double-coded sample selection.
Agreement measure, interpretation approach, and pooled and per-code findings.
Revisions, retesting, final coding allocation and limitations.

Transparency about disagreement is more credible than a decontextualised reliability number. Use the Calibration Round Pack before scaling: decide whether ICR fits, freeze the frame, independently double-code, record disagreement, revise openly and retest where needed.

FAQs

What is intercoder reliability in qualitative research?

It is the degree to which independent coders apply the same coding frame to the same defined data units in the same way. It assesses the coding process, not every aspect of qualitative research quality.

When should I use Cohen’s kappa for qualitative coding?

Use it when two coders independently assign categorical codes to comparable units and chance-corrected agreement serves the project’s stated purpose. The analysis design, coding units and code structure determine whether it fits.

What is an acceptable intercoder reliability score?

No universal cut-off is defensible across all studies. Select and justify an interpretation framework before reviewing results, consider the consequences of the decisions and inspect code-level findings.

Should reflexive thematic analysis use intercoder reliability?

Usually no. Reflexive thematic analysis does not assume one correct coding of fixed meaning. Use method-coherent practices such as reflexive documentation, analytic dialogue, transparent interpretation and a clear record of how themes developed.

About Glasgow Research — Glasgow Research helps B2B SaaS teams turn customer and market research into product decisions. Work with us.