Practical Error Analysis for Biomedical Named Entity Recognition

An entity recognizer can gain half an F1 point while becoming worse on the cases that matter to a downstream linker. The aggregate number is useful for comparison, but it cannot tell us whether errors come from tokenization, missing clinical abbreviations, inconsistent boundaries, label confusion, or annotation noise.

Error analysis is the bridge between an evaluation result and the next experiment. The goal is not to collect colourful examples. It is to partition errors so that each large bucket suggests a different intervention.

Start with the evaluation contract#

For exact-match span evaluation, a predicted entity is correct only when its start offset, end offset, and type equal a gold entity:

With the resulting counts,

Before inspecting examples, write down the contract:

• Are spans represented by characters, bytes, or token indexes?
• Are end offsets inclusive or exclusive?
• Is evaluation strict, overlap-based, or both?
• Are discontinuous or nested entities supported?
• Does Unicode normalization happen before offsets are created?
• Is micro averaging used across entity types?

An offset conversion bug can look exactly like a boundary weakness. Reconstruct the substring from every evaluated span and assert that it equals the stored entity text.

Pair spans before classifying errors#

Construct a bipartite graph between unmatched predictions and unmatched gold entities. Connect spans that overlap. A simple deterministic pairing strategy is:

1. prefer pairs with the greatest intersection-over-union;
2. break ties with matching entity type;
3. leave spans with no overlap unpaired.

For character intervals $a$ and $b$,

This pairing does not change the official metric. It only provides a stable unit for diagnosis. A maximum-weight bipartite matching is useful when nested entities make greedy pairing ambiguous; for flat clinical NER, sorted greedy matching is often sufficient and easier to audit.

Use a mutually intelligible taxonomy#

The first pass should classify each error by its visible geometry:

Then add orthogonal attributes rather than creating hundreds of composite labels:

• entity type and document section;
• mention length in tokens;
• seen or unseen normalized surface form;
• abbreviation, misspelling, number, or negation cue;
• sentence length and distance to relevant context;
• model confidence and margin between the top labels;
• annotator or source corpus, when those fields are available.

This structure lets us ask useful questions such as “Are right-boundary errors concentrated in medication mentions containing dose and route?” without hard-coding that scenario into the primary taxonomy.

Build an auditable error record#

Do not reduce the analysis to a notebook cell that prints nearby text. Store a serializable record that can be sorted, sampled, and diffed between model versions.

type Span = {
  start: number;
  end: number; // exclusive
  label: string;
  text: string;
};
 
type Geometry =
  | "exact"
  | "left-boundary"
  | "right-boundary"
  | "containment"
  | "crossing"
  | "spurious"
  | "missed";
 
type ErrorRecord = {
  documentId: string;
  prediction?: Span;
  gold?: Span;
  geometry: Geometry;
  typeCorrect: boolean | null;
  context: string;
  confidence?: number;
  attributes: string[];
};
 
function overlaps(a: Span, b: Span): boolean {
  return Math.max(a.start, b.start) < Math.min(a.end, b.end);
}

Include document identifiers and offsets, but handle clinical text according to the data-use agreement. An error dashboard should not become an uncontrolled copy of sensitive notes.

Slice by mechanism, not only by label#

Per-type recall reveals that MEDICATION is weaker than PROBLEM, but it still mixes several mechanisms. More diagnostic slices include:

• lexical novelty: mention surface form never occurred in training;
• context dependence: abbreviation requires its section or surrounding tokens;
• composition: multi-token mention contains a head plus modifiers;
• format variation: unit, punctuation, casing, or OCR noise;
• label interaction: two adjacent entities create an illegal or unlikely BIO transition;
• guideline ambiguity: equally plausible spans receive different annotations.

Always report each slice’s support beside its metric. A 30-point recall drop on three entities is an investigation prompt, not a reliable population estimate.

Inspect the pipeline boundaries#

In a biomedical system, the recognizer rarely stands alone. A predicted span may be passed to assertion detection, normalization, and entity linking. Evaluate at those interfaces.

Tokenization and offsets#

Compare character spans before subword alignment and after reconstruction. Hyphens, Vietnamese combining marks, units such as mg/mL, and de-identified placeholders deserve explicit tests. If the gold label begins inside a tokenizer token, document the alignment policy rather than silently rounding.

CRF constraints#

Count illegal BIO sequences before any repair step. If a CRF eliminates illegal transitions but boundary errors persist, the issue is probably in emissions or annotation rather than structural decoding. Inspect transition scores only after confirming their row/column convention.

Entity linking#

Measure whether the gold concept is recoverable from the gold span and from the predicted span. The difference separates retrieval weakness from upstream boundary damage. A linker tolerant of modifiers may make some strict NER errors operationally harmless; an exact lexical linker may amplify them.

Compare models on the same entities#

Two aggregate reports hide whether model B truly fixes model A. Join predictions by document and gold entity, then build four sets:

1. correct for both models;
2. fixed by the new model;
3. regressed by the new model;
4. wrong for both.

Read comparable samples from the fixed and regressed sets. If improvements mainly come from frequent short mentions while regressions hit long rare mentions, the same F1 delta has a different interpretation than a uniform improvement.

A useful review sheet includes the sentence, gold span, both predictions, both confidences, taxonomy fields, and a short analyst note. Blind the model names during manual review when practical; knowledge of which system is “new” invites motivated explanations.

Turn buckets into experiments#

An error bucket is valuable when it changes a decision:

Avoid changing the model, decoder, augmentation, and annotation rules in one step. The taxonomy should make a small causal experiment possible.

A compact review protocol#

For each model checkpoint:

1. freeze the evaluation script and dataset version;
2. compute strict metrics and per-type supports;
3. pair overlapping unmatched spans;
4. assign geometry automatically;
5. stratify by type, novelty, length, and section;
6. manually review balanced samples from the largest and most costly buckets;
7. compare fixes and regressions against the previous model;
8. write one hypothesis, one intervention, and one expected slice-level change.

The result is not merely a better report. It is a feedback system connecting data, model structure, decoding, and downstream behavior—exactly the information a single F1 score necessarily discards.

References#