Dense Retrieval Versus Lexical Retrieval for Medical Entity Linking

Medical entity linking maps a mention in context to an identifier in a terminology such as RxNorm or ICD. The first hard problem is candidate generation: among thousands or millions of concept names and synonyms, retrieve a small set that still contains the correct concept.

Lexical and dense retrievers fail differently. Lexical search is excellent when a noisy mention shares a rare token with the terminology. Dense retrieval can bridge paraphrases and abbreviations, but may return semantically plausible concepts that miss a decisive qualifier such as route, dosage form, laterality, or negation. A reliable linker treats their disagreement as information.

Formalizing candidate retrieval#

Let $m$ be a mention, $c$ its surrounding context, and $\mathcal{E}$ the catalogue of concept entries. Candidate generation returns

The immediate objective is recall at $k$:

A reranker cannot recover the gold concept if candidate generation drops it. Candidate recall is therefore measured before end-to-end accuracy, preferably at several budgets such as 10, 50, and 100.

What lexical retrieval preserves#

A BM25-style index represents each concept by text fields: canonical name, synonyms, code-system labels, and sometimes curated abbreviations. For query terms $q$, a simplified BM25 contribution is

The score rewards exact term overlap, especially for rare terms, while controlling for repeated terms and document length.

For medical catalogues, this behaviour has valuable properties:

• rare drug stems, units, and anatomical terms remain decisive;
• results can be explained by matching tokens;
• new catalogue entries are searchable without model retraining;
• field boosts can preserve the authority of canonical names over broad synonyms;
• CPU indexing and querying are mature and inexpensive.

Its weakness is equally clear. “High blood pressure” and “hypertension” share no content word. Vietnamese clinical shorthand may not overlap an English terminology label. Tokenization can separate codes, units, and punctuation in harmful ways.

What dense retrieval learns#

A dual encoder maps the mention-context query and each catalogue entry into a shared vector space:

With normalized vectors, retrieval uses cosine similarity:

Catalogue vectors are computed offline and stored in an approximate nearest neighbour index. The query encoder runs online.

Dense retrieval can connect synonyms, translations, and contextual paraphrases without explicit overlap. Context can disambiguate a short mention: “cold” near symptom language differs from “cold” in a storage instruction. Fine-tuning on mention–concept pairs can adapt the geometry to a target corpus.

However, embedding similarity is a compressed signal. Concepts in the same semantic neighbourhood can be dangerously close even when they differ in an attribute essential to coding. Training data may also teach popularity priors: frequent concepts crowd out a rare but lexically exact answer.

Failure modes are complementary#

These are hypotheses to verify on the target data, not universal outcomes. A model pretrained primarily on English prose may behave differently on Vietnamese notes and bilingual terminology entries.

Hybrid candidate generation#

The simplest useful hybrid takes the union of top candidates from both systems, then reranks them. If the budget is fixed, combine ranks rather than raw scores: BM25 and cosine values have unrelated scales and drift across queries.

Reciprocal rank fusion assigns

where absent entries contribute zero and $\kappa$ dampens the advantage of the first few ranks.

lexical ← BM25.search(normalize(mention), top_k = 100)
dense   ← ANN.search(encode(mention, context), top_k = 100)
 
for each concept in union(lexical, dense):
    score[concept] ← 0
    if concept in lexical:
        score[concept] += 1 / (κ + rank_lexical[concept])
    if concept in dense:
        score[concept] += 1 / (κ + rank_dense[concept])
 
return top 50 concepts by score, with both source ranks attached

Keep provenance—the source rank and score from each retriever—as reranker features and debugging fields. A cross-encoder can then read the mention context alongside each candidate name and synonyms, while exact-match flags preserve lexical evidence.

Index design is an experiment#

A concept is rarely represented by one string. Possible fields include:

• preferred name;
• synonyms and abbreviations;
• parent terminology labels;
• semantic type;
• definition;
• code and source vocabulary.

Blindly concatenating every field can hurt both systems. Lexical queries may match generic definition words. Dense embeddings may be dominated by a long definition and lose the short canonical name.

For lexical retrieval, index separate fields with explicit boosts. For dense retrieval, test one vector per synonym versus one vector per concept. Synonym-level vectors increase index size but prevent an uncommon alias from being averaged away. Deduplicate concept identifiers after retrieval.

Evaluation that reveals the bottleneck#

Report more than end-to-end top-1 accuracy:

Candidate Recall@k

gold ∈ top-k

Upper bound for reranking

MRR

mean 1 / rank

Rewards early retrieval

Abstention

risk–coverage

For unsupported mentions

Slice these metrics by:

• exact normalized synonym match versus no match;
• seen versus unseen mention surface;
• abbreviation and misspelling;
• language and code system;
• concept frequency;
• mention length;
• whether context is needed for disambiguation.

Evaluate the union oracle as well: how often does either system retrieve the gold concept? If union recall barely exceeds the better single system, fusion adds latency without complementary evidence. If it rises substantially, reranking is the next bottleneck.

A practical development order#

1. Build a normalized exact/synonym matcher as a transparent baseline.
2. Add a BM25 index with field-aware analysis and inspect missed gold synonyms.
3. Establish Recall@k and slice reports before training a dense model.
4. Add a pretrained dense retriever and log lexical/dense disagreement.
5. Fine-tune with hard negatives that share semantic type or lexical tokens.
6. Fuse candidates by rank and train a cross-encoder reranker.
7. Calibrate an abstention decision for mentions outside the terminology.

Hard negatives matter. Random concepts are too easy: the model learns broad semantic type while ignoring attributes. Useful negatives include sibling concepts, same drug with a different form, overlapping lexical names, and high-ranked mistakes from the current retriever.

Choosing a system#

Use lexical retrieval alone when terminology coverage is strong, mentions are clean, latency is strict, and explanations matter. Dense retrieval becomes valuable when paraphrase, multilingual text, and context-driven ambiguity dominate.

For noisy clinical entity linking, a hybrid is often the most defensible starting point—not because two models are automatically better, but because exact lexical evidence and learned semantic evidence make different, inspectable mistakes. The right candidate generator is the one whose complementarity survives slice-level evaluation on the actual notes.

References#