{ "cells": [ { "cell_type": "markdown", "id": "b1934393", "metadata": {}, "source": [ "# Phenotype Profile Matching: Classic Semantic Similarity vs LLM Embeddings\n", "\n", "A core use of semantic similarity is matching a **patient's phenotype profile** to\n", "**disease profiles**, as tools like Exomiser and Phenomizer do. This notebook compares\n", "ontology-based methods with LLM-embedding methods on a small, controlled version of\n", "that task. All methods run through the same OAK interfaces.\n", "\n", "**Task.** We take a pool of 60 OMIM diseases annotated in HPO. For each disease we simulate\n", "three noisy \"patients\":\n", "\n", "- 3 of the disease's phenotypes are sampled;\n", "- each is replaced, with probability 0.8, by an ancestor 1 or 2 levels up (imprecise\n", " annotation);\n", "- 4 confounding phenotypes are added, drawn from the *other* diseases in the pool.\n", "\n", "Every method then ranks all 60 diseases against each patient, and we record the rank of the\n", "true disease. (The noise levels were chosen so the task is neither trivial nor hopeless;\n", "with milder noise every method scores close to perfectly.)\n", "\n", "**Methods**\n", "\n", "| family | method | description |\n", "|---|---|---|\n", "| ontology | Jaccard BMA | best-match average of pairwise ancestor-set Jaccard |\n", "| ontology | Resnik BMA | best-match average of the information content of the most informative common ancestor |\n", "| ontology | flattened closure | Jaccard of the union of all ancestors of the two profiles (one vector per profile) |\n", "| embedding | cosine BMA | best-match average of pairwise cosine, for each OLS model |\n", "| embedding | mean vector | cosine of the averaged term vectors (one vector per profile), for each OLS model |\n", "\n", "The \"flattened\" and \"mean vector\" methods merge each profile into a single vector. That\n", "is what makes them easy to put in an off-the-shelf vector database, so it is worth\n", "knowing how much accuracy that costs." ] }, { "cell_type": "code", "execution_count": 1, "id": "e6861cdf", "metadata": { "execution": { "iopub.execute_input": "2026-10-05T01:30:39.532070Z", "iopub.status.busy": "2026-10-05T01:30:39.531803Z", "iopub.status.idle": "2026-10-05T01:30:41.221412Z", "shell.execute_reply": "2026-10-05T01:30:41.219797Z" } }, "outputs": [], "source": [ "import warnings\n", "warnings.filterwarnings(\"ignore\", category=UserWarning, module=\"eutils\")\n", "warnings.filterwarnings(\"ignore\", category=DeprecationWarning)\n", "\n", "import matplotlib.pyplot as plt\n", "import seaborn as sns\n", "\n", "# reference palette (categorical slots in fixed order; sequential blue ramp)\n", "BLUE, ORANGE, AQUA = \"#2a78d6\", \"#eb6834\", \"#1baf7a\"\n", "INK, INK2, GRID, SURFACE = \"#0b0b0b\", \"#52514e\", \"#e4e3df\", \"#fcfcfb\"\n", "plt.rcParams.update({\n", " \"figure.facecolor\": SURFACE, \"axes.facecolor\": SURFACE, \"savefig.facecolor\": SURFACE,\n", " \"axes.edgecolor\": GRID, \"axes.labelcolor\": INK2, \"axes.titlecolor\": INK,\n", " \"axes.grid\": True, \"grid.color\": GRID, \"grid.linewidth\": 0.6,\n", " \"axes.spines.top\": False, \"axes.spines.right\": False,\n", " \"xtick.color\": INK2, \"ytick.color\": INK2, \"text.color\": INK,\n", " \"font.size\": 10, \"axes.titlesize\": 11, \"figure.dpi\": 110,\n", "})\n", "SEQUENTIAL = sns.blend_palette([\"#f0efec\", \"#86b6ef\", \"#2a78d6\", \"#0d366b\"], as_cmap=True)" ] }, { "cell_type": "code", "execution_count": 2, "id": "8bf66020", "metadata": { "execution": { "iopub.execute_input": "2026-10-05T01:30:41.225299Z", "iopub.status.busy": "2026-10-05T01:30:41.224703Z", "iopub.status.idle": "2026-10-05T01:30:44.259661Z", "shell.execute_reply": "2026-10-05T01:30:44.257917Z" } }, "outputs": [], "source": [ "import random\n", "\n", "import numpy as np\n", "import pandas as pd\n", "\n", "from oaklib import get_adapter\n", "from oaklib.datamodels.vocabulary import IS_A\n", "from oaklib.utilities.embeddings.closure_embeddings import closure_embeddings\n", "from oaklib.utilities.embeddings.vector_utils import cosine_similarity_matrix, jaccard_similarity_matrix\n", "\n", "hp = get_adapter(\"sqlite:obo:hp\")\n", "ols = get_adapter(\"ols:hp\")\n", "MODELS = ols.embedding_models()" ] }, { "cell_type": "markdown", "id": "47d4b1ee", "metadata": {}, "source": [ "## Disease profiles and simulated patients\n", "\n", "Disease annotations come from the HPO `phenotype.hpoa` file. We keep OMIM diseases with\n", "10 to 30 positive phenotype annotations and sample a pool of 60." ] }, { "cell_type": "code", "execution_count": 3, "id": "0779a285", "metadata": { "execution": { "iopub.execute_input": "2026-10-05T01:30:44.263780Z", "iopub.status.busy": "2026-10-05T01:30:44.263144Z", "iopub.status.idle": "2026-10-05T01:30:48.030321Z", "shell.execute_reply": "2026-10-05T01:30:48.028795Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "60 diseases, 180 patients, 930 distinct phenotype terms\n", "Keratosis palmoplantaris striata I\n", " patient: ['Abnormal nail morphology', 'Abnormality of nail color', 'Hyperkeratosis', 'Severe global developmental delay', 'Bronchiectasis', 'Postnatal growth retardation', 'Abnormal pyramidal sign']\n" ] } ], "source": [ "HPOA = \"https://github.com/obophenotype/human-phenotype-ontology/releases/latest/download/phenotype.hpoa\"\n", "hpoa = pd.read_csv(HPOA, sep=\"\\t\", comment=\"#\", dtype=str)\n", "hpoa = hpoa[(hpoa.aspect == \"P\") & hpoa.qualifier.isna() & hpoa.database_id.str.startswith(\"OMIM:\")]\n", "profiles = hpoa.groupby(\"database_id\").hpo_id.apply(lambda x: sorted(set(x)))\n", "disease_names = hpoa.groupby(\"database_id\").disease_name.first()\n", "profiles = profiles[profiles.apply(len).between(10, 30)]\n", "\n", "rng = random.Random(7)\n", "pool = sorted(rng.sample(list(profiles.index), 60))\n", "pool_terms = sorted({t for d in pool for t in profiles[d]})\n", "parents = {}\n", "for s, _, o in hp.relationships(predicates=[IS_A]):\n", " parents.setdefault(s, []).append(o)\n", "\n", "def generalize(t, levels):\n", " for _ in range(levels):\n", " if not parents.get(t):\n", " break\n", " t = rng.choice(parents[t])\n", " return t\n", "\n", "def noisy_patient(disease, n=3, p_imprecise=0.8, n_confounders=4):\n", " q = [generalize(t, rng.choice([1, 2])) if rng.random() < p_imprecise else t\n", " for t in rng.sample(profiles[disease], n)]\n", " others = [t for t in pool_terms if t not in profiles[disease]]\n", " return q + rng.sample(others, n_confounders)\n", "\n", "patients = [(d, noisy_patient(d)) for d in pool for _ in range(3)]\n", "all_terms = sorted(set(pool_terms) | {t for _, q in patients for t in q})\n", "print(f\"{len(pool)} diseases, {len(patients)} patients, {len(all_terms)} distinct phenotype terms\")\n", "example, example_patient = patients[0]\n", "print(disease_names[example])\n", "print(\" patient:\", [hp.label(t) for t in example_patient])" ] }, { "cell_type": "markdown", "id": "15f78a7c", "metadata": {}, "source": [ "## The OAK API, for one patient and one disease\n", "\n", "The interface returns a full best-match breakdown. Below we compute the same quantities\n", "in bulk with numpy, so all methods can be scored quickly." ] }, { "cell_type": "code", "execution_count": 4, "id": "03ea5c02", "metadata": { "execution": { "iopub.execute_input": "2026-10-05T01:30:48.034333Z", "iopub.status.busy": "2026-10-05T01:30:48.034041Z", "iopub.status.idle": "2026-10-05T01:31:00.474439Z", "shell.execute_reply": "2026-10-05T01:31:00.472420Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "best match average: 0.652\n", " Abnormal nail morphology -> Nail dystrophy (0.86)\n", " Abnormality of nail color -> Yellow nails (0.79)\n", " Hyperkeratosis -> Orthokeratotic hyperkeratosis (0.84)\n", " Severe global developmental delay -> Nail dystrophy (0.30)\n", " Bronchiectasis -> Hyperhidrosis (0.35)\n", " Postnatal growth retardation -> Nail dystrophy (0.36)\n", " Abnormal pyramidal sign -> Streaks of hyperkeratosis along each finger onto the palm (0.47)\n" ] } ], "source": [ "sim = ols.embedding_termset_similarity(\n", " example_patient, profiles[example], model=\"text-embedding-3-large_pca512\", labels=True\n", ")\n", "print(\"best match average:\", round(sim.average_score, 3))\n", "for bm in sim.subject_best_matches.values():\n", " print(f\" {bm.match_source_label} -> {bm.match_target_label} ({bm.score:.2f})\")" ] }, { "cell_type": "markdown", "id": "5701581d", "metadata": {}, "source": [ "## Vectors and scores\n", "\n", "Closure vectors for all terms are built in one call, so they share a vocabulary.\n", "Information content (IC) for Resnik is computed from the ontology structure, so each\n", "dimension of the closure vector can carry its ancestor's IC." ] }, { "cell_type": "code", "execution_count": 5, "id": "6dc6ed1a", "metadata": { "execution": { "iopub.execute_input": "2026-10-05T01:31:00.477246Z", "iopub.status.busy": "2026-10-05T01:31:00.476979Z", "iopub.status.idle": "2026-10-05T01:31:01.720068Z", "shell.execute_reply": "2026-10-05T01:31:01.718701Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'harrier-oss-v1-27b_pca512': 930, 'llama-embed-nemotron-8b_pca512': 930, 'text-embedding-3-large_pca512': 930, 'text-embedding-3-small_pca512': 930}\n" ] } ], "source": [ "ids, vocab, C = closure_embeddings(hp, all_terms)\n", "cix = {t: i for i, t in enumerate(ids)}\n", "ic = dict(hp.information_content_scores(vocab, object_closure_predicates=[IS_A]))\n", "IC = np.array([ic.get(v, 0.0) for v in vocab], dtype=np.float32)\n", "\n", "vectors = {}\n", "for model in MODELS:\n", " mids, M = ols.entity_embeddings(all_terms, model=model)\n", " vectors[model] = dict(zip(mids, M))\n", "print({m: len(v) for m, v in vectors.items()})" ] }, { "cell_type": "code", "execution_count": 6, "id": "2bbf3fa5", "metadata": { "execution": { "iopub.execute_input": "2026-10-05T01:31:01.723347Z", "iopub.status.busy": "2026-10-05T01:31:01.723043Z", "iopub.status.idle": "2026-10-05T01:31:20.966559Z", "shell.execute_reply": "2026-10-05T01:31:20.964582Z" } }, "outputs": [ { "data": { "text/plain": [ "118800" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def bma(S):\n", " # symmetric best-match average over a (patient x disease) similarity matrix\n", " return 0.5 * (S.max(axis=1).mean() + S.max(axis=0).mean())\n", "\n", "def resnik_matrix(a, b):\n", " # IC of the most informative common ancestor, for every pair of terms\n", " A, B = C[[cix[t] for t in a]], C[[cix[t] for t in b]]\n", " return (A[:, None, :] * B[None, :, :] * IC).max(axis=2)\n", "\n", "def score_all(patient, disease):\n", " p_rows, d_rows = C[[cix[t] for t in patient]], C[[cix[t] for t in disease]]\n", " scores = {\n", " (\"ontology\", \"Jaccard BMA\"): bma(jaccard_similarity_matrix(p_rows, d_rows)),\n", " (\"ontology\", \"Resnik BMA\"): bma(resnik_matrix(patient, disease)),\n", " (\"ontology\", \"flattened closure\"): jaccard_similarity_matrix(\n", " p_rows.max(axis=0, keepdims=True), d_rows.max(axis=0, keepdims=True))[0, 0],\n", " }\n", " for model in MODELS:\n", " v = vectors[model]\n", " P = np.array([v[t] for t in patient if t in v])\n", " D = np.array([v[t] for t in disease if t in v])\n", " name = model.replace(\"_pca512\", \"\")\n", " scores[(\"embedding: cosine BMA\", name)] = bma(cosine_similarity_matrix(P, D))\n", " scores[(\"embedding: mean vector\", name)] = cosine_similarity_matrix(\n", " P.mean(axis=0, keepdims=True), D.mean(axis=0, keepdims=True))[0, 0]\n", " return scores\n", "\n", "rows = []\n", "for i, (true_disease, patient) in enumerate(patients):\n", " for candidate in pool:\n", " for (family, method), score in score_all(patient, profiles[candidate]).items():\n", " rows.append((i, true_disease, candidate, family, method, score))\n", "scores = pd.DataFrame(rows, columns=[\"patient\", \"disease\", \"candidate\", \"family\", \"method\", \"score\"])\n", "len(scores)" ] }, { "cell_type": "markdown", "id": "6605935f", "metadata": {}, "source": [ "## Results\n", "\n", "For each of the 180 patients and each method, we find the rank of the true disease among\n", "the 60 candidates (1 is best; ties share the average rank)." ] }, { "cell_type": "code", "execution_count": 7, "id": "79246da2", "metadata": { "execution": { "iopub.execute_input": "2026-10-05T01:31:20.971621Z", "iopub.status.busy": "2026-10-05T01:31:20.971225Z", "iopub.status.idle": "2026-10-05T01:31:21.818242Z", "shell.execute_reply": "2026-10-05T01:31:21.816790Z" } }, "outputs": [ { "data": { "text/html": [ "
| \n", " | \n", " | MRR | \n", "hits_at_1 | \n", "hits_at_5 | \n", "median_rank | \n", "MRR 95% CI low | \n", "MRR 95% CI high | \n", "
|---|---|---|---|---|---|---|---|
| family | \n", "method | \n", "\n", " | \n", " | \n", " | \n", " | \n", " | \n", " |
| ontology | \n", "Jaccard BMA | \n", "0.673 | \n", "0.500 | \n", "0.900 | \n", "1.5 | \n", "0.623 | \n", "0.719 | \n", "
| Resnik BMA | \n", "0.564 | \n", "0.389 | \n", "0.817 | \n", "2.0 | \n", "0.510 | \n", "0.615 | \n", "|
| embedding: cosine BMA | \n", "text-embedding-3-large | \n", "0.536 | \n", "0.372 | \n", "0.756 | \n", "2.0 | \n", "0.480 | \n", "0.589 | \n", "
| harrier-oss-v1-27b | \n", "0.509 | \n", "0.317 | \n", "0.783 | \n", "2.0 | \n", "0.453 | \n", "0.560 | \n", "|
| text-embedding-3-small | \n", "0.473 | \n", "0.294 | \n", "0.711 | \n", "3.0 | \n", "0.419 | \n", "0.526 | \n", "|
| ontology | \n", "flattened closure | \n", "0.456 | \n", "0.289 | \n", "0.661 | \n", "3.0 | \n", "0.403 | \n", "0.512 | \n", "
| embedding: mean vector | \n", "text-embedding-3-large | \n", "0.399 | \n", "0.228 | \n", "0.611 | \n", "4.0 | \n", "0.346 | \n", "0.452 | \n", "
| embedding: cosine BMA | \n", "llama-embed-nemotron-8b | \n", "0.380 | \n", "0.222 | \n", "0.556 | \n", "4.5 | \n", "0.330 | \n", "0.431 | \n", "
| embedding: mean vector | \n", "harrier-oss-v1-27b | \n", "0.367 | \n", "0.183 | \n", "0.606 | \n", "4.0 | \n", "0.317 | \n", "0.417 | \n", "
| llama-embed-nemotron-8b | \n", "0.309 | \n", "0.167 | \n", "0.456 | \n", "7.0 | \n", "0.260 | \n", "0.357 | \n", "|
| text-embedding-3-small | \n", "0.301 | \n", "0.117 | \n", "0.528 | \n", "5.0 | \n", "0.256 | \n", "0.346 | \n", "