Rebuilding Whel

A follow-up to Building Whel and An Update on Whel. If you’re new here: Whel is a drug-repurposing research tool that mines scattered evidence (trials, registries, pharmacovigilance data, target databases, patient forums) for signals that existing drugs might help under-researched hormonal conditions affecting women.

A note before I start – most of what follows is about taking things out. The last few weeks of work on Whel have included a lot of rebuilding the machinery/engine that scores data. Oops. This period of “undoing” was also, I think, the most useful stretch of work I’ve done on this project. So this is the post where I explain why the model is now a different one than it was earlier in June.

As always, I work on this nearly every day, the numbers move, and the site is the real source of truth. The Roadmap tracks what’s live versus planned, and the Methodology changelog now carries a dated entry for every version of the engine, including the big one I detail below.

The numbers!

In the last two posts I said Whel carried “271 active repurposing signals.” As of this writing, the headline number is 183 drug-condition pairs across the same six conditions (endometriosis, PMDD, PCOS, adenomyosis, vulvodynia, and perimenopause & menopause), drawn from 433 verbatim evidence claims across 186 source documents, covering 171 distinct compounds.

Let’s start with the biggest change: 183 is smaller than 271. I will address this head-on. The database did not shrink. What changed is the unit. The old “signal” counted each arm-reading separately, so a single drug-condition relationship could show up as three or four “signals” if it appeared in three or four evidence arms. The new engine treats the drug-condition pair as the atom, and scores all of its evidence arms into one record. So a number that used to double- and triple-count now counts once. On top of that, the whole corpus was re-extracted and re-scored during the migration, so 183 and 271 aren’t really the same kind of thing and shouldn’t be read as “went down.” I’d rather have a number that means something precise than a bigger number that means something fuzzy.

The distribution across conditions, for the curious: Perimenopause & Menopause (50), PCOS (37), Endometriosis (33), Vulvodynia (30), PMDD (18), Adenomyosis (15).

From “signals” to a substrate

The most central change is that Whel is no longer a flat list of signals with a grade stapled to each one. It’s now what I’ve been calling a substrate, a single scored layer that reads every drug-condition pair through three evidence arms and resolves them into one verdict.

If you read the last post, you’ll remember four research arms: Direct Research, Cross-Condition Signals, Pathway Insights, and Community Forum Reports. There are now three: Direct, Pathway, and Community. Cross-condition didn’t disappear, it got demoted. It used to be a scored arm in its own right, which always sat a little awkwardly, because “this drug helped a different condition” isn’t really independent evidence about your condition; it’s a hypothesis generator. So cross-condition is now a derived lens layered on top of the scored arms, not one of the things being scored.

Every arm is scored on the same five dimensions, each 0-2, summing to an arm strength out of 10. The dimensions got renamed and sharpened in the process: what used to be replication, source quality, biological plausibility, and direction consistency are now corroboration, rigor, plausibility, and consistency (specificity kept its name). That is, I think, more than simply relabeling. Corroboration in particular absorbed a job that used to live somewhere much more problematic, which is the next section.

The arm strength then gets discounted by a female-applicability multiplier, a six-band scale (F1 through F6, ×1.00 down to ×0.50) that asks how far the underlying evidence was actually generated in women versus extrapolated from male-derived or mixed populations. A signal built on a male-only trial and pointed at a female condition gets marked down, visibly, with the reasoning shown. That discount is the part of Whel I care about most (and that I am most happy is finally live!!), because it’s the entire reason I started this project in the first place. The result sorts into one of four frozen tiers (Strong (≥8.0), Moderate (6.0–7.9), Emerging (3.5–5.9), Exploratory (<3.5)) which currently breaks down as 11 Strong, 60 Moderate, 68 Emerging, and 44 Exploratory.

There’s also a new, separate validation status on each pair: clinical, unvalidated signal, or preliminary, that captures whether a pair is anchored by real direct research or is being surfaced more tentatively. Right now that’s 108 clinical, 43 unvalidated signals, and 32 preliminary. The full model is laid out on the signal types & scoring page.

Killing a circular grade

Another major chage. In the last post I introduced (with some pride) an L0-L3 external validation grade, which was a four-step ladder meant to show whether a signal was independently supported in the broader clinical record, sitting beside Whel’s internal confidence tier.

An outside reviewer pointed out, correctly, that the L-grade was doing two contradictory jobs at once. It was being used as an input into how Whel scored a signal internally, and as an external benchmark of that same signal. Which means the thing was grading its own homework. A score that partly determines itself and then validates itself is circular, and circularity is exactly the failure mode an evidence tool is supposed to avoid.

So I retired the L0-L3 grade entirely. In its place:

The in-pipeline job, “how well-corroborated is this within the evidence Whel actually ingested,” now lives inside the five-dimension rubric as the corroboration dimension, scored only against sources the engine has actually read. The external job, “is this independently supported out in the wider clinical record,” became a clean external ladder (E0-E3) that is computed strictly after the fact and never feeds back into the score. Same information, but the two roles are now physically separated so neither can contaminate the other.

I did the same kind of separation for the two external comparators. The Open Targets knowledge-graph check, which used to float beside the score as its own validation item, got absorbed into the Pathway arm where it belongs as an input. And MATRIX (which, if you recall, is Every Cure’s public drug-disease prediction layer, that I wrote about last time) is now positioned explicitly as an independent external comparator shown next to the score and never folded into it. The rule I settled on, and which I now apply everywhere: anything that validates the score is not allowed to also be the score. The reasoning for all of this is written up in the methodology.

The scoring model is now Opus 4.8

Whel uses a large language model to do one specific job: read sources and extract structured facts, then score those facts against the fixed rubric. It does not generate clinical advice, doses, or treatment plans, and the deterministic parts of the pipeline (the female-applicability multiplier, the imprecision caps, the final tier assignment) are computed in code, not by the model.

Since the last post I upgraded that model from Claude Opus 4.6 to Claude Opus 4.8 (released May 2026). In the last post I leaned pretty heavily on WHBench, the women’s-health clinical benchmark where Opus 4.6 was the strongest of 22 models at 72.1%. No women’s-health-specific benchmark has evaluated 4.8 yet, so I can’t claim an improvement there. What I can point to is that Opus 4.8 scored 55.8% on HealthBench Professional (a set of physician-authored clinical tasks), up from a prior release, and that no frontier model is near the ceiling on these tasks. There’s also recent work in Nature Medicine finding that general-purpose frontier models now outperform specialized clinical tools on medical reasoning, which is part of why I’m comfortable building on a general model rather than waiting for a women’s-health-specific one that doesn’t exist.

I read those results the way I’d want a reader t0, as evidence the model handles clinical text well, and as nothing remotely like a guarantee that any individual extraction is correct. The reference-fabrication work I cited last time still applies in spirit, since confident, plausible-looking, wrong structured output is the failure mode I’m designing against. Which is the whole reason the score is computed in code and the model is fenced into the narrow job of reading.

Why I made these choices

Almost every change above trades a more flattering surface for a more defensible one — fewer counted signals but a cleaner unit; one fewer arm but a better account of what cross-condition data is; a deleted grade but no more circular self-validation; a model upgrade paired with a more careful statement of its limits.

Where this is and what’s next

If you want to poke at it, the signal types & scoring page documents the three arms and five dimensions; the methodology page carries the external E0-E3 ladder and the full update log; the external references page has the MATRIX coverage audit; and the technical architecture page explains which steps are the model’s and which are deterministic code.

Still on the roadmap (a lot): the two-rater validation study; disproportionality statistics (PRR/ROR) on the adverse-event data; the manual primary-source curation pass for well-studied pairs where the automated pipeline returns representative but not exhaustive sources; ontology-grounded entity resolution; a knowledge-graph grounding layer built with BioCypher; and a citable open-data export with a DOI. The grounding work is the through-line for the next few weeks, same instinct as this rebuild.

Thanks as always! The site is whel.bio, and my messages are open. The fastest way to make this better is to tell me where I am making a fool of myself.