A scan on its own misses context a clinician has on paper — age, mass shape, breast density. I wanted to know whether a model that reads both the image and the clinical record makes a more reliable call than either one alone.
How I built it
01
Imaging stream
MedCLIP (ResNet50 backbone) turns each scan into a feature vector.
02
Clinical stream
TabNet learns from the structured record, using attention to pick the clinical features that matter.
03
Fusion
Both feature vectors are concatenated into one representation; a classification head outputs the diagnosis probability.
04
Serving
An async FastAPI service in a multi-stage Docker build, with a small clinician dashboard and saved scalers and weights.
05
Testing
Frontend tests (Vitest) and backend API contract tests (pytest) run in GitHub Actions on every push.
// two inputs, one diagnosis
feature_fusion — live
[ DECISION LOG ]
What went wrong, and what I decided.
// 3 decisions
D1
Fine-tuning all of MedCLIP overfit almost at once.
what happened
Validation loss climbed after the first epochs — a few hundred patients can’t re-teach a foundation model.
decision
Kept the pretrained encoder frozen and trained only the heads.
result
Stable training and better generalisation.
D2
Accuracy hid a useless model.
what happened
The data is imbalanced, so a model that called almost everything “benign” still scored well.
decision
Weighted the loss and chose models by ROC-AUC, reporting both.
result
Model selection that reflects missed cancers, not just the majority class.
D3
XGBoost beat TabNet on the clinical data alone.
what happened
But trees don’t produce a feature vector that can be fused with the image features.
decision
Used TabNet for the clinical stream so both streams could be concatenated.
result
One fused model that outperformed the single-modality baselines.