Multimodal Breast Cancer Detection
Early fusion of MedCLIP mammogram embeddings with TabNet clinical record features — beating single-modality baselines on real, imbalanced clinical data.
ML Engineer & Data Scientist
I build machine-learning systems for messy, real-world data — and I document what didn't work.
class CancerFusionModel(nn.Module):
"""Mammogram + clinical record → diagnosis."""
def __init__(self):
super().__init__()
self.img = MedCLIP(frozen=True)
self.tab = TabNet(clinical_dim=23)
self.head = nn.Linear(512 + 64, 2)
def forward(self, x_img, x_tab):
z = torch.cat([self.img(x_img),
self.tab(x_tab)], dim=1)
return self.head(z) # early fusion
Early fusion of MedCLIP mammogram embeddings with TabNet clinical record features — beating single-modality baselines on real, imbalanced clinical data.
Resilient nationwide polling unit extraction pipeline spanning all 36 states, validated with 99.9% consistency against official records.
Comparative research evaluating Vision Transformer (ViT) against DINOv2 self-supervised foundation representations with Grad-CAM interpretability inspection.
Averaging per-modality logit probabilities underperformed early fusion by ~4 AUC points because image and clinical modalities could not interact during representation learning.
A small medical dataset quickly ruined pretrained representation spaces when fine-tuned end-to-end. Freezing the MedCLIP encoder produced stable, superior generalization.
Initial models predicted the majority class ('benign') for almost all cases, showing high accuracy but zero clinical utility. Shifted model selection strictly to ROC-AUC and weighted loss.