Harnessing the Power of Foundation Models for Healthcare and Life Sciences

Videos
ML+X
Healthcare
Medical imaging
Foundation models
Multimodal learning
Deep learning
Embeddings
Image data
Computer vision
Retrieval
Zero-shot learning
LLM
Microsoft Copilot
Azure
Presenter

Jameson Merkow

Date

April 1, 2025

About this resource

Jameson Merkow, AI engineer at Microsoft, presents an overview of recent advances in applying foundation models to healthcare and life sciences. This forum explores the use of multimodal embedding models, large language models (LLMs), and Azure-integrated tools to support clinical workflows and biomedical research.

Microsoft’s healthcare AI offerings

Jameson introduces a suite of tools available through Microsoft’s AI Foundry and Azure ML:

  • MedImage Insight (MI2): A powerful multimodal embedding model trained across nine medical imaging modalities.
  • MedImage Parse: A segmentation model that responds to natural language prompts across modalities.
  • CXR ReportGen: An image-to-text model that outputs structured radiology findings and highlights supporting regions in images.

All tools are open source and deployable within a secure, tenant-controlled Azure environment.

Applications and use cases

Jameson covers a range of use cases, including:

  • Image classification using pretrained embeddings or lightweight adapters.
  • Outlier detection for clinical trial QC and data curation.
  • Survival prediction from paired MRI and histopathology data using multimodal fusion.
  • Image search across 2D, 3D, and cross-modality benchmarks.
  • Integration with multi-agent systems for orchestrating model calls and generating clinical reports.
  • Drift monitoring using multimodal embeddings and patient metadata.

Why this matters

Rather than retraining LLMs for each medical task, Microsoft’s approach combines task-specific vision/language models with general-purpose reasoning agents. This modular strategy enables faster deployment, clearer interpretability, and easier fine-tuning across domains.

See also