Sergio Sánchez Santiesteban

Recent advancements in artificial intelligence have led to the development of multimodal AI

systems capable of integrating diverse types of data such as images, text, and structured knowledge.

These systems offer significant potential across various domains, particularly in medicine,

where complex and heterogeneous data types—ranging from histological images to genomic

sequences and radiology reports—are critical for accurate diagnosis, prognosis, and treatment

planning. However, current multimodal models face challenges in scalability, knowledge integration,

and effective cross-modal alignment, particularly when applied to the intricacies of

medical data. This thesis addresses these challenges by proposing novel methodologies for enhancing

multimodal AI models through knowledge integration, self-supervised learning, and

fine-grained alignment techniques.

First, we introduce a knowledge graph-augmented multimodal model that enables efficient and

scalable access to external knowledge, reducing the need for ever-larger model architectures.

This approach is demonstrated to outperform state-of-the-art models on vision-language tasks

using a fraction of the training data and parameters.

Next, we adapt Self-Supervised Learning (SSL) techniques to the medical domain, specifically

for the integration of histological images and genomic data in cancer prognosis. The Selfsupervised

Histology-Genomic (SHG) model is introduced, leveraging specialized SSL tasks

designed to capture complex relationships between phenotypic and genomic information. Empirical

evaluations on multiple cancer datasets from The Cancer Genome Atlas (TCGA) show

that the SHG model significantly improves survival prediction across five cancer types.

Finally, we focus on enhancing radiology report generation by developing a framework that improves

the alignment between medical images and their corresponding textual reports. Our proposed

methods introduce a region-specific Retrieval Augmented Generation (RAG) approach

to enhance the generation of clinical reports by incorporating relevant retrieved information.

Additionally, we integrate locally aligned phrase grounding annotations to ensure the generated

content is more precise and contextually aligned with clinical data. Evaluation on large-scale

public datasets demonstrate that the proposed framework produces more accurate and clinically

relevant radiology reports.

Overall, this thesis contributes to the development of more efficient and scalable multimodal AI

models, particularly for applications in the medical domain. By integrating external knowledge,

adapting SSL to medical data, and improving image-text alignment, the proposed methodologies

offer significant advancements in the capabilities of AI systems for medical diagnostics

and decision-making.

Multimodal foundation models, pre-trained on large-scale data, effectively capture vast amounts of factual and commonsense knowledge. However, these models store all their knowledge within their parameters, requiring increasingly larger models and training data to capture more knowledge. To address this limitation and achieve a more scalable and modular integration of knowledge, we propose a novel knowledge graph-augmented multimodal model. This approach enables a base multimodal model to access pertinent information from an external knowledge graph. Our methodology leverages existing general domain knowledge to facilitate vision-language pre-training using paired images and text descriptions. We conduct comprehensive evaluations demonstrating that our model outperforms state-of-the-art models and yields comparable results to much larger models trained on more extensive datasets. Notably, our model reached a 145 Cider score on MS COCO Captions using only 2.9 million samples, outperforming a 1.4B parameter model by 1.7% despite having 11 times fewer parameters.