The Structure-First Approach for Multimodal Data Integration

Whether diagnosing cancer or cardiovascular disease, physicians rely on a combination of data sources, including clinical records, laboratory tests, imaging procedures, tissue analyses, and physiological measurements such as ECGs. Accurate diagnosis and risk assessment can only be achieved by combining these diverse types of information in a meaningful way. Physicians organise and interpret information according to overarching medical concepts: Diseases are linked to specific organs and biological systems, grouped into categories and subcategories, and assessed based on their stage, severity, or progression. In other words, clinical reasoning is guided by semantic relationships rather than data types.

TWIN-X aims to incorporate these clinical categorisation practices into AI models in order to render diagnosis and clinical decision-making both more precise and more efficient. To achieve this, the project follows a novel structure-first approach that complements existing digital twin models based on different types of data.

Rather than merging all available data into a single mathematical representation, often referred to as a latent space, TWIN-X first organises information in a hierarchical manner that reflects clinical context. Information from different data sources, so-called multimodal data, is translated into clinically meaningful units and structured around organ systems, diseases, treatments, and their progression over time. These data are mapped to internationally recognised healthcare standards, ensuring that medical information is represented consistently across different hospitals and healthcare settings. This way, TWIN-X enables AI models to reason over patient information in a transparent and trustworthy way.