We use cookies to understand how you use our site and to improve your experience. This includes personalizing content and advertising. To learn more, click here. By continuing to use our site, you accept our use of cookies. Cookie Policy.

Features Partner Sites Information LinkXpress hp
Sign In
Advertise with Us
LGC Clinical Diagnostics

Download Mobile App




AI Tool Combines Data from Medical Images with Text to Predict Cancer Prognoses

By LabMedica International staff writers
Posted on 10 Jan 2025
Print article
Image: The unique AI tool predicts cancer prognoses and responses to treatment (Photo courtesy of Shutterstock)
Image: The unique AI tool predicts cancer prognoses and responses to treatment (Photo courtesy of Shutterstock)

The integration of visual data (such as microscopic and X-ray images, CT and MRI scans) with textual information (like exam notes and communications between doctors of different specialties) is a crucial aspect of cancer care. While artificial intelligence (AI) tools have been increasingly employed in clinical settings, their primary application has been in diagnostics rather than prognosis. AI aids doctors in reviewing images and detecting disease-related anomalies, such as abnormally shaped cells, but developing computerized models that can combine various types of data has been a challenge. One of the difficulties is the need to train these models with large amounts of labeled and paired data, like a microscope slide showing a cancerous tumor alongside the clinical notes of the patient from whom the tumor was obtained. However, curated and annotated datasets are often scarce. Researchers have now developed an AI model capable of integrating both visual and textual data. After training on 50 million medical images of standard pathology slides and more than 1 billion pathology-related texts, the model surpassed traditional methods in its ability to predict the prognoses of thousands of cancer patients, identify individuals with lung or gastroesophageal cancers likely to benefit from immunotherapy and pinpoint melanoma patients most at risk of experiencing a recurrence.

The model, named MUSK (multimodal transformer with unified mask modeling), was developed by researchers at Stanford Medicine (Stanford, CA, USA). MUSK marks a significant departure from the typical use of AI in clinical settings, and the researchers believe it has the potential to transform how AI can guide patient care. In AI terminology, MUSK is considered a foundation model. Foundation models, which are pretrained on large datasets, can be further fine-tuned with additional training to handle specific tasks. Since MUSK was designed to utilize unpaired multimodal data that does not meet the traditional requirements for training AI, it can leverage a much larger pool of data for its initial learning phase. As a result, subsequent training only requires smaller, more specialized datasets. Essentially, MUSK is a ready-to-use tool that doctors can customize to answer specific clinical questions.

To develop MUSK, the researchers gathered microscopic tissue slides, pathology reports, and follow-up data (including patient outcomes) from The Cancer Genome Atlas, a national database, for individuals with 16 major cancer types, such as breast, lung, colorectal, pancreatic, kidney, bladder, and head and neck cancers. This data was used to train MUSK to predict disease-specific survival or the percentage of patients who have not died from a specific disease within a given time frame. According to the study, published in Nature, MUSK accurately predicted disease-specific survival for all cancer types 75% of the time. In comparison, traditional predictions based on a person’s cancer stage and other clinical risk factors were correct 64% of the time. In another example, MUSK was trained to analyze extensive data to predict which patients with lung cancer or cancers of the gastric and esophageal tracts are most likely to benefit from immunotherapy.

For non-small cell lung cancer, MUSK identified patients who responded well to immunotherapy approximately 77% of the time. In contrast, the conventional method of predicting immunotherapy response based on PD-L1 expression was correct only 61% of the time. Similarly, when the researchers trained MUSK to identify melanoma patients at high risk of relapse within five years after initial treatment, the model was accurate about 83% of the time, which is roughly 12% more accurate than other foundation models.

“MUSK can accurately predict the prognoses of people with many different kinds and stages of cancer,” said Ruijiang Li, MD, an associate professor of radiation oncology. “We designed MUSK because, in clinical practice, physicians never rely on just one type of data to make clinical decisions. We wanted to leverage multiple types of data to gain more insight and get more precise predictions about patient outcomes.”

“What’s unique about MUSK is the ability to incorporate unpaired multimodal data into pretraining, which substantially increases the scale of data compared with paired data required by other models,” added Li.

Gold Member
C-Reactive Protein Reagent
CRP Ultra Wide Range Reagent Kit
Gold Member
Fully Automated Cell Density/Viability Analyzer
BioProfile FAST CDV
New
Silver Member
Epstein-Barr Virus Test
ReQuest EB VCA IgM ELISA Kit
New
Electroporation System
Gibco CTS Xenon

Print article

Channels

Microbiology

view channel
Image: The breakthrough system offers a faster way to diagnose bloodborne infections (Photo courtesy of Melio)

Culture-Free Platform Rapidly Identifies Blood Stream Infections

Neonatal sepsis is a life-threatening condition that results from bloodstream infections in newborns under 28 days old. Due to their immature immune systems, newborns are especially vulnerable to infections.... Read more

Technology

view channel
Image: Human tear film protein sampling methods (Photo courtesy of Clinical Proteomics. 2024 Mar 13;21:23. doi: 10.1186/s12014-024-09475-8)

New Lens Method Analyzes Tears for Early Disease Detection

Bodily fluids, including tears and saliva, carry proteins that are released from different parts of the body. The presence of specific proteins in these biofluids can be a sign of health issues.... Read more

Industry

view channel
Image: The game-changing immunoassay diagnostics platform delivers results from whole blood sample in 10 minutes (Photo courtesy of SpinChip)

bioMérieux Acquires Norwegian Immunoassay Start-Up SpinChip Diagnostics

bioMérieux (Marcy l’Étoile, France) has agreed to acquire SpinChip Diagnostics (Oslo, Norway), the developer of a game-changing immunoassay diagnostics platform. The small benchtop analyzer is well adapted... Read more
Copyright © 2000-2025 Globetech Media. All rights reserved.