Browse AI research papers authored by Ruth Cytter Kuint. Found 2 papers.

Reliable extraction of structured data from radiology reports using Large Language Models (LLMs) remains challenging, especially for complex, non-English texts like Hebrew. This study introduces an agent-based uncertainty-aware approach to improve the trustworthiness of LLM predictions in medical applications. We analyzed 9,683 Hebrew radiology reports from Crohn's disease patients (from 2010 to 2023) across three medical centers. A subset of 512 reports was manually annotated for six gastrointestinal organs and 15 pathological findings, while the remaining reports were automatically annotated using HSMP-BERT. Structured data extraction was performed using Llama 3.1 (Llama 3-8b-instruct) with Bayesian Prompt Ensembles (BayesPE), which employed six semantically equivalent prompts to estimate uncertainty. An Agent-Based Decision Model integrated multiple prompt outputs into five confidence levels for calibrated uncertainty and was compared against three entropy-based models. Performance was evaluated using accuracy, F1 score, precision, recall, and Cohen's Kappa before and after filtering high-uncertainty cases. The agent-based model outperformed the baseline across all metrics, achieving an F1 score of 0.3967, recall of 0.6437, and Cohen's Kappa of 0.3006. After filtering high-uncertainty cases (greater than or equal to 0.5), the F1 score improved to 0.4787, and Kappa increased to 0.4258. Uncertainty histograms demonstrated clear separation between correct and incorrect predictions, with the agent-based model providing the most well-calibrated uncertainty estimates. By incorporating uncertainty-aware prompt ensembles and an agent-based decision model, this approach enhances the performance and reliability of LLMs in structured data extraction from radiology reports, offering a more interpretable and trustworthy solution for high-stakes medical applications.

## Overview

* Research on extracting data from Hebrew radiology reports using large language models (LLMs)
* Developed uncertainty-aware approach with agent-based decision making
* Analyzed 9,683 reports from Crohn's disease patients across three medical centers
* Used Llama 3.1 model with Bayesian Prompt Ensembles
* Achieved improved accuracy after filtering uncertain predictions
* Demonstrated better reliability for medical applications

## Plain English Explanation

Getting useful information from medical reports written in languages like Hebrew is hard for AI systems. This research team developed a smarter way to do it by making the AI express how confident it is in its answers.

Think of it like having multiple doctors look at the same medical report. Each doctor might interpret things slightly differently. By combining their views and noting where they agree or disagree, you get a more reliable final opinion.

The [agent-based uncertainty awareness](https://aimodels.fyi/papers/arxiv/agent-based-uncertainty-awareness-improves-automated-radiology) system works like a chief doctor who looks at all these opinions and decides how confident they can be in the final diagnosis. When the system isn't sure, it says so instead of making risky guesses.

## Key Findings

The research showed impressive improvements in accuracy when using this new approach. The system performed best when it could admit uncertainty about difficult cases:

* F1 score improved from 0.3967 to 0.4787 after filtering uncertain predictions
* Achieved 64.37% recall on initial testing
* System could clearly separate between correct and incorrect predictions
* [Multi-agent diagnostic assistance](https://aimodels.fyi/papers/arxiv/magda-multi-agent-guideline-driven-diagnostic-assistance) showed better calibrated uncertainty estimates

## Technical Explanation

The team used **Llama 3.1** (8B parameter version) as their base model. They created six different ways to ask the same question (prompts) and used **Bayesian Prompt Ensembles** to estimate uncertainty.

The [language models retrieval](https://aimodels.fyi/papers/arxiv/language-models-retrieval-augmented-generation-automated-structured) system processed reports in multiple stages:

1. Initial processing of 9,683 Hebrew radiology reports
2. Manual annotation of 512 reports for validation
3. Automatic annotation of remaining reports using HSMP-BERT
4. Implementation of agent-based decision model with five confidence levels

## Critical Analysis

Several limitations deserve consideration:

* Limited to Hebrew language reports only
* Focused specifically on Crohn's disease cases
* Relatively small manually annotated dataset
* [Automated structured generation](https://aimodels.fyi/papers/arxiv/enhancing-llms-impression-generation-radiology-reports-through) may need further validation

The research could benefit from testing across more languages and medical conditions. Additionally, the impact of different prompt designs needs more investigation.

## Conclusion

This research represents a significant step forward in making AI systems more trustworthy for medical applications. The [agentic LLM workflows](https://aimodels.fyi/papers/arxiv/agentic-llm-workflows-generating-patient-friendly-medical) approach of combining multiple perspectives with uncertainty awareness shows promise for improving medical data extraction.

The ability to know when to express uncertainty rather than make potentially dangerous mistakes is crucial for healthcare applications. This work provides a foundation for developing more reliable AI systems in medicine.

Agent-Based Uncertainty Awareness Improves Automated Radiology Report Labeling with an Open-Source Large Language Model

Automatic conversion of free-text radiology reports into structured data using Natural Language Processing (NLP) techniques is crucial for analyzing diseases on a large scale. While effective for tasks in widely spoken languages like English, generative large language models (LLMs) typically underperform with less common languages and can pose potential risks to patient privacy. Fine-tuning local NLP models is hindered by the skewed nature of real-world medical datasets, where rare findings represent a significant data imbalance. We introduce SMP-BERT, a novel prompt learning method that leverages the structured nature of reports to overcome these challenges. In our studies involving a substantial collection of Crohn's disease radiology reports in Hebrew (over 8,000 patients and 10,000 reports), SMP-BERT greatly surpassed traditional fine-tuning methods in performance, notably in detecting infrequent conditions (AUC: 0.99 vs 0.94, F1: 0.84 vs 0.34). SMP-BERT empowers more accurate AI diagnostics available for low-resource languages.

## Overview

- This paper explores the use of prompt-learning techniques for structured information extraction from Crohn's disease radiology reports in a low-resource language.
- The researchers develop a novel approach that leverages pre-trained language models and customized prompts to extract relevant medical information from radiology reports, even in languages with limited available data.
- The proposed method demonstrates promising results for improving the accuracy and efficiency of information extraction in this clinical domain, potentially aiding clinicians and researchers working with Crohn's disease patients.

## Plain English Explanation

In this study, the researchers looked at a way to help doctors and scientists better understand medical information from X-ray and imaging reports about a disease called Crohn's disease. Crohn's disease is a condition that can cause inflammation and other problems in the digestive system.

The researchers developed a new technique that uses [pre-trained language models](https://aimodels.fyi/papers/arxiv/general-purpose-vs-domain-adapted-large-language) and customized "prompts" to extract important details from radiology reports, even in languages where there is not a lot of existing data available. Prompts are short instructions that help the language model focus on the right information.

This approach was tested on Crohn's disease radiology reports and showed promising results for improving the accuracy and efficiency of extracting relevant medical details. This could be very useful for clinicians treating Crohn's patients and researchers studying the disease, especially in areas where the local language has limited medical data available.

## Technical Explanation

The paper presents a novel approach for structured information extraction from Crohn's disease radiology reports in a low-resource language, leveraging the power of [prompt-learning](https://aimodels.fyi/papers/arxiv/towards-efficient-patient-recruitment-clinical-trials-application).

The researchers first fine-tune a pre-trained language model on a small corpus of Crohn's disease radiology reports in the target low-resource language. They then design customized prompts that guide the model to extract specific medical entities and attributes related to the Crohn's disease diagnosis, such as the location and severity of intestinal inflammation.

To evaluate their method, the team collects a dataset of Crohn's disease radiology reports in the low-resource language and annotates them with ground-truth structured information. They compare the performance of their prompt-based approach to other information extraction techniques, including [named entity recognition](https://aimodels.fyi/papers/arxiv/llms-biomedicine-study-clinical-named-entity-recognition) and [radiology report generation](https://aimodels.fyi/papers/arxiv/mrscore-evaluating-radiology-report-generation-llm-based) models.

The results demonstrate that the prompt-learning approach outperforms the baseline methods, achieving higher accuracy in extracting the relevant medical information from the Crohn's disease radiology reports. The researchers also conduct ablation studies to understand the impact of different prompt design choices on the model's performance.

## Critical Analysis

The paper presents a thoughtful and well-designed approach to addressing the challenge of structured information extraction in a low-resource language setting. The use of prompt-learning techniques is a promising direction, as it allows the model to focus on the specific medical entities and attributes relevant to Crohn's disease without requiring a large annotated dataset.

However, the paper does not extensively discuss the limitations of the proposed method. For example, it would be interesting to understand how the approach would scale to more diverse or noisy radiology report data, or how it might perform on other types of medical documents beyond radiology reports.

Additionally, the paper could have provided more insight into the potential biases or errors that might arise from the prompt-learning approach, and how these could be mitigated. A more thorough discussion of the ethical considerations, such as the potential misuse of the extracted information or the impact on patient privacy, would also strengthen the critical analysis.

Overall, this research represents an important step forward in leveraging [advanced language models](https://aimodels.fyi/papers/arxiv/serpent-vlm-self-refining-radiology-report-generation) for clinical data extraction in low-resource settings. Further exploration of the limitations and potential risks of the approach, as well as its broader applicability, would make for a more comprehensive understanding of the technique and its implications.

## Conclusion

This paper introduces a novel prompt-learning based approach for structured information extraction from Crohn's disease radiology reports in a low-resource language. The researchers demonstrate that their method outperforms baseline techniques, highlighting the potential of this approach to aid clinicians and researchers working with Crohn's disease patients, especially in regions with limited medical data availability.

While the paper presents a well-designed and promising solution, further research is needed to fully understand the limitations, potential biases, and broader applicability of the prompt-learning technique. Nonetheless, this work represents an important contribution to the field of clinical natural language processing and information extraction, with implications for improving patient care and advancing medical research in low-resource settings.

Ruth Cytter Kuint

Authored papers

Can AI learn to "double-check" its work when reading tricky medical reports in other languages?

Can prompt-learning unlock the secrets hidden in Crohn's disease radiology reports, even when the language is rare and data is scarce?

Claim this author profile