ru
ru
Source
Nature
DATE OF PUBLICATION
03/13/2023
Authors
Artem Shelmanov Dmitry Dylov Daniil Chesakov Alexander Selivanov Oleg Rogov Irina Fedulova
Share

Medical image captioning via generative pretrained transformers

Abstract

The proposed model for automatic clinical image caption generation combines the analysis of radiological scans with structured patient information from the textual records. It uses two language models, the Show-Attend-Tell and the GPT-3, to generate comprehensive and descriptive radiology records. The generated textual summary contains essential information about pathologies found, their location, along with the 2D heatmaps that localize each pathology on the scans. The model has been tested on two medical datasets, the Open-I, MIMIC-CXR, and the general-purpose MS-COCO, and the results measured with natural language assessment metrics demonstrated its efficient applicability to chest X-ray image captioning.

Full text

Similar publications

IJCAI
Probabilistically Robust Watermarking of Neural Networks
Mikhail Pautov, Nikita Bogdanov, Stanislav Pyatkin, Oleg Rogov, Ivan Oseledets
SOURCE
Aging Cell
Epistemic uncertainty challenges aging clock reliability in predicting rejuvenation effects
Dmitrii Kriukov, Ekaterina Kuzmina, Evgeniy Efimov, Dmitry Dylov, Ekaterina Khrameeva
SOURCE
Chaos, Solitons & Fractals
Suppressing modulation instability with reinforcement learning
Nikolay Kalmykov, Rishat Zagidullin, Oleg Rogov, Sergey Rykovanov, Dmitry Dylov
SOURCE
EMNLP
LM-Polygraph: Uncertainty Estimation for Language Models
Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, Timothy Baldwin, Artem Shelmanov
SOURCE
AACL-IJCNLP
Uncertainty Estimation for Debiased Models: Does Fairness Hurt Reliability?
Gleb Kuzmin, Artem Vazhentsev, Artem Shelmanov, Xudong Han, Simon Šuster, Maxim Panov, Alexander Panchenko, Timothy Baldwin
SOURCE
Language Resources and Evaluation
Text augmentation for semantic frame induction and parsing
Saba Anwar, Artem Shelmanov, Nikolay Arefyev, Alexander Panchenko, Chris Biemann
SOURCE
Language Resources and Evaluation
NEREL: a Russian information extraction dataset with rich annotation for nested entities, relations, and wikidata entity links
Natalia Loukachevitch, Ekaterina Artemova, Tatiana Batura, Pavel Braslavski, Vladimir Ivanov, Suresh Manandhar, Alexander Pugachev, Igor Rozhkov, Artem Shelmanov, Elena Tutubalina, Alexey Yandutov
SOURCE
Contact us
Join AIRI
You can ask us a question or suggest a joint project in the field of AI

Message sent.

Thank you!

Something went wrong. Try again