ru
ru
Source
Nature
DATE OF PUBLICATION
03/13/2023
Authors
Artem Shelmanov Dmitry Dylov Daniil Chesakov Alexander Selivanov Oleg Rogov Irina Fedulova
Share

Medical image captioning via generative pretrained transformers

Abstract

The proposed model for automatic clinical image caption generation combines the analysis of radiological scans with structured patient information from the textual records. It uses two language models, the Show-Attend-Tell and the GPT-3, to generate comprehensive and descriptive radiology records. The generated textual summary contains essential information about pathologies found, their location, along with the 2D heatmaps that localize each pathology on the scans. The model has been tested on two medical datasets, the Open-I, MIMIC-CXR, and the general-purpose MS-COCO, and the results measured with natural language assessment metrics demonstrated its efficient applicability to chest X-ray image captioning.

Full text

Similar publications

MICCAI
Centerline-Diameters Data Structure for Interactive Segmentation of Tube-Shaped Objects
Ilyas Sirazitdinov, Dmitry Dylov
SOURCE
ACL
Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantification
Ekaterina Fadeeva, Aleksandr Rubashevskii, Artem Shelmanov, Sergey Petrakov, Haonan Li, Hamdy Mubarak, Evgenii Tsymbalov, Gleb Kuzmin, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov
SOURCE
IJCAI
Probabilistically Robust Watermarking of Neural Networks
Mikhail Pautov, Nikita Bogdanov, Stanislav Pyatkin, Oleg Rogov, Ivan Oseledets
SOURCE
Aging Cell
Epistemic uncertainty challenges aging clock reliability in predicting rejuvenation effects
Dmitrii Kriukov, Ekaterina Kuzmina, Evgeniy Efimov, Dmitry Dylov, Ekaterina Khrameeva
SOURCE
Chaos, Solitons & Fractals
Suppressing modulation instability with reinforcement learning
Nikolay Kalmykov, Rishat Zagidullin, Oleg Rogov, Sergey Rykovanov, Dmitry Dylov
SOURCE
EMNLP
LM-Polygraph: Uncertainty Estimation for Language Models
Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, Timothy Baldwin, Artem Shelmanov
SOURCE
AACL-IJCNLP
Uncertainty Estimation for Debiased Models: Does Fairness Hurt Reliability?
Gleb Kuzmin, Artem Vazhentsev, Artem Shelmanov, Xudong Han, Simon Šuster, Maxim Panov, Alexander Panchenko, Timothy Baldwin
SOURCE
Contact us
Join AIRI
You can ask us a question or suggest a joint project in the field of AI

Message sent.

Thank you!

Something went wrong. Try again