ru
ru
Source
ACL / GenBench
DATE OF PUBLICATION
11/16/2024
Authors
Alexander Panchenko Anton Razzhigaev Denis Dimitrov Elizaveta Goncharova Maxim Kurkin Irina Abdullaeva Anastasia Lysenko
Share

OmniGen: A Multimodal Benchmark for Generalization Across Text, Visual, and Audio Modalities

Abstract

We introduce OmniDialog — the first trimodal comprehensive benchmark grounded in a knowledge graph (Wikidata) to evaluate the generalization of Large Multimodal Models (LMMs) across three modalities. Our benchmark consists of more than 4,000 dialogues, each averaging 10 turns, all annotated and cross-validated by human experts. The dialogues in our dataset are designed to prevent shortcut learning by incorporating various formats and misleading or irrelevant multimodal cues. We also evaluate both multimodal and unimodal models to gain insights into how they process modality inputs introduced in the conversation.

Full text DOWNLOAD pdf

Similar publications

ACL
A System for Answering Simple Questions in Multiple Languages
Anton Razzhigaev, Mikhail Salnikov, Valentin Malykh, Pavel Braslavski, Alexander Panchenko
SOURCE
CLEF / PAN
Overview of the multilingual text detoxification task at pan 2024
Daryna Dementieva, Daniil Moskovskiy, Nikolay Babakov, Abinew Ali Ayele, Naquee Rizwan, Florian Schneider, Xintong Wang, Seid Muhie Yimam, Dmitry Ustalov, Elisei Stakovskii, Alisa Smirnova, Ashaf Elnagar, Animesh Mukherjee, Alexander Panchenko
SOURCE
CLEF / PAN
PAN 2024 Multilingual TextDetox: Exploring Cross-lingual Transfer Using Large Language Models
Vitaly Protasov
SOURCE
CLEF / PAN
SomethingAwful at PAN 2024 TextDetox: Uncensored Llama 3 Helps to Censor Better.
Sergey Pletenev
SOURCE
ECCV / AIM
AIM 2024 Challenge on Video Saliency Prediction: Methods and Results
Andrey Moskalenko, Alexey Bryncev, Dmitry Vatolin, Radu Timofte, Gen Zhan, Li Yang, Yunlong Tang, Yiting Liao, Jiongzhi Lin, Baitao Huang, Morteza Moradi, Mohammad Moradi, Francesco Rundo, Concetto Spampinato, Ali Borji, Simone Palazzo, Yuxin Zhu, Yinan Sun, Huiyu Duan, Yuqin Cao, Ziheng Jia, Qiang Hu, Xiongkuo Min, Guangtao Zhai, Hao Fang, Runmin Cong, Xiankai Lu, Xiaofei Zhou, Wei Zhang, Chunyu Zhao, Wentao Mu, Tao Deng, Hamed R. Tavakoli
SOURCE
NLDB
S3: A Simple Strong Sample-effective Multimodal Dialog System
Elisei Rykov, Egor Malkershin, Alexander Panchenko
SOURCE
NLDB
Konstruktor: A Strong Baseline for Simple Knowledge Graph Question Answering
Maria Lysyuk, Mikhail Salnikov, Pavel Braslavski, Alexander Panchenko
SOURCE
Contact us
Join AIRI
You can ask us a question or suggest a joint project in the field of AI

Message sent.

Thank you!

Something went wrong. Try again