Advancing Sequential Manga Colorization for AR through Data Synthesis

Abstract

Manga colorization in augmented reality (AR) environments presents unique challenges, particularly when colorizing manga pages captured in photos under various real-world conditions. Testing models in AR settings for manga colorization has been a significant challenge, primarily because of the absence of suitable datasets tailored for this task. To address this, we propose a benchmark for evaluating existing colorization models. We first collected a relatively small dataset of manga book photos taken in settings suitable for AR applications. Then, we developed a method that leverages a pretrained diffusion model to generate synthetic photos from scans of manga pages. Using large datasets of manga scans, we created an extensive synthetic dataset. Combining both real and synthetic data, we established a comprehensive benchmark for manga colorization in AR scenarios. We tested existing models for natural image and manga colorization on this benchmark. As a result, our evaluation showed that current models are not well-suited for AR-based colorization tasks, indicating a need for further improvement.