SkipCLM: Enchancing Crosslingual Alignment of Decoder Transformer Models via Contrastive Learning and Skip Connection

Abstract

This paper proposes SkipCLM, a novelmethod for improving multilingual machinetranslation in Decoder Transformers. Weaugment contrastive learning for cross-lingualalignment with a trainable skip connection topreserve information crucial for accurate targetlanguage generation. Experiments withXGLM-564M on the Flores-101 benchmarkdemonstrate improved performance, particularlyfor en-de and en-zh direction translations,compared to direct sequence-to-sequencetraining and existing contrastive learning methods.Code is available at: https://github.com/snlp/skipclm.