Neural Harmonics: Bridging Spectral Embedding and Matrix Completion in Self-Supervised Learning

Аннотация

Self-supervised methods received tremendous attention thanks to their seeminglyheuristic approach to learning representations that respect the semantics of thedata without any apparent supervision in the form of labels. A growing bodyof literature is already being published in an attempt to build a coherent andtheoretically grounded understanding of the workings of a zoo of losses used inmodern self-supervised representation learning methods. In this paper, we attemptto provide an understanding from the perspective of a Laplace operator and connectthe inductive bias stemming from the augmentation process to a low-rank matrixcompletion problem. To this end, we leverage the results from low-rank matrixcompletion to provide theoretical analysis on the convergence of modern SSLmethods and a key property that affects their downstream performance.