Héctor Pastén
Cargo: Colaborador Asociado
Especialidad:
Matemáticas avanzadas y fundamentos teóricos de la IA
Matemático, especializado en teoría de números, con investigación en áreas como variable compleja, análisis p-ádico, geometría algebraica y lógica matemática. Obtuvo su doctorado en Queen’s University (Canadá) en 2014, tras lo cual se desempeñó como investigador en la Universidad de Harvard y en el Institute for Advanced Study de Princeton.
PUBLICACIONES
Understanding how Transformers work and how they process information is key to the theoretical and empirical advancement of these machines. In this work, we demonstrate the existence of two phenomena in Transformers, namely isolation and continuity. Both of these phenomena hinder Transformers to learn even simple pattern sequences. Isolation expresses that any learnable sequence must be isolated from another learnable sequence, and hence some sequences cannot be learned by a single Transformer at the same time. Continuity entails that an attractor basin forms around a learned sequence, such that any sequence falling in that basin will collapse towards the learned sequence. Here, we mathematically prove these phenomena emerge in all Transformers that use compact positional encoding, and design rigorous experiments, demonstrating that the theoretical limitations we shed light on occur on the practical scale.
Edificio de Innovación UC, Piso 2
Vicuña Mackenna 4860
Macul, Chile
Vicuña Mackenna 4860
Macul, Chile