Understanding the expressive power of transformers through the lens of formal language theory @SimonsInstituteTOC

Simons Institute | Understanding the expressive power of transformers through the lens of formal language theory @SimonsInstituteTOC | Uploaded 2 months ago | Updated 18 hours ago
Anthony Lin (TU Kaiserslautern)
https://simons.berkeley.edu/talks/anthony-lin-tu-kaiserslautern-2024-07-02
Synthesis of Models and Systems

What can you *learn* with transformers? I will discuss recent developments at the intersection of formal language theory and machine learning, especially with the goal to understand the expressiveness of transformers. Among others, I will discuss results connected my recent work (e.g. from ICLR’24).

Controller Synthesis Beyond the Worst Case

Social Behavior Prediction from Video Observations

$Agnostic Proper Learning of Monotone Functions: Beyond the Black-Box Correction Barrier Jane Lange (Massachusetts Institute of Technology) https://simons.berkeley.edu/talks/jane-lange-massachusetts-institute-technology-2024-08-02 Sublinear Graph Simplification We give the first agnostic, efficient, proper learning algorithm for monotone Boolean functions. Given 2^Õ(sqrt(n)/ε) uniformly random examples of an unknown function f:{±1}^n→{±1}, our algorithm outputs a hypothesis g:{±1}^n→{±1} that is monotone and (opt+ε)-close to f, where opt is the distance from f to the closest monotone function. The running time of the algorithm (and consequently the size and evaluation time of the hypothesis) is also 2^Õ(sqrt(n)/ε), nearly matching the lower bound of Blais et al (RANDOM 15). We also give an algorithm for estimating up to additive error ε the distance of an unknown function f to monotone using a run-time of 2^Õ(sqrt(n)/ε). Previously, for both of these problems, sample-efficient algorithms were known, but these algorithms were not run-time efficient. Our work thus closes this gap in our knowledge between the run-time and sample complexity. This work builds upon the improper learning algorithm of Bshouty and Tamon (JACM 96) and the proper semiagnostic learning algorithm of Lange, Rubinfeld, and Vasilyan (FOCS 22), which obtains a non-monotone Boolean-valued hypothesis, then corrects it to monotone using query-efficient local computation algorithms on graphs. This black-box correction approach can achieve no error better than 2opt+ε information-theoretically; we bypass this barrier by a) augmenting the improper learner with a convex optimization step, and b) learning and correcting a real-valued function before rounding its values to Boolean. Our real-valued correction algorithm solves the poset sorting problem of [LRV22] for functions over general posets with non-Boolean labels.$

Streaming Algorithms for Connectivity Augmentation

Sublinear algorithms in social networks via core-periphery decomposition

Synthesis and Verification of Finite Horizon Tasks

Bridging the data gap between LLMs and children

Illuminating protein space with generative models

Sparsification for communication-efficient distributed symmetry-breaking

Understanding Generalization from Pre-training Loss to Downstream Tasks