How to combine classic algorithms and AI - Learning with Algorithmic Supervision via Relaxations Felix Petersen 2021-11-01 | Learning with Algorithmic Supervision via Continuous RelaxationsFelix Petersen, Christian Borgelt, Hilde Kuehne, Oliver DeussenPublished at NeurIPS 2021Paper: arxiv.org/pdf/2110.05651.pdfCode:github.com/Felix-Petersen/algovision
Computer Vision Models with LLM Training Dynamics (TrAct) Felix Petersen 2024-12-03 | Official video for our NeurIPS 2024 Paper "TrAct: Making First-layer Pre-Activations Trainable".TrAct: Making First-layer Pre-Activations TrainableFelix Petersen Christian Borgelt, Stefano ErmonAbstract: We consider the training of the first layer of vision models and notice the clear relationship between pixel values and gradient update magnitudes: the gradients arriving at the weights of a first layer are by definition directly proportional to (normalized) input pixel values. Thus, an image with low contrast has a smaller impact on learning than an image with higher contrast, and a very bright or very dark image has a stronger impact on the weights than an image with moderate brightness. In this work, we propose performing gradient descent on the embeddings produced by the first layer of the model. However, switching to discrete inputs with an embedding layer is not a reasonable option for vision models. Thus, we propose the conceptual procedure of (i) a gradient descent step on first layer activations to construct an activation proposal, and (ii) finding the optimal weights of the first layer, i.e., those weights which minimize the squared distance to the activation proposal. We provide a closed form solution of the procedure and adjust it for robust stochastic training while computing everything efficiently. Empirically, we find that TrAct (Training Activations) speeds up training by factors between 1.25x and 4x while requiring only a small computational overhead. We demonstrate the utility of TrAct with different optimizers for a range of different vision models including convolutional and transformer architectures.NeurIPS 2024Paper:arxiv.org/abs/2410.23970Code:github.com/Felix-Petersen/tract
Newton Losses: Using Curvature Information for Learning with Differentiable Algorithms - NeurIPS2024 Felix Petersen 2024-11-28 | Official video for our NeurIPS 2024 Paper "Newton Losses: Using Curvature Information for Learning with Differentiable Algorithms".Felix Petersen, Christian Borgelt, Tobias Sutter, Hilde Kuehne, Oliver Deussen, Stefano ErmonAbstract: When training neural networks with custom objectives, such as ranking losses and shortest-path losses, a common problem is that they are, per se, non-differentiable. A popular approach is to continuously relax the objectives to provide gradients, enabling learning. However, such differentiable relaxations are often non-convex and can exhibit vanishing and exploding gradients, making them (already in isolation) hard to optimize. Here, the loss function poses the bottleneck when training a deep neural network. We present Newton Losses, a method for improving the performance of existing hard to optimize losses by exploiting their second-order information via their empirical Fisher and Hessian matrices. Instead of training the neural network with second-order techniques, we only utilize the loss function's second-order information to replace it by a Newton Loss, while training the network with gradient descent. This makes our method computationally efficient. We apply Newton Losses to eight differentiable algorithms for sorting and shortest-paths, achieving significant improvements for less-optimized differentiable algorithms, and consistent improvements, even for well-optimized differentiable algorithms.NeurIPS 2024Paper:arxiv.org/abs/2410.19055Code:github.com/Felix-Petersen/newton-losses
Convolutional Differentiable Logic Gate Networks - NeurIPS Oral - difflogic Felix Petersen 2024-11-11 | Official 5 minute version video for our NeurIPS 2024 Oral Paper "Convolutional Differentiable Logic Gate Networks".Felix Petersen, Hilde Kuehne, Christian Borgelt, Julian Welzel, Stefano ErmonAbstract: With the increasing inference cost of machine learning models, there is a growing interest in models with fast and efficient inference. Recently, an approach for learning logic gate networks directly via a differentiable relaxation was proposed. Logic gate networks are faster than conventional neural network approaches because their inference only requires logic gate operators such as NAND, OR, and XOR, which are the underlying building blocks of current hardware and can be efficiently executed. We build on this idea, extending it by deep logic gate tree convolutions, logical OR pooling, and residual initializations. This allows scaling logic gate networks up by over one order of magnitude and utilizing the paradigm of convolution. On CIFAR-10, we achieve an accuracy of 86.29% using only 61 million logic gates, which improves over the SOTA while being 29x smaller.NeurIPS 2024Paper:arxiv.org/abs/2411.04732Code:github.com/Felix-Petersen/difflogic
AI for Formula Translation - Neural Machine Translation for Mathematical Formulae - ACL 2023 Felix Petersen 2023-12-10 | Neural Machine Translation for Mathematical FormulaeFelix Petersen, Moritz Schubotz, Andre Greiner-Petter, Bela GippPublished at ACL 2023arxiv.org/abs/2305.16433
ISAAC Newton - Accelerating Neural Network Training with Input-based Approximate Curvature Felix Petersen 2023-04-29 | Official video for our ICLR 2023 Paper "ISAAC Newton: Input-based Approximate Curvature for Newton's Method".Felix Petersen, Tobias Sutter, Christian Borgelt, Dongsung Huh, Hilde Kuehne, Yuekai Sun, Oliver DeussenICLR 2023Paper:openreview.net/pdf?id=0paCJSFW7jCode:github.com/Felix-Petersen/isaac
Differentiable Top-k Classification Learning | New ImageNet SOTA Felix Petersen 2022-07-17 | Differentiable Top-k Classification LearningFelix Petersen, Hilde Kuehne, Christian Borgelt, Oliver DeussenICML 2022Abstract:The top-k classification accuracy is one of the core metrics in machine learning. Here, k is conventionally a positive integer, such as 1 or 5, leading to top-1 or top-5 training objectives. In this work, we relax this assumption and optimize the model for multiple k simultaneously instead of using a single k. Leveraging recent advances in differentiable sorting and ranking, we propose a differentiable top-k cross-entropy classification loss. This allows training the network while not only considering the top-1 prediction, but also, e.g., the top-2 and top-5 predictions. We evaluate the proposed loss function for fine-tuning on state-of-the-art architectures, as well as for training from scratch. We find that relaxing k does not only produce better top-5 accuracies, but also leads to top-1 accuracy improvements. When fine-tuning publicly available ImageNet models, we achieve a new state-of-the-art for these models.Paper:arxiv.org/abs/2206.07290Code:github.com/Felix-Petersen/difftopk
GenDR: The Generalized Differentiable Renderer Felix Petersen 2022-06-13 | GenDR: A Generalized Differentiable RendererFelix Petersen, Bastian Goldluecke, Christian Borgelt, Oliver DeussenCVPR 2022Paper:arxiv.org/abs/2204.13845Code:github.com/Felix-Petersen/gendrMusic: Autumn Leaves by Maarten Schellekens
Monotonic Differentiable Sorting Networks for Learning to Rank (diffsort) Felix Petersen 2022-03-21 | Monotonic Differentiable Sorting NetworksFelix Petersen, Christian Borgelt, Hilde Kuehne, Oliver DeussenICLR 2022Paper:arxiv.org/abs/2203.09630Code:github.com/Felix-Petersen/diffsortConference Page:
How to make AI Fair - Post-processing for Individual Fairness Felix Petersen 2021-11-30 | Post-processing for Individual Fairnessby Felix Petersen*, Debarghya Mukherjee*, Yuekai Sun, Mikhail YurochkinPublished at NeurIPS 2021Paper:Code:github.com/Felix-Petersen/fairness-post-processing
diffsort - Differentiable Sorting Networks for Scalable Sorting and Ranking Supervision Felix Petersen 2021-07-14 | Differentiable Sorting Networks for Scalable Sorting and Ranking SupervisionFelix Petersen, Christian Borgelt, Hilde Kuehne, Oliver DeussenPaper:arxiv.org/abs/2105.04019Code:github.com/Felix-Petersen/diffsortConference Page:icml.cc/virtual/2021/poster/8439
