Uploaded April 2024 | Updated November 2026, 4 days ago
Francois Lanusse
"A New Era of Multi-Modal Self-Supervised Learning for Astrophysics"
Deep Learning has seen a recent shift in paradigm, from training specialized models on dedicated datasets, so-called Foundation Models, trained in a self-supervised manner on vast amounts of data and then adapted to solve specific tasks with state-of-the-art performance. This new paradigm has been exceptionally successful not only for large language models (LLMs) but in other domains such as vision models. However applications of this new approach in astrophysics are still very scarce, for reasons ranging from new architectures to the (surprising) lack of availability of suitable large scale datasets.
In this talk, I will discuss our recent work on deploying such a Foundation Model approach in the context of representation learning for astronomical photometric and spectroscopic observations of galaxies. Our aim is to embed these inhomogeneous observations (e.g. different types of measurements, different instruments, etc...) into a shared embedding space, in a completely self-supervised manner. These embeddings can then be used for a variety of downstream applications (e.g. redshift estimation, morphology classification) with very simple machine learning methods and reach near optimal performance.
More specifically, I will present our AstroCLIP method which allows us to align embeddings between data modalities, and our more recent results on building highly effective image embeddings based on a vision transformer architecture. I will also comment on these results from an information theory point of view, and conclude on future prospects for this approach in the context of upcoming large scale galaxy surveys.
Francois Lanusse
"A New Era of Multi-Modal Self-Supervised Learning for Astrophysics"
Deep Learning has seen a recent shift in paradigm, from training specialized models on dedicated datasets, so-called Foundation Models, trained in a self-supervised manner on vast amounts of data and then adapted to solve specific tasks with state-of-the-art performance. This new paradigm has been exceptionally successful not only for large language models (LLMs) but in other domains such as vision models. However applications of this new approach in astrophysics are still very scarce, for reasons ranging from new architectures to the (surprising) lack of availability of suitable large scale datasets.
In this talk, I will discuss our recent work on deploying such a Foundation Model approach in the context of representation learning for astronomical photometric and spectroscopic observations of galaxies. Our aim is to embed these inhomogeneous observations (e.g. different types of measurements, different instruments, etc...) into a shared embedding space, in a completely self-supervised manner. These embeddings can then be used for a variety of downstream applications (e.g. redshift estimation, morphology classification) with very simple machine learning methods and reach near optimal performance.
More specifically, I will present our AstroCLIP method which allows us to align embeddings between data modalities, and our more recent results on building highly effective image embeddings based on a vision transformer architecture. I will also comment on these results from an information theory point of view, and conclude on future prospects for this approach in the context of upcoming large scale galaxy surveys.






