Uploaded September 2021 | Updated September 2026, 8 hours ago
Stefan Mihalas • Allen Institute for Brain Sciences • (9/17/21)
Abstract:
Computing with a mess: how nonstationary, heterogeneous and noisy components help the brain’s computational power While artificial neural networks have taken inspiration from biological ones, one salient difference exists at the level of components. Biological neurons and synapses have heterogeneous transfer functions, which are non-stationary in time and highly stochastic, however artificial networks are generally built with homogeneous, stationary and deterministic neurons and synapses. It seems difficult to imagine how evolution built a computational machine with such messy components. In this talk I will show that each of these properties can be used to benefit the computations. The non-stationarity of transfer functions can be used as a form of long-short term memory. Surprisingly, for predicting the future state of a high-dimensional but dynamically simple system, a task which is often encountered in an environment, training models with a biologically-inspired nonstationarity outperform parameter-matched RNN and LSTM networks. The heterogeneity can allow a network to better approximate a function with fewer units. However, the most intriguing observation regards the biological noise. While each individual neuron in the brain is highly variable, when observed at a population level, the noise spans a low dimensional manifold. Based on electrophysiological recordings in the mouse visual cortex, I will show that this manifold is aligned in the directions of smooth transforms in the environment, directions which are useful to build an invariance over. I will show that after such an invariance is learned, the noise will help 1-shot learning of new classes. Finally, I will show that such an invariance-aligned noise can be generated in artificial neural networks. Taken together these results paint a picture in which the diverse, constantly changing and often stochastic characteristics of biological neurons, when properly combined, can help networks perform etologically relevant computations.
Stefan Mihalas • Allen Institute for Brain Sciences • (9/17/21)
Abstract:
Computing with a mess: how nonstationary, heterogeneous and noisy components help the brain’s computational power While artificial neural networks have taken inspiration from biological ones, one salient difference exists at the level of components. Biological neurons and synapses have heterogeneous transfer functions, which are non-stationary in time and highly stochastic, however artificial networks are generally built with homogeneous, stationary and deterministic neurons and synapses. It seems difficult to imagine how evolution built a computational machine with such messy components. In this talk I will show that each of these properties can be used to benefit the computations. The non-stationarity of transfer functions can be used as a form of long-short term memory. Surprisingly, for predicting the future state of a high-dimensional but dynamically simple system, a task which is often encountered in an environment, training models with a biologically-inspired nonstationarity outperform parameter-matched RNN and LSTM networks. The heterogeneity can allow a network to better approximate a function with fewer units. However, the most intriguing observation regards the biological noise. While each individual neuron in the brain is highly variable, when observed at a population level, the noise spans a low dimensional manifold. Based on electrophysiological recordings in the mouse visual cortex, I will show that this manifold is aligned in the directions of smooth transforms in the environment, directions which are useful to build an invariance over. I will show that after such an invariance is learned, the noise will help 1-shot learning of new classes. Finally, I will show that such an invariance-aligned noise can be generated in artificial neural networks. Taken together these results paint a picture in which the diverse, constantly changing and often stochastic characteristics of biological neurons, when properly combined, can help networks perform etologically relevant computations.
