Opening the black box: Machine learning for Next-Level Animal Science

Deep learning of artificial neural networks (ANNs) is currently revolutionizing many aspects of our society from self-driving cars to computer-generated art and even complex conversations. While originally inspired by the brain, ANNs have independently evolved driven by technical applications and differ from their biological analogue in many structural as well as functional characteristics. For example, biological neural networks are generally more versatile and not constrained to only one specific task. This project aims to understand these differences and explores the relationship between structure and function to gain a better mechanistic understanding of brains and ANNs alike – both are still merely a black box today.

By carrying out numerical experiments with simple ANNs, the project investigates to what extent certain properties of the brain can be reproduced in-silico and which ingredients might be missing. Although ANNs are a crude approximation of the brain, they share important functional similarities - an objective function, a network architecture and a learning rule. This conceptual approach using ANNs as minimal brain models does not attempt to accurately simulate a biological brain. Instead, it aims to reveal general, yet biologically relevant principles about the biophysics of learning. The project currently focusses on three aspects: multitask learning, neurogenesis, and network sparsity.

Typically, ANNs are trained to perform a single task and learning further tasks generates conflicts. Learning a new task might overwrite a previously acquired one, which is known as catastrophic forgetting. In striking contrast, even the simplest animal can learn multiple tasks. The researchers find that repetitively alternating between tasks during learning can allow the network to accommodate multiple tasks. Interestingly, such multi-task learning generates an emergent partitioning of the network, where different sub-tasks are stored in different nodes.

Biological brains process information and learn new tasks while their neural networks are changing, both in size and topology. How does the addition of new nodes during training affect network stability and learning efficiency? The researchers assess the performance of growing networks by numerical experiments as well as analytical approaches. They find that the addition of nodes during training can stabilise the network, allowing for an increased learning rate.

While many ANNs are densely connected such that any node from one layer can be linked to any node from another layer, biological neural networks might be rather sparsely connected. Sparsely connected networks can be computationally more efficient but how to best train them and how to impose sparsity is still poorly understood. The researchers identify several transitions in network performance as a function of sparsity.

Progress (August 2023)

Using numerical experiments, the researchers have revealed multiple interesting features of multi-task learning, neurogenesis and network sparsity as described above. They are currently developing analytical frameworks to explain the underlying mechanisms and compile a story for a manuscript as well as a documentation for other groups to use their tools.