Machine learning models typically need gigantic data sets and a lot of energy, whereas the brain consumes as much power as a single light bulb. Aalto’s new assistant professor uses neuroscience to make computer programs more efficient.
Stéphane Deny started as a new assistant professor in the beginning of December. Image: Mikko Raskinen / Aalto University
What do you research and why?
In the last decade, we have seen a big push in machine learning that has yielded a lot of impressive results in domains like visual recognition and language understanding. This push has been mostly driven by the industry, and basically with two engines: a lot of data and a lot of compute. It has not really been inspired by how the brain works.
I think this is a missed opportunity. The brain is much more efficient than the current systems used in machine learning: It requires much less data than current models that are trained on gigantic corpuses of text or images, and it needs much less energy. The brain consumes the same amount of power as a light bulb — just 20 watts — to function, and that is much more efficient than the forms of GPUs and algorithms that are currently used in any machine learning application. So I think there is still a lot to learn from the brain, and this is why I work at this interface.
How did you become a professor?
I was really intrigued by this complexity, and wanted to understand it from a more theoretical point-of-view. This is how I decided to move to a lab in Stanford, which was on the theoretical side of neuroscience. After this experience, I moved to do a postdoc at Facebook AI. There, I spent two years developing machine learning tools inspired by the brain. This led me to apply to professorships, and it is how I landed here at Aalto.
What is the high point of your career?
I had a collaboration at Stanford with a group of physicists, with whom we came up with an explanation of that structure with a simple principle of efficiency: If the goal of the retina is to send the visual information to the brain in the most efficient way, then it should be sent through these different pathways, with these different pre-processings. Some of our predictions were precisely aligned with the processing done by the primate retina. Here, then, we could explain, from first principles, a complex structure that was found in the brain. It was very exciting to me.
And then, this same principle turned out to be also useful in machine learning. The principle of efficiency behind our discovery in the retina has been described in the 1960s as the redundancy-reduction principle, and it tells you that each neuron should carry information as independently of other neurons as possible. This principle has been very useful for understanding the structure of the visual system, but it was largely ignored in machine learning. When I joined Facebook, one of my projects was to develop a model that was using this principle. That work resulted in a model for visual recognition without supervision which was state-of-the-art at the time, a year ago.