[Archimedes Seminar Series] Are activation functions required for learning in all deep networks?

Abstract:Activation functions play a pivotal role in deep neural networks, enabling them to tackle complex tasks like image recognition. However, activation functions also introduce significant challenges for deep learning theory, network dynamics analysis, and properties such as interpretability and privacy preservation. In this talk, we revisit the necessity of activation functions across various scenarios. Specifically, we explore expressing network outputs through high-order interactions among input elements using multilinear algebra. This approach allows us to attain the necessary expressivity via these high-order interactions. Yet, the question remains: Is this expressivity alone sufficient for effective learning? Our recent research, presented at ICLR’24, unveils meticulously designed networks, which achieve strong performance even in demanding tasks, such as ImageNet image recognition. 

BioGrigoriosChrysosis currently an Assistant Professor at University of Wisconsin-Madison, USA. He obtained hismaster’s degree in Electrical, Electronic Engineering and Computer Science from the National Technical University of Athens in 2014. Sequentially, he pursued a PhD degree at Imperial College London. He was a post-doctoral researcher with the Laboratory for Information and Inference Systems at the École PolytechniqueFédéralede Lausanne (EPFL), Switzerland