Title: Scale Equivariant Graph Metanetworks

Speaker: Giannis Kalogeropoulos, Ph.D. student of Department of Informatics and Telecommunications of the National and Kapodistrian University of Athens (NKUA)

Abstract: We introduce a graph metanetwork framework that allows scaling and permutation equivariant neural network processing.

This paper pertains to an emerging machine learning paradigm: learning higher- order functions, i.e. functions whose inputs are functions themselves, particularly when these inputs are Neural Networks (NNs). With the growing interest in architectures that process NNs, a recurring design principle has permeated the field: adhering to the permutation symmetries arising from the connectionist structure of NNs. However, are these the sole symmetries present in NN parameterizations? Zooming into most practical activation functions (e.g. sine, ReLU, tanh) answers this question negatively and gives rise to intriguing new symmetries, which we collectively refer to as scaling symmetries, that is, non-zero scalar multiplications and divisions of weights and biases. In this work, we propose Scale Equivariant Graph MetaNetworks - ScaleGMNs, a framework that adapts the Graph Metanetwork (message-passing) paradigm by incorporating scaling symmetries and thus rendering neuron and edge representations equivariant to valid scalings. We introduce novel building blocks, of independent technical interest, that allow for equivariance or invariance with respect to individual scalar multipliers or their product and use them in all components of ScaleGMN. Furthermore, we prove that, under certain expressivity conditions, ScaleGMN can simulate the forward and backward pass of any input feedforward neural network. Experimental results demonstrate that our method advances the state-of-the-art performance for several datasets and activation functions, highlighting the power of scaling symmetries as an inductive bias for NN processing. The source code is publicly available at https://github.com/jkalogero/scalegmn.

Short Bio: Giannis holds an MEng in Electrical and Computer Engineering from the National Technical University of Athens, focusing on Computer Science. During his studies, he studied a lot of state-of-the-art deep learning techniques and cultivated a strong interest in Geometric Deep Learning. While working on his Diploma Thesis, he employed Graph Neural Networks and incorporated external knowledge for the multimodal task of Visual Dialog.

He has professional and research experience in studying and implementing machine learning models in a wide range of fields. Specifically, as a Machine Learning Engineer, he has worked on Natural Language Processing and Time Series Forecasting problems, employing, among others, pre-trained Language Models. Moreover, he has dove deeper into the MLOps techniques for orchestrating the whole lifecycle of an ML model. Finally, he has published and presented at an IEEE conference his work on Machine Learning and Edge Computing.

Room: Pythagoras, Archimedes Unit (1 Artemidos str., ART1 building, ground floor)

and virtually via Zoom Meetings - https://zoom.us/j/97800909461?pwd=hZN57oYaZnqG75U2OZYkK9vC0M0Tdj.1

• Meeting ID: 978 0090 9461

• Passcode: oIuc3O

Check for additional ways to participate/dial in at the bottom of the announcement.

Video: https://neurips.cc/virtual/2024/oral/97993

Stay tuned!

For ways to receive news about the NLP Group and its meetings, as well as to get check the latest information about the meetings of Archimedes NLP Theme and AUEB NLP Group, check http://nlp.cs.aueb.gr/news.html. To subscribe to the mailing list of AUEB NLP Group, send a message with subject "subscribe nodigest" to This email address is being protected from spambots. You need JavaScript enabled to view it.. If you have an AUEB account and want to receive announcements about AUEB NLP Group Meetings through MS Teams, subscribe to "AUEB NLP Group meetings" group on MS Teams (code: 01j65ny). Team members can also send text messages (chat) to other team members.

If you are an AI researcher or practitioner, please consider becoming a member of the Hellenic Artificial Intelligence Society (EETN, http://www.eetn.gr/en/).

Dial-in information:

One tap mobile

+302311180599,,97800909461#,,,,*314861# Greece

+302111984488,,97800909461#,,,,*314861# Greece

Dial by your location

• +30 231 118 0599 Greece

• +30 211 198 4488 Greece

Meeting ID: 978 0090 9461

Passcode: 314861

Find your local number: https://zoom.us/u/ab4yJClOiz