Workshop: Theory towards Brains, Machines and MindsWorkshop: Theory towards Brains, Machines and Minds

Title: Continuous Model of Deep Neural Networks

Sho Sonoda, RIKEN Center for Advanced Intelligence Project

What a neural network has learned after the training is less interpretable, because it has millions of trainable parameters, or neurons. Thus, a neural network is often compared to a black-box. The speaker aims to turn the black-box into a white-box. Namely, he aims to interpret what neural networks have learned, and to develop a more interpretable learning machine. One approach to tackling a large number of parameters is continuous modeling. In this talk, the speaker introduces two types of continuous models: integral representation and flow representation. The integral representation corresponds to a continuous limit of the width, or the number of hidden units, and helps to “convexify” the network. On the other hand, the flow representation corresponds to a continuous limit of the depth, or the number of hidden layers, and helps to understand the network by recasting it as a gradient flow.

[1] S.Sonoda, "Unitary Kernel Quadrature for Training Parameter Distributions", arXiv:1902.00648, (2019).
[2] S.Sonoda, I.Ishikawa, M.Ikeda, K.Hagihara, Y.Sawano, T.Matsubara, N.Murata, "The global optimum of shallow neural network is attained by ridgelet transform", arXiv:1805.07517, (2018).
[3] S.Sonoda, N.Murata, "Transport Analysis of Infinitely Deep Neural Network", Journal of Machine Learning Research, 20(2):1-52, (2019).
[4] S.Sonoda, N.Murata, "Neural Network with Unbounded Activation Functions is Universal Approximator", Applied and Computational Harmonic Analysis, 43(2):233-268, (2017).