Fully Connected Committee Machines: A Comprehensive Analysis

In this article, we explore the capacity of neural networks to store information and understand how it is affected by various factors such as the number of hidden layers, the size of each layer, and the activation function used. We present several results that shed light on these questions and demystify some long-standing assumptions in the field.
Firstly, we define the capacity of a neural network as the amount of information it can store. We show that this capacity is determined by the number of hidden layers and the size of each layer. Specifically, we find that the capacity increases linearly with the number of hidden layers and quadratically with the size of each layer.
Next, we investigate how activation functions affect the capacity of a neural network. We consider several common activation functions, including sigmoid, tanh, and ReLU, and show that they all have a similar impact on the capacity. Specifically, we find that the capacity increases as the activation function becomes more nonlinear.
We then explore how the size of the input data affects the capacity of a neural network. We show that the capacity grows exponentially with the size of the input data, which means that larger datasets can store more information.
Finally, we discuss some implications of our results for deep learning research. In particular, we highlight the importance of choosing an appropriate activation function and the need to consider the capacity of a neural network when training it. We also suggest several directions for future research, including investigating how other factors such as regularization and optimization techniques affect the capacity of a neural network.
In summary, our article provides new insights into the capacity of neural networks and its relationship with various factors such as the number of hidden layers, activation functions, and input size. We demystify some long-standing assumptions in the field and highlight the importance of choosing an appropriate activation function when training a deep neural network. Our findings have important implications for deep learning research and suggest several directions for future work.