Neural Network Architectures and Training Strategies for Solving Ordinary Differential Equations

This article discusses various neural network architectures used for approximation and optimization, focusing on their advantages and limitations. The authors highlight two main types of neural networks: discrete and continuous networks, with each having its strengths and weaknesses. Discrete networks are simpler to implement but restrict the model’s capacity, while continuous networks offer more flexibility but require careful parameterization.
The article then delves into the total accuracy error of a neural network model, which can be broken down into three components: approximation error, optimisation error, and generalisation error. To achieve excellent agreement between predicted and reference trajectories, it is crucial to select the appropriate architecture and fine-tune the model hyperparameters. The authors demonstrate that they can construct a network expressive enough to provide a small approximation error and excellent generalization capability.
The article also introduces the concept of activation functions, which are continuous nonlinear scalar functions that act component-wise on vectors. The architecture of the neural network determines the space of functions F = {fρ : I → O, ρ ∈ Ψ} that can be represented, with the weights chosen to approximate accurately a map of interest f : I → O.
In supervised learning, the authors emphasize the importance of minimising a purposefully designed loss function Loss(ρ) to optimize the weights ρ. Overall, this article provides a comprehensive overview of neural network architectures and their applications in approximation and optimization, making complex concepts more accessible and easier to understand for readers.