# Stretched and measured neural predictions of complex network dynamics

2301.04900

0

0

π§

## Abstract

Differential equations are a ubiquitous tool to study dynamics, ranging from physical systems to complex systems, where a large number of agents interact through a graph with non-trivial topological features. Data-driven approximations of differential equations present a promising alternative to traditional methods for uncovering a model of dynamical systems, especially in complex systems that lack explicit first principles. A recently employed machine learning tool for studying dynamics is neural networks, which can be used for data-driven solution finding or discovery of differential equations. Specifically for the latter task, however, deploying deep learning models in unfamiliar settings - such as predicting dynamics in unobserved state space regions or on novel graphs - can lead to spurious results. Focusing on complex systems whose dynamics are described with a system of first-order differential equations coupled through a graph, we show that extending the model's generalizability beyond traditional statistical learning theory limits is feasible. However, achieving this advanced level of generalization requires neural network models to conform to fundamental assumptions about the dynamical model. Additionally, we propose a statistical significance test to assess prediction quality during inference, enabling the identification of a neural network's confidence level in its predictions.

Get summaries of the top AI research delivered straight to your inbox:

## Overview

- Differential equations are widely used to study the dynamics of physical and complex systems
- Data-driven machine learning approaches, such as neural networks, offer an alternative to traditional methods for uncovering models of dynamical systems
- However, deploying deep learning models in unfamiliar settings can lead to inaccurate predictions
- This paper focuses on complex systems described by first-order differential equations coupled through a graph, and explores ways to improve the generalizability of neural network models

## Plain English Explanation

Differential equations are a powerful tool for understanding how things change over time, from the motion of planets to the behavior of crowds. However, traditional methods for modeling these systems can be challenging, especially for complex systems with many interacting parts.

Machine learning offers a promising alternative, as neural networks can be trained on data to uncover the underlying dynamics of a system. These models can be used to predict the future behavior of a system or even discover the differential equations that govern its dynamics.

But there's a catch: when you use these neural networks in unfamiliar situations, like trying to predict what will happen in parts of the system you haven't seen before, the results can be unreliable. This paper tackles that challenge, focusing on complex systems that can be described by a set of interconnected differential equations.

The key insight is that by incorporating fundamental assumptions about the dynamical model into the neural network architecture, you can significantly improve its ability to generalize beyond the training data. This allows the model to make accurate predictions even in regions of the state space or on novel graphs that were not included in the original dataset.

To assess the quality of these predictions, the paper also proposes a statistical significance test that can tell you how confident the neural network is in its forecasts. This helps you distinguish reliable predictions from those that might be spurious.

## Technical Explanation

This paper explores the use of neural networks as a data-driven approach for discovering and modeling the dynamics of complex systems, which are often described by systems of coupled first-order differential equations.

The authors note that while neural networks have been successful in learning dynamical systems and discovering differential equations from data, their performance can degrade when deployed in unfamiliar settings, such as predicting dynamics in unobserved regions of the state space or on novel graph structures.

To address this challenge, the paper proposes a neural network architecture that incorporates fundamental assumptions about the underlying dynamical model, such as the structure of the differential equations and the nature of the interactions between system components. This allows the neural network to better generalize beyond the training data, making accurate predictions in previously unseen scenarios.

The authors also introduce a statistical significance test to quantify the confidence level of the neural network's predictions during inference. This helps distinguish reliable forecasts from potentially spurious results, which is especially important when using these models in safety-critical applications or for exploring the dynamics of complex systems.

The paper presents several experiments demonstrating the effectiveness of their approach, including its ability to accurately predict the behavior of dynamical systems on new graph structures and in unobserved regions of the state space.

## Critical Analysis

The paper presents a promising approach for improving the generalizability of neural networks in modeling the dynamics of complex systems. By incorporating fundamental assumptions about the underlying differential equations and the structure of the interactions within the system, the authors demonstrate that neural networks can be made more robust to unfamiliar settings.

However, the paper does not extensively explore the limitations of this approach. For example, it would be valuable to understand how the method scales to systems with very high-dimensional state spaces or with more complex coupling structures between the differential equations. Additionally, the proposed statistical significance test, while a valuable addition, could benefit from further validation and analysis of its performance in different scenarios.

It would also be interesting to see how this approach compares to other techniques for improving the generalizability of neural networks in dynamical systems, such as stable neural differential equations or Gaussian process methods. A more comprehensive comparison could provide additional insights and guidance for practitioners.

Overall, this paper makes a valuable contribution to the field of data-driven modeling of complex dynamical systems, particularly in the context of neural networks. The proposed approach and the statistical significance test offer a promising step towards more reliable and trustworthy predictions, which will be increasingly important as these models are deployed in real-world applications.

## Conclusion

This paper presents a novel approach for improving the generalizability of neural networks when modeling the dynamics of complex systems described by coupled first-order differential equations. By incorporating fundamental assumptions about the structure of the underlying dynamical model into the neural network architecture, the authors demonstrate that these models can make accurate predictions in unfamiliar settings, such as unobserved regions of the state space or on new graph structures.

The introduction of a statistical significance test also enables the identification of confident predictions, which is crucial for the safe and reliable deployment of these models in real-world applications. While the paper does not fully explore the limitations of the approach, it represents an important step forward in the field of data-driven modeling of complex dynamical systems using neural networks.

This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!

## Related Papers

πΈ

### Learning Governing Equations of Unobserved States in Dynamical Systems

Gevik Grigorian, Sandip V. George, Simon Arridge

0

0

Data-driven modelling and scientific machine learning have been responsible for significant advances in determining suitable models to describe data. Within dynamical systems, neural ordinary differential equations (ODEs), where the system equations are set to be governed by a neural network, have become a popular tool for this challenge in recent years. However, less emphasis has been placed on systems that are only partially-observed. In this work, we employ a hybrid neural ODE structure, where the system equations are governed by a combination of a neural network and domain-specific knowledge, together with symbolic regression (SR), to learn governing equations of partially-observed dynamical systems. We test this approach on two case studies: A 3-dimensional model of the Lotka-Volterra system and a 5-dimensional model of the Lorenz system. We demonstrate that the method is capable of successfully learning the true underlying governing equations of unobserved states within these systems, with robustness to measurement noise.

5/8/2024

π

### Learning Individual Interactions from Population Dynamics with Discrete-Event Simulation Model

Yan Shen, Fan Yang, Mingchen Gao, Wen Dong

0

0

The abundance of data affords researchers to pursue more powerful computational tools to learn the dynamics of complex system, such as neural networks, engineered systems and social networks. Traditional machine learning approaches capture complex system dynamics either with dynamic Bayesian networks and state space models, which is hard to scale because it is non-trivial to prescribe the dynamics with a sparse graph or a system of differential equations; or a deep neural networks, where the distributed representation of the learned dynamics is hard to interpret. In this paper, we will explore the possibility of learning a discrete-event simulation representation of complex system dynamics assuming multivariate normal distribution of the state variables, based on the observation that many complex system dynamics can be decomposed into a sequence of local interactions, which individually change the system state only minimally but in sequence generate complex and diverse dynamics. Our results show that the algorithm can data-efficiently capture complex network dynamics in several fields with meaningful events.

5/16/2024

π§

### Neural Laplace for learning Stochastic Differential Equations

Adrien Carrel

0

0

Neural Laplace is a unified framework for learning diverse classes of differential equations (DE). For different classes of DE, this framework outperforms other approaches relying on neural networks that aim to learn classes of ordinary differential equations (ODE). However, many systems can't be modelled using ODEs. Stochastic differential equations (SDE) are the mathematical tool of choice when modelling spatiotemporal DE dynamics under the influence of randomness. In this work, we review the potential applications of Neural Laplace to learn diverse classes of SDE, both from a theoretical and a practical point of view.

6/10/2024

### Dynamical stability and chaos in artificial neural network trajectories along training

Kaloyan Danovski, Miguel C. Soriano, Lucas Lacasa

0

0

The process of training an artificial neural network involves iteratively adapting its parameters so as to minimize the error of the network's prediction, when confronted with a learning task. This iterative change can be naturally interpreted as a trajectory in network space -- a time series of networks -- and thus the training algorithm (e.g. gradient descent optimization of a suitable loss function) can be interpreted as a dynamical system in graph space. In order to illustrate this interpretation, here we study the dynamical properties of this process by analyzing through this lens the network trajectories of a shallow neural network, and its evolution through learning a simple classification task. We systematically consider different ranges of the learning rate and explore both the dynamical and orbital stability of the resulting network trajectories, finding hints of regular and chaotic behavior depending on the learning rate regime. Our findings are put in contrast to common wisdom on convergence properties of neural networks and dynamical systems theory. This work also contributes to the cross-fertilization of ideas between dynamical systems theory, network theory and machine learning

4/10/2024