Neural networks are machine learning instruments capable of solving a vast array of problems ranging from image classification to decision making. Developers of neural networks algorithms have many consequential decisions they must make in order to adapt a model to a particular task. Recurrent neural networks (RNNs) are an important instance among these algorithms — they are popular and powerful due to their ability to integrate information over time. A structured understanding of how RNNs are affected by the decisions made during model adaptation will improve RNN performance. Unlike conventional feed-forward neural networks, RNNs are allowed to evolve over time. The study of time-dependent systems is known mathematically as dynamical systems analysis. Hence, we can build upon existing machine learning literature by incorporating a dynamical systems perspective to describe an RNNs regime. Through this lens we investigate how an RNN which operates in different dynamical regimes (chaotic or stable) displays optimal temporal integration abilities. We establish a relationship between a mathematical metric for the dynamical regime and the qualitative solution for the network. Our preliminary results further suggest improved performance when the RNN is encouraged to operate in a chaotic dynamical regime.

Causal inference is a large and long-standing field that attempts to answer a very fundamental question: does X cause Y? This question arises in many fields such as healthcare, genomics, biology, and econometrics. The complex systems analyzed in these fields often have many interacting components mathematically represented as nodes and edges (connectivity) of a network. Network inference is the study of the time-dependent behavior of these nodes to reverse-engineer the network connectivity. The theory of causal inference has many contending definitions of causality such as state-of-the-art techniques Granger Causality (GC) and Convergent Cross Mapping (CCM). These algorithms are computationally tractable and easy to use but require strong mathematical assumptions. We simulate networks of harmonic and Kuramoto oscillators and attempt to reconstruct their ground-truth network structure using observations of oscillator displacements over time. Our analysis investigates the performance of inference methods on Erdos-Renyi and scale-free random graphs. We show that the GC and CCM inference methods systematically fail to determine network structure by returning overly sparse or dense connectivity results. These findings challenge the applications of such top-down inference approaches to physical and biological systems. Using a few basic assumptions, we demonstrate how networked systems of coupled oscillators can be successfully reconstructed if perturbations of the system are allowed. We propose a Perturbation Causal Inference (PCI) algorithm that uses systematic perturbations and tracks how perturbation cascades spread through the network. Using changepoint detection, correlation, and windowed variance statistics, we predict causal relationships between nodes in the graph. Our analysis shows that PCI works at scale and efficiently returns high accuracy reconstructions of large networks with varying coupling strengths and connectivity structures. We conclude by proposing future applications of perturbation inference methods into neuroscience and make a connection with Hebbian learning rules.

Dynamical systems are ubiquitous in science and engineering. Inferring the mathematical laws that govern dynamical systems typically requires a 'scientist-in-the-loop' to guide the discovery process, via their expert knowledge and intuition about the system. Getting computers to perform this task automatically, without the guidance of a domain expert in the loop, is a grand challenge in the field of data science. A number of algorithms have been developed to infer such laws. One leading algorithm is Sparse Identification of Nonlinear Dynamics (SINDy), which applies simple linear regression coupled with sparsification to optimize a model over a large library of candidate functions. This algorithm is purely data-driven and makes no use of information that may be known previously about the dynamical system, such as symmetries and conservation laws. In this work, we develop a framework for incorporating and enforcing symmetries and conservation laws in SINDy so that the inferred models are consistent with prior domain knowledge. We analytically show how to propagate symmetries and conservation laws through the SINDy function library, and from this analytically derive linear constraints on the resultant linear regression. These constraints can be incorporated into the regression problem using options available in standard quadratic optimization packages. We implement this method and show that it provides improved accuracy and robustness on the task of inferring several canonical dynamical systems.