Is there a clear intersection of chaos theory and machine learning?

Question

There was a chaos theory related question on data mining here: What are the practical applications of chaos theory in data mining?, but it was deemed too broad. I'm going to try to tackle this topic in a more focused and terse way.

Premise

Models rooted in chaos theory have helped scientists understand complex systems. Some cool examples are: robotics, cryptography, and bird migrations. One of the chief characteristics is the sensitivity to initial conditions. While I am not an expert in either field, but I see certain similarities between the two. For instance, certain machine learning techniques like cluster analysis also have initial conditions (the initial cluster centers). Chaos theory and machine learning also are used for predictive analysis.

I concede it's not the most elegant comparison; there are many differences between the two. Now don't quote me on this, but I think of chaos theory as a kind of blue-print for modeling complex systems, and I think of machine learning as a tool for optimizing and best utilizing high dimensional data. In machine learning the model is only part of the ML block diagram. It could be a regression model, clustering model or something else entirely. So again, it's not fair to compare them in all aspects. With this disclaimer out of the way, here is my question:

Question

Is there a particular field of machine learning that is devoted to applying chaos theory models, or are researchers merely cherry-picking a few models from chaos theory? Or do machine learning researchers stand to gain little from chaos theory since they can go with atheoretical approaches and toss theory out the window and just use a multitude of features?

Answer 1

In my understanding (though I wouldn't be surprised to be challenged on this), machine learning and statistics tackle partially similar problems, but machine learning focuses on the specific problem of prediction, and machine learning methods are most often not based on a model of the data-generating process. Secondly, statistics focuses on data-generating processes that involve randomness, while "Chaos Theory" a.k.a. nonlinear dynamics focuses on deterministic processes. Therefore, machine learning is two steps removed from nonlinear dynamics or chaos.

There is a weak relationship between machine learning and the phenomenon of chaos since both are about prediction, or rather predictability in the second case. However, chaos is about limits of predictability due to insufficient knowledge of initial conditions even though there is a perfect model of the process, while machine learning is about the practical problem of actually predicting without caring or knowing much about the underlying process.

There is also a link between chaos and statistics insofar as it can be shown that specific chaotic systems can be mapped onto random processes. The basic idea is that chaotic dynamics amplifies differences in states, which means over time more and more details of the initial conditions come to matter. If the not-infinitely-precise knowledge of initial conditions is conceptualized as random, that means the large-scale output of a chaotic system can be considered random. For more details see e.g. here. However, nonlinear dynamics tends to focus on low-dimensional dynamical systems and their even lower-dimensional attractors, while randomness in many real world situations handled by statistics and machine learning has not to do with not knowing the 100th digit of a few initial conditions, but with not knowing anything of the state of very high-dimensional influences.

I hope this helps clarify matters.

Answer 2

Chaos theory studies dynamic deterministic systems that are highly sensitive to initial conditions, such that small changes in the initial conditions lead to large "chaotic" changes in outputs. The result is a deterministic system that fully determined by its initial conditions, but is unpredictable over the long-term by any approximating method. This differs substantially from the subject of statistics and machine learning, since the latter are concerned with stochastic systems that are generally not highly sensitive to initial conditions.

While chaos theory is not strongly related to statistics and machine learning, there are some areas of overlap. The main application of this kind of system in statistics and machine learning is in the creation of deterministic algorithms that generate pseudo-random numbers. These algorithms are designed to be chaotic deterministic systems that are highly sensitive to initial inputs, such that they generate "chaotic" outputs that can be treated as effectively random. More generally, it is possible to look at chaotic deterministic systems and analyse them via the methods of statistics, which in effect ignores the deterministic dynamics of the system and treats it as a stochastic process. This can yield an understanding of the distribution of outputs that come out of a chaotic system under a particular input distribution.