Why we try to capture variability?

Question

I am new to Statistics and I have a Mathematics background. In Statistics, particularly in Linear Regression and Principal Component Analysis (PCA) so far what I have understood is that the main idea is to try to capture as much as possible variability present in the data. In linear regression, while calculating $ R^{2} (R \ squared)$ measure we are checking the proportion of variability captured by our model and in PCA we are forming a new basis along which our data has the maximum possible variability. Is there any significant result behind this logic? I mean why we have to go after variability? Any help in this matter will be appreciated.

Answer 1

Statistics is the interface between math (models of the world) and our perception of reality. I suspect what you are looking for is not proof, but an understanding of the assumptions.

Math proofs are a formal logic system that works because it is self contained (in my background as a chemist this would be termed a adiabetic). All proofs rest on assumptions, and incompleteness theorems show that a system cannot prove its own consistency nor prove every statement true.

Data is percieved information about the world (even if technology has captured it). The underlying data generating processes are many and complex and no real world physical system is close to adiabetic,allowing external influences to perturb the system being investigated. Quantum theory tells us that we can never know every physical detail of a system perfectly.

There are uncertainties on both sides of the equation.

The question stats attempts to answer is what can data tell us about the model we have, or vice versa what our model can tell us about our data. The two won't match, so what we are interested in is how much they don't match, i.e. How much does our data vary outside the constraints of our model.

A popular saying on this site is that all models are wrong, but some are useful. Measuring variance explained allows us to assess one aspect of this usefulness, but it is far from the only one. The metric employed should be appropriate to the questions being asked.

So some basic assumptions in assessing variance (I'm sure it will be incomplete, so feel free to comment)

1. The model is not perfect but explains a maximal proportion of observed phenomena
2. The data is not pure, it contains noise and biases that are unrelated to the model
3. We need a model that explains as much of the data generating process as possible
4. We need a model able to ignore the noise
5. Processes external to the system under investigation have negligible influence.

Tools exist for assessing the validity of these assumptions, which is why stats is so complicated, but can reveal so much.

It is important to understand the purpose of statistics (something commonly misunderstood by both mathematicians and scientists). The point of statistics is not proof or truth, it is assessing risk.

Answer 2

In many cases the reason we use regression is to explain variability. In that sense, how much variability is explained is one of the key measures of success.

This may be more clear with an example. I recently worked on a project where we created a regression model to explain employee performance. We did this because our stakeholders (senior management) wanted to know why some employees were performing well and others weren't. That is, why do we see variance in employee performance?

Phrased this way it should be clear that a key performance metric for our model is how much variability it correctly anticipates.

Answer 3

Here's my few cents..

Co-movement of independent and dependent variable is the key here. Let's say we want to find out how height changes with age and we have data for 100 people. Let's say we know that our independent variable (height) varies a lot across the 100 observations, but we want to find out how much of it comes from the co-movement of height and age. Hence, we fit a model and to estimate of how much of the variance in height can be explained from co-movement w.r.t. age.

If in our data, everyone has the same age, the model won't be able to explain ANY of the height's variance, we'll need to find something that explains the movement (variance) of the independent variable. Explaining the movement (variance) of independent variable is a good starting point for all predictive models.

In PCA, the objective is rotate the data to get the best axis for cleanest perspective. Using variance to change the basis is just a way to get this perspective on how data is scattered on a hyperplane.