I am new to Statistics and I have a Mathematics background. In Statistics, particularly in Linear Regression and Principal Component Analysis (PCA) so far what I have understood is that the main idea is to try to capture as much as possible variability present in the data. In linear regression, while calculating $ R^{2} (R \ squared)$ measure we are checking the proportion of variability captured by our model and in PCA we are forming a new basis along which our data has the maximum possible variability. Is there any significant result behind this logic? I mean why we have to go after variability? Any help in this matter will be appreciated.


3 Answers 3


Statistics is the interface between math (models of the world) and our perception of reality. I suspect what you are looking for is not proof, but an understanding of the assumptions.

Math proofs are a formal logic system that works because it is self contained (in my background as a chemist this would be termed a adiabetic). All proofs rest on assumptions, and incompleteness theorems show that a system cannot prove its own consistency nor prove every statement true.

Data is percieved information about the world (even if technology has captured it). The underlying data generating processes are many and complex and no real world physical system is close to adiabetic,allowing external influences to perturb the system being investigated. Quantum theory tells us that we can never know every physical detail of a system perfectly.

There are uncertainties on both sides of the equation.

The question stats attempts to answer is what can data tell us about the model we have, or vice versa what our model can tell us about our data. The two won't match, so what we are interested in is how much they don't match, i.e. How much does our data vary outside the constraints of our model.

A popular saying on this site is that all models are wrong, but some are useful. Measuring variance explained allows us to assess one aspect of this usefulness, but it is far from the only one. The metric employed should be appropriate to the questions being asked.

So some basic assumptions in assessing variance (I'm sure it will be incomplete, so feel free to comment)

  1. The model is not perfect but explains a maximal proportion of observed phenomena
  2. The data is not pure, it contains noise and biases that are unrelated to the model
  3. We need a model that explains as much of the data generating process as possible
  4. We need a model able to ignore the noise
  5. Processes external to the system under investigation have negligible influence.

Tools exist for assessing the validity of these assumptions, which is why stats is so complicated, but can reveal so much.

It is important to understand the purpose of statistics (something commonly misunderstood by both mathematicians and scientists). The point of statistics is not proof or truth, it is assessing risk.

  • $\begingroup$ Wonderful answer!! I need a little bit more time to assess the aspects described here with my little knowledge ( i would rather say I only have some information not knowledge until now) of Statistics. Thanks ☺️ $\endgroup$
    – Satish
    Apr 20, 2019 at 18:55
  • $\begingroup$ It is a very short concise answer to a sprawling issue, which is how do we align the philosophies of science and maths. I'll do my best to develop the answer to any follow up. $\endgroup$
    – ReneBt
    Apr 21, 2019 at 8:55

In many cases the reason we use regression is to explain variability. In that sense, how much variability is explained is one of the key measures of success.

This may be more clear with an example. I recently worked on a project where we created a regression model to explain employee performance. We did this because our stakeholders (senior management) wanted to know why some employees were performing well and others weren't. That is, why do we see variance in employee performance?

Phrased this way it should be clear that a key performance metric for our model is how much variability it correctly anticipates.

  • $\begingroup$ I am totally convinced with the logic that we are trying to capture variability. But my question is why variability? As a Mathematics student, it's hard to accept it without any formal proof. @indigochild as we see in your model it was a good measure but why I should accept it in general? $\endgroup$
    – Satish
    Apr 20, 2019 at 3:08
  • $\begingroup$ @Satish Maybe someone else can provide a more satisfying answer. At its heart, this isn't an issue of mathematics. The answer lies in the informal logic surrounding how users interpret it. It isn't subject to any kind of formal proof. $\endgroup$ Apr 20, 2019 at 3:33

Here's my few cents..

Co-movement of independent and dependent variable is the key here. Let's say we want to find out how height changes with age and we have data for 100 people. Let's say we know that our independent variable (height) varies a lot across the 100 observations, but we want to find out how much of it comes from the co-movement of height and age. Hence, we fit a model and to estimate of how much of the variance in height can be explained from co-movement w.r.t. age.

If in our data, everyone has the same age, the model won't be able to explain ANY of the height's variance, we'll need to find something that explains the movement (variance) of the independent variable. Explaining the movement (variance) of independent variable is a good starting point for all predictive models.

In PCA, the objective is rotate the data to get the best axis for cleanest perspective. Using variance to change the basis is just a way to get this perspective on how data is scattered on a hyperplane.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.