Data Analytics and Prediction I have a data analytics maths question. I would like to know what approach to take to solve such a problem, what is the best model to use etc. and how to go about doing it.
I have past historical data of variable $X$ and variable $Y$ with respect to time. I know that variable $Y$ somehow is affected by variable $X$. That is, if  variable $X$ increases, I know variable $Y$ should increase.
However, my friend is suspecting that although variable $X$ has increased, variable $Y$ should not have increased to such a large extent on some of the occasions. He claims that on some occasions, the increase in variable $Y$ is abnormal. He claims that an increase in variable $X$ should have increased variable $Y$ but to a lesser extent on some occasions, hence claiming there might be other factors at play which have caused variable $Y$ to increase on those particular occasions, and not solely on the increase in variable $X$.
My question I am trying to figure out is mainly:

*

*How do I figure out whether an increase in variable $X$ has indeed caused variable $Y$ to increase by such a large extent?


*How do I know whether on those occasions, when both $X$ and $Y$ increased, is it only $X$ which caused $Y$ to increase and not some other unknown factors?


*How can I predict in the future how much will $Y$ be increased if $X$ is increased? This will help me to determine whether the increase in $Y$ is actually solely due to the increase in $X$ or due to other factors too.
I would like to know which data analytics model would best be used to solve this, and how I shall go about solving this problem.
Thanks.
 A: The questions you are posing may need quite long answers. I will try to summarize my opinions.
1. Your question concerns causality. That is really a deep concept to use in data analysis. Keep in mind that empirically observed covariation is a necessary but not sufficient condition for causality (The Cognitive Style of PowerPoint, of Edward Tufte). That being said, in order to ask causal question we should first define causality. A standard approach relies on the potential outcomes model (Neyman, 1923; Rubin, 1974), where we observe outcomes $Y$, and some treatment $X$ which is suspected to affecting $Y$. For simplicity, let us assume $X \in {0, 1}$. Then, we postulate the existence of counterfactuals $Y(X = 0)$ and $Y(X = 1)$, which denote the outcomes a given unit would experience under the possible values of $X$. The causal effect of $X$ on $Y$ is then defined as $\tau = Y(X = 1) - Y(X = 0)$. Without this framework (or alternative models that define some concept of causality), we would just observe correlations among variables. The difference is that if $X$ and $Y$ are simply correlated, then nothing can be done to affect $Y$ through $X$, while if $X$ actually causes $Y$, then we could exogenously vary $X$ so to modify $Y$. So, to answer your question, you should implement any estimator tailored for causality. I suggest to have a look at the book of Imbens and Rubin (2015), which really is a bible for causal analysis.
2. This is a nice question. Such unobserved factors really invalidate causal analysis when they are correlated with either $X$ or $Y$. If this holds, then we risk to attribute to $X$ some causal effect on $Y$ (as defined above) which is actually due to such unobserved factors (called confounders). What we should do is to include these variables in the analysis, so to control for their influence. The main issue is that they are often unobservable.
3. I do not think that prediction may help you understanding how much $X$ causes $Y$ - only about how much they are correlated, for the reasons described above. Anyway, this question seems to shift the focus from causality to prediction. For that, you could apply any machine learning predictor yielding a satisfying test error, thus using the estimated function to predict future outcomes by plugging $X$ values - formally, we estimate $\hat{Y} = \hat{f} ( \cdot )$, and we predict a future outcome at time $t + h$ as $Y_{t + h} = \hat{f} ( X_{h + t} )$, where $\hat{f} ( \cdot )$ has been estimated using $X_1, \dots, X_t$.
