Regression performed using Principal Component Analysis

I have a dataset consisting of 10 correlated variables. I need to explain a response variable using these 10 variables, so I am using PCA to reduce dimensionality.

Say, I use the first 3 components from PCA as regressors in Linear regression. Once I have performed Linear reg, I will have 3 beta values associated with three components. How do I split the beta value between 10 varaiables in each PCA component?

• Why do you want to go back to the original 10 variables? – EdM Jun 12 '15 at 19:06
• So, my response variable is sales. And the 10 independent variables which are highly correlated to each other are different kinds of marketing spends. Multicollinearity between the different marketing channels is common because a company usually increases spending simultaneously across different marketing channels. Therefore, there is a need to use PCA to identify a few components. But eventually, I need to associate a beta value with each of the different marketing channels to capture their effectiveness. – Preyas Jun 13 '15 at 7:25

So, my response variable is sales. And the 10 independent variables which are highly correlated to each other are different kinds of marketing spends. Multicollinearity between the different marketing channels is common because a company usually increases spending simultaneously across different marketing channels. Therefore, there is a need to use PCA to identify a few components. But eventually, I need to associate a beta value with each of the different marketing channels to capture their effectiveness

You're not going to be able to principally answer the question from your comment with the data (at the very least, the method) as you described.

You say you want to measure the effectiveness of each channel. While you don't define effectiveness in this context, I would venture to guess it's something like this: "I have one dollar, I'd like to know if it is better to spend it in channel A or channel B?"

This issue is that, according to your qualitative description of your data, you've never measured this situation. You've only measured the situation where you split each dollar across channels, this is one of the reasons you have such high multicollinearity in your predictors. This is going to result in large variance in your estimated parameters. If you waited another year an re-estimated them, it is highly likely that the individual parameters would change, even if the underlying process stayed the same. This means that your determination of which channels are most effective will be very unstable, and sensitive the the exact data you collected instead of to the underlying statistical process.

The parameter estimates for principal components will have the same issue, since that variance is real and has to go somewhere. I would expect your parameter estimates for the principal components to also have high variance.

Setting aside statistical issues for the moment, why not use the principal components directly? Say the first principal component is:

$$.2 C_1 + .5 C_2 + .3 C_3$$

Why not allocate $.2$ of every dollar to channel 1, $.5$ to channel $2$ an $.3$ to channel 3? With more principal components you could weight these determinations with the proportion of the variance explained.

How do I split the beta value between 10 varaiables in each PCA component?

I suppose after all that I owe you a resolution for your original question : )

Let's say you want to keep two principal components in the model, and in terms of the original predictors $C_1, C_2$ and $C_3$ they are

$$PC_1 = .2 C_1 + .5 C_2 + .3 C_3$$ $$PC_2 = .5 C_1 + .1 C_2 + .1 C_3$$

and the estimated model in these components is

$$Y = 2 PC_1 + PC_2$$

Then it's very easy to unwind the model equation to be a function of the original predictors

\begin{align} Y &= 2 PC_1 + PC_2 \\ &= 2 (.2 C_1 + .5 C_2 + .3 C_3) + (.5 C_1 + .1 C_2 + .1 C_3) \\ &= .9 C_1 + 1.1 C_2 + .7 C_3 \end{align}