# learning an hierarchical linear model - overfitting / identifiability issues?

I'm currently looking at a paper where they've done this using two layer support vector regression and I'm trying to figure out whether they have biased the performance of their classifier and whether there is a better way of doing this. I have a sequence of bins and within each of these bins I have counts of specific features (all bins have the same set of features) and I have a number $Y$ associated with that sequence which I am trying to predict.

So following the paper - for each bin I can learn a model which predicts Y_{i} (predicting of Y using information from bin i) ie $Y_{i} = \beta_{i} X{i} + \epsilon$. Now given the prediction of each of these classifiers, in the paper they then use these to learn another linear model to predict $Y$.

They use the same data to learn both the $\beta$s for the individual bins and the coefficients for the second model.

My question is this....is there an overfitting problem here - essentially using the same training data twice? Can the parameters of the model be learned reliably - ie can we actually learn the coefficients of both "levels" of the model at once......? Or is there a better way of formulating such a model?

If this way of explaining doesnt make sense, let me know and I'll try to make a diagram/explain it more. Thanks

-
Sorry, I don't think I got it: the first model is $Y_i = \beta_i X_i + \epsilon$? So for every sample $X_i$ (you call them bins) we have different coefficients $\beta_i$? I don't think it's the case. And even after that, am I right that the second model is $Y_i = \beta \hat{Y}_i + \epsilon$, where $\hat{Y}$ is an output of the first predictor? –  Dmitry Laptev Jun 7 '12 at 22:32
sorry $\beta_{i}$ is the vector of coefficients for the model trained on bin $i$. So I kind of thing of this as giving the relative importance of the the features in bin $i$ for predicting $Y_{i}$ .... so using $\hat{Y}$ as the matrix of predictions from the first model the second model would be $Y = \alpha\hat{Y} + \epsilon$ - the alphas corresponding to the importance of each of the bins. Really my problem with their model is that really $Y_{i}$ and $Y$ are the same. Which makes me think that the model is going to be overfitted –  Nathan Harmston Jun 8 '12 at 7:46