How well does a Linear Model fit a subset of the parent dataset?

A linear regression model has been created based on a dataset with observations which can be sorted into several different categories.

I have been asked to assess how well this regression model (which was created using the entire dataset) fits the observations by the category subsets. Correct me if needed, but this seems to render R-squared useless - (1) because the means of predictions and observations inside a specific category are not equal (sum of $\hat{Y_i}$ does not equal sum of $Y_i$), and the relationship that SSE + SSR = SSTotal no longer holds within each category.

Calculating the correlation coefficient between the overall model's fits for a single category, and those category's observations, would be equivalent to fitting a new least-squares model specific to that category and calculating an R-squared value, but that would not be assessing the fit of the OVERALL model to that category. Does that make sense?

What other options would you recommend here? Would some kind of likelihood ratio test perhaps be appropriate? I confess I'm not sure what else to do.

You can compute an $R^2$ value on subsets of your dataset. This will give you a measure of fit (let's call it $R_c^2$) for each category.

Here is how to do it:

Let $S_c$ be a set containing the indices of all your data points that belong to category $c$. Then you can compute the corresponding $R_c^2$ for this set:

$R_c^2 = 1 - \frac{RSS_c}{TSS_c}$

where you compute the residual sum of squares $RSS_c$ and the total sum of squares $TSS_c$ on data points that belong to category $c$. In formulas:

$RSS_c = \Sigma_{i \in S_c} (ŷ_i - y_i)^2$

$TSS_c = \Sigma_{i \in S_c} (y_i - \overline y_S)^2$.

Notation:

$y_i$ is the true value for data point $i$.

$ŷ_i$ is your prediction for data point $i$ (using the full model).

$\overline y_S = \frac{1}{|S_c|} \Sigma_{i \in S_c} y_i$ is the mean over all true value for $y$ in $S_c$.

• This approach makes sense but it seems to me it could be problematic, as illustrated in my post at stats.stackexchange.com/a/13317/919. The issue lies in what the intended meaning of "how well ... fits" is. The use of $R^2$ masks any changes in the actual sizes of residuals within each category, potentially eliminating the most important information of all. It is possible for responses in one category to be beautifully fit (relative to other categories) but for the $R_c^2$ to be zero. – whuber Jul 31 '18 at 11:35