8
$\begingroup$

I have a linear regression problem. In short, I have a dataset, I divided it into two subsets. One subset is used to find the linear regression (training subset), another is used to evaluate it (evaluation subset). My question is how to evaluate the result of this linear regression after applying it to the evaluation subset of data?

Here are the details:

In the training subset, I do linear regression: $y = ax + b$, where $y$ is groundtruth (also known as target), $x$ is an independent variable. Then I found $a$ and $b$. ($x$ and $y$ are given in the training subset).

Now, using $a$ and $b$ found above from the training subset, apply them to the evaluation subset, I found $y' = ax' + b$. In other words, these $y'$ are found from linear regression with $x'$. Now, in addition to $y'$, I also have $y$ from the evaluation set. How do I evaluate my result (how much $y'$ differ from $y$)? Any general mathematical model to do that? It needs to be some sort of math model/formula. I can think of different ways to do it, but they are all kinda ad-hoc or simple, but this is for a scientific work, so things that sound ad-hoc cannot be used here, unfortunately.

Any idea?

$\endgroup$
  • 6
    $\begingroup$ I don't think this kind of assessment is generally used with simple regression models. What would it tell you that you wouldn't find out from using the entire dataset to generate your regression parameters? Normally the reason to use an evaluation dataset is to prevent overfitting, but that's not an issue when you already know that your model is going to contain just one independent variable. $\endgroup$ – octern Nov 10 '12 at 23:21
  • $\begingroup$ To be clear, @octern was talking about linear regression. Training-test split is perfectly good practice to do this with logistic regression. It could even make sense in linear regression if the test-set has different/non-stationary distribution than the training, or comes from a different time-period (e.g. advertising campaign, shopping behavior). $\endgroup$ – smci May 8 '17 at 8:33
11
$\begingroup$

I'd agree with @Octern that one rarely sees people using train/test splits (or even things like cross-validation) for linear models. Overfitting is (almost) certainly not an issue with a very simple model like this one.

If you wanted to get a sense for your model's "quality", you may want to report confidence intervals (or their Bayesian equivalents) around your regression coefficients. There are several ways to do this. If you know/can assume that your errors are normally distributed, there's a simple formula (and most popular data analysis packages will give you these values). Another popular alternative is to compute them through resampling (e.g., bootstrapping or jackknifing), which makes fewer assumptions about the distribution of errors. In either case, I'd use the complete data set for the computation.

$\endgroup$
  • 1
    $\begingroup$ You should also do residual analysis by plotting. $\endgroup$ – kjetil b halvorsen Sep 16 '16 at 12:07
1
$\begingroup$

if you really are fine with your linear trainig model and want to know how well it would predict your test data, then all you would have to do is to use the linear model formula you already have and include the estimated coefficients a (= intercept) and b(regression coefficient, also called slope) resulting from first model.

should look like y= a + b*X here some imaginary numbers... y= 2 + 0.5*X

Which software are you using? Are you using R ? if so, you can use the predict.lm() function and apply it on your 2nd dataset.

$\endgroup$
1
$\begingroup$

While this largely depends on exactly what your goals are, a simple and standard way to do this would be measuring the mean squared error (MSE). So if you have your test dataset $\mathcal{D}$ which consist of input/output pairs, $\mathcal{D} = \{(x_1, y_1), (x_2, y_2), \ldots, (x_n, y_n)\}$ and your parameters $a$ and $b$, then the MSE can be calculated as

$$ \text{MSE}_{a,b} = \frac{1}{n}\sum_{i=1}^n (y_i - (ax_i + b))^2. $$

This is probably a sensible way to measure your error also since this is likely the criteria you used for finding the parameters $a$ and $b$. If you want to get a better idea of how well your estimated parameters generalize, you should look into something like cross validation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.