0
$\begingroup$

I am R-tool beginner. I have a question regarding how to know the performance of a linear regression model by using validation data. My approach was

  1. Create training and validation data sets from original data set. "train" is name of my training data set and "valid" is name of my validation data set. "category" will be my target variable and "date_time" is my independent variable.

  2. Use training data set to create a regression model

    attach(train)

    lreg=lm(category~date_time)

  3. Now do predictions for validation data set using model created with training data set

    p=predict(lreg,valid)

  4. Now check the accuracy by finding the values of ACC, AUC.

    mmetric(valid$category,p,"AUC")

    mmetric(valid$category,p,"ACC")

Now if AUC and ACC have small values then it means that model created by training data set is not good in making predictions.

Is my approach correct ?

Thanks and regards!

$\endgroup$
  • 1
    $\begingroup$ Your questions are nearly entirely addressed at stats.stackexchange.com/questions/43310/… . $\endgroup$ – rolando2 Nov 16 '12 at 0:12
  • $\begingroup$ Thanks for redirecting me. As per my understanding I think I am doing the correct thing in my question. creating a model with training data and doing prediction of validation data. Then calculating the performance. But I shall really appreciate it if someone could just answer in Yes or No so that I can be more certain. Regards $\endgroup$ – user16603 Nov 16 '12 at 22:02
  • 2
    $\begingroup$ Please do not erase your question and replace it w/ something else. CV is intended to build a permanent repository of information about statistics via archived questions & answers. You are welcome to edit your Q and add updated info below the original question, but don't just erase what you initially asked. You may want to read our FAQ. $\endgroup$ – gung Dec 17 '12 at 0:47
  • $\begingroup$ You might want to explain what your outcome is like. For example, are the 9 categories ordered? $\endgroup$ – Dimitriy V. Masterov Jan 16 '13 at 1:48
1
$\begingroup$

You should take such a result w/a pinch of salt. For instance, due to a particular mix, despite a bad train result, you can get a good test result. Therefore, it's better if you do a cross-validation-like approach and try to define multiple train-test scenario and see if you have consistency across them.

PS, if your output is like classes, you may wanna you classification, not regression ...

$\endgroup$
  • $\begingroup$ thanks, Categories have values from 1 -9. And my aim was to guess category from independent variables. So I am confused about what to do. Should I do classification or regression? Can you please tell bit about how to perform classification in r-tool? However also please have a look at my question here: stats.stackexchange.com/questions/43040/… $\endgroup$ – user16603 Nov 15 '12 at 22:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy