1
$\begingroup$

I have 50 variables, mostly numeric, a few categorical, and my target is continuous. In addition, I have 300,000 observations.

I am looking for a way to predict the target and find which variables relate to it.

I ran a series of scatter plots, which gave nothing interesting (no linear relation). I ran a linear regression with stepwise, again, nothing. When I say nothing, I mean that a couple of variables did stay in the model, with significant P-Values, but R squared was very low, around 3%, so the significance is due to the large sample.

I wanted to know if there is something else I could try. I am using R and Rattle for this. Can I do more, or should I give up?

Also, how do I check if my predictions are correct? When my target is categorical, it's easy, I make a guess, either I am right or wrong, I take a testing set, calculate sensitivity and specificity and it's very clear. How do I validate with a continuous variable?

$\endgroup$
2
  • $\begingroup$ There are whole books on this subject. I recommend the book Applied Predictive Modeling by Kuhn and Johnson or reading into the caret package in R. $\endgroup$ Feb 12, 2016 at 14:32
  • $\begingroup$ That was a great question. However there's some things you should try to do if you want to get help quickly on SO: 1) Define R as a tag. 2) Always include a dput of your data More Info. This allows us to look at your data, play with it in our own ways, and easily help you out. 3) Register with SO. I usually don't answer questions from new users because they tend to not give credit to a correct answer or upvote those that helped them. $\endgroup$ Feb 28, 2017 at 14:07

1 Answer 1

1
$\begingroup$

I have 50 variables, most of them numeric, a few categorical, and my variable of interest is continuous. In addition, I have something like 300,000 observations.

Continuous how? I assume real-valued numbers? If so, their distribution might be important. Anything that deviates strongly from a normal distribution will disrupt some of the underlying principles in your regression model. Post it here if you do not know how to interpret it.

I am looking for a way to predict the target variable and to find which variables relate to it. I ran a series of scatter plots, which gave nothing interesting (no linear relation). I ran a linear regression with stepwise, again, nothing. [I wanted to know if there is something else I could try.]

As previously mentioned a linear model might not be the best way to go on with this. One normally tries kernel methods at that point, look into kernel ridge regression and Gaussian Processes for that. R should have support for both. You have 300k datapoints, which is a bit problamatic because some SVM solvers love to compute the whole kernel matrix, which in your case would be 300kx300k -- not so nice. Sequential Minimal Optimization (SMO) only computes partial values along the way and is the algorithm to use here. Apart from these models there is a simple trick you could try out. Currently, you have linear regression (hopefully regularized?), which gives predictions by: $$ f(x) = \mathbf{w}^{T}x+b $$ Seldomly people have luck by adding a few more terms to this equation via expending their feature vector, i.e. $$ f(x) = {\mathbf{w}_1}^T\mathbf{x} + {\mathbf{w}_2}^T{\mathbf{x}^2} + {\mathbf{w}_3}^T{\mathbf{x}^3} + ... $$ but again. If this works, it is usually luck.

I am using R and Rattle for this. Can I do more, of should I give up ?

You should never give up and obviously you can do much more, see above.

In addition, how do I check if my predictions are correct ?

Your linear regression essentially does something like this: $$\mathop {\min }\limits_{\mathbf{w},b} E(\mathcal{X})$$ $$E(\mathcal{X}) = {\sum\limits_{\left( {\mathbf{x},y} \right) \in \mathcal{X}} {\left( {f\left( \mathbf{x} \right) - y} \right)} ^2} = {\sum\limits_{\left( {\mathbf{x},y} \right) \in \mathcal{X}} {\left( {\left[ {{\mathbf{w}^T}\mathbf{x} + b} \right] - y} \right)} ^2}$$ $\mathcal{X}$ is your data (50x300k). The model minimizes the squared loss of the predictions of $f(x)$ c.f. the target values ($y$), so your method of testing (one or multiple hold-out datasets still holds), you just compute this measure over your test/validation-dataset.

I hope this helps. If not, ask.

Edit: I forgot to mention. Apart from these more "classical" approaches to regression, we now also have Deep Learning but it might take some time to get into it.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.