0
$\begingroup$

I'm using the Regression Learner tool in MATLAB to do robust linear regression on a set of variables. However, with only one variable I get a higher R-squared value than when I'm adding one or two extra variables, which I thought was impossible (R2=0.979 for three variables and R2=0.992 for one). Does anybody have an idea of what the problem could be and how to solve it?

This is the info about my variables:

enter image description here

And here are the results I get when using one and three variables, respectively:

enter image description here

enter image description here

$\endgroup$
1
  • 1
    $\begingroup$ I don't think the intercept of 944 make sense (with SE of 120)? I suspect that column/3/4 are adding redundancy causing some sort of lack of convergence $\endgroup$
    – seanv507
    Commented Jan 5, 2022 at 10:08

2 Answers 2

4
$\begingroup$

You are not using OLS, so why do you think $R^2$ should increase when you add more variables? the premise of your question is flawed.

You should show what command you used to generate the output. I can see that you probably used robustfit function in MATLAB. This function is not OLS. It uses an iterative algorithm to weight the observations, looking for what it thinks are outliers and down weighting them.

With such algorithms like MATLAB robustfit, $R^2$ can change in any way when you add variables: increase, decrease or stay the same.

$\endgroup$
2
  • $\begingroup$ OP: This answer seems consistent with mine. Further, what is the correspondence between column_1 etc and x1 etc in your question? $\endgroup$
    – Nick Cox
    Commented Jan 5, 2022 at 15:37
  • 1
    $\begingroup$ This is the most direct answer to the question, namely that R-square does not behave as you would expect because you aren’t using plain regression. To me the bigger question is then to make sense of your data. This is a disappointing thread: the OP appears to have drifted away and there is not much support for any of the answers. I will upvote this one. $\endgroup$
    – Nick Cox
    Commented Jan 8, 2022 at 10:42
1
$\begingroup$

ORIGINAL ANSWER:

All bets are off without knowing what MATLAB is doing here under the heading (robust fit).

The results show quite different models: one forced through the origin (intercept zero) and the other omitting a predictor that the first model declares strongly significant. It's implausible -- despite the otherwise appealing figures of merit -- that either model is really a good summary of the data.

My guesses (expanding on remarks by others) are that

  1. $R^2$ is here added by analogy; the usual interpretation that each regression can be thought of as maximising $R^2$ just does not apply.

  2. A better answer is to be found in a closer inspection of your data, with attention to correlations among predictors and also the possibility of extreme skewness and/or outliers.

  3. Most of the P-values are too good to be true.

The dataset is perhaps a little too large for you to give here, although that would be ideal. We at least need a scatter plot matrix for the five variables concerned.

If you can post a listing of your data, please don't use an image but a portable listing with say a line of column headers and then data that are separated by spaces or commas.

I'd guess that most people active here don't use MATLAB.

EDIT:

You have added a listing giving a sight of your data, for which thanks. Presumably y here is y in your first results and x1 x2 x3 are three of the columns column_1 column_2 column_3 column_4.

From a largely graphical analysis I can't see that multiple regression makes sense at all -- at least without a subtle, domain-specific analysis that draws upon a physical (chemical, engineering, whatever) understanding of the set-up. The context calibration of sensors leads me to hope for some simple or least strong relationships.

The two graphs following are (1) quantile plots for each variable (just the ordered values against a cumulative probability scale) and (2) a scatter plot matrix. I imagine that these are, or should be, easy in MATLAB.

enter image description here

enter image description here

Simple cautions start with noticing

  • Two outliers on y (informally) and hence marked skewness. See the quantile plots.

  • The fact that x2 is almost constant. This may make complete sense in context but such a variable can be awkward at best as a predictor.

  • The presence of puzzling structure in plots of y versus x1 x2 x3 individually and the absence of even rough linearity.

I would guess further that the data are a pooling of results from quite different circumstances.

None of these facts rule out multiple regression absolutely, but unfortunately they make it less surprising that you get some weird and even contradictory results.

Whatever "robust fit" means, the main idea of any robust linear regression should be to get fair results if $y = Xb$ is in essence a good idea, but there are awkward complications in the data. But $y = Xb$ has to be in essence a good idea.

I very tentatively ran a quantile regression on this (which may not qualify as "robust", according to people's definitions) and got the calibration plot below. Here calibration plot is sometimes used in what I read to mean a plot of observed response versus fitted or predicted response, but may not correspond to any usage in your field.

A pessimist could only bounce this back for more guidance. An optimist might see a hint of a model that might work well for some of the data with a mix of many data points that behave quite differently.

(I used Stata, but don't think anything hinges on that. I haven't tried to look for explanations of what robust fit means here. "Robust regression" can mean many different things depending on who is talking, including just plain or vanilla regression with Huber-Eicker-White-sandwich standard errors.)

enter image description here

$\endgroup$
5
  • 1
    $\begingroup$ In full fairness, MATLAB is pretty clear that by default uses a bisquare weighting to perform iteratively reweighted least squares; function doc robustfit - the first link if one googles or bings: matlab robust regression. Whether or not the OP does something extra/different to that (or indeed uses default arguments), it is impossible to tell with certainty based on the information provided in the post. $\endgroup$
    – usεr11852
    Commented Jan 8, 2022 at 0:59
  • $\begingroup$ That’s a very helpful comment. Rightly or wrongly, I didn’t see it as my job here to chase up documentation on software I don’t routinely use. I still suggest that even that kind of algorithm can’t find linear structure reliably unless it’s present in thev data. (Turn and turn around, I do look out for Stata questions and don’t expect people who don’t use Stata to work hard at reading its documentation.) $\endgroup$
    – Nick Cox
    Commented Jan 8, 2022 at 10:36
  • $\begingroup$ I view it as part of the CV.SE game especially if the doc is "relatively straightforward" to find. I get a better idea of underlying principles, I view professional software as a good exemplar as I often write small routines for my work anyway. That said, I almost never answer Stata questions cause I don't want to "play along" for more than 5' - despite Stata having some great docs. (+1 to the general post) $\endgroup$
    – usεr11852
    Commented Jan 8, 2022 at 13:09
  • 1
    $\begingroup$ I can't claim to be absolutely consistent in that I will sometimes Google in search of a better answer. In this case it seemed to me that the puzzlement over $R^2$ is a secondary issue, so I put quite a lot of effort into seeking a better analysis, which the OP has ignored to date. $\endgroup$
    – Nick Cox
    Commented Jan 8, 2022 at 13:41
  • 1
    $\begingroup$ I have used R occasionally, but clearly there are many, many active people here who use it a lot and generally R users can be expected to look after each other. The issue is trickier with software like MATLAB and Stata because in practice there are fewer active users here. $\endgroup$
    – Nick Cox
    Commented Jan 8, 2022 at 13:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.