0
$\begingroup$

I am running a mlr in python on a dataset with 2D feature vectors, X1 and X2 on a single response, Y. The data ends up being funnel-shaped, as below: X1 v Y, with the colors being X2. Original Data, X1 v Y with the colors marking X2

It was difficult to fit any linear fit here, so I tried to do a log-log transform with the following results.

Log-Log Transformation

As you can see above the funnel is narrowed. Without going into too much detail, my r-squared score improved to about 0.79, but my mean absolute error was still high. As I am trying to use this model for predictions, I am trying to reduce the mean error as much as possible. I also tried a log transformation of the response only, and tried a polynomial fit.

Polynomial Fit

However the results ended up being not much better than the log-log transform. MAE went down to about 0.35 (which I interpret as about a 35% deviation from the mean?) and. r-squared at about 0.79.

Is there anything else I can do to transform the data so a linear or mlr or polynomial fit can work? I need as low error as possible. Thank you!!!!

EDIT:

The data is a data set of solar irradiance and cloud cover. Irradiance is measured as Global Horizontal Irradiance "GHI" and cloud cover as a decimal between 0 and 1 inclusive. The target variable is a type of solar irradiance called Diffuse Horizontal Irradiance "DHI" which results from reflected or scattered sunlight falling upon a measurement device.

In all data here, GHI (the total measured sunlight) is on the X axis, and the diffused (reflected) irradiance on the Y axis. Cloud cover is indicated by the colors. Hope this helps.

$\endgroup$
15
  • 1
    $\begingroup$ You are coy about what the data really are. I agree that heteroscedasticity is evident but the reasons may include constraints on the variables, either singly or jointly. The edges to the scatter are sharper than is common. The first plot is X1 versus Y which I take to mean that Y is on the horizontal axis, although usages differ (see stats.stackexchange.com/questions/146533/…). The colour coding of X2 does nothing for me, and I suggest that you plot a scatter plot matrix with Y, X1, X2 and tell us their ranges [minimum, maximum]. $\endgroup$ – Nick Cox Mar 21 '20 at 11:57
  • 1
    $\begingroup$ The log transform overcorrects, which sometimes occurs if variables are bounded. If in principle they are limited, that is important information. $\endgroup$ – Nick Cox Mar 21 '20 at 11:59
  • $\begingroup$ MAE when the response is logged is in quite different units from the original. It can even go up if you forget about the units. $\endgroup$ – Nick Cox Mar 21 '20 at 12:02
  • $\begingroup$ What is the colo[u]r coding in any case? It looks like a rainbow scheme which makes sense in physics but not in psychology (what the mind thinks) or physiology (what the eye sees). $\endgroup$ – Nick Cox Mar 21 '20 at 12:30
  • $\begingroup$ Sorry if I was being coy, I was just trying to simplify things. X1 is a measurement of Solar Irradiation picked up by a sensor. X2 is a measure of cloud cover from 0-1. Y is a measure of solar radiation picked up by indirect irradiation (like bouncing off clouds, other surfaces, etc) $\endgroup$ – Tuomas Talvitie Mar 21 '20 at 13:20
0
$\begingroup$

A whole slew of transformations didn't work, now resorting to fitting a GLS model.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.