Timeline for Applying log transformation to input data has almost no effect on pearson correlation, what does that tell me about my data?
Current License: CC BY-SA 4.0
14 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| May 13, 2022 at 18:12 | vote | accept | Janilson | ||
| May 13, 2022 at 17:45 | answer | added | EdM | timeline score: 1 | |
| May 13, 2022 at 17:30 | comment | added | Janilson | Yeah, that's a fair argument. Maybe I'm just looking too hard for something that isn't there. | |
| May 13, 2022 at 17:26 | comment | added | dipetkov | I understand that your analysis might be more complex. But for the step that you show in this question, you end up trying complex transformation of x just because of one influential point. It biases the regression, judging from the scatterplot of y against x. | |
| May 13, 2022 at 17:15 | comment | added | Janilson | @dipetkov I've already applied a preprocessing step of standardizing the variables and using a mahalanobis distance threshold to remove a few outliers so I felt a bit hesitant to remove more. I also have my doubts about trusting the scatter plot eye test due to the clearly curved QQ plot. Maybe it's the best I can do without gathering more data, but I was hoping to squeeze as much insight as I could out of this before resorting to that. | |
| May 13, 2022 at 17:06 | comment | added | Janilson | @EdM both variables strictly positive, x range in about 0.2 to 1.3, y range is between 10 and 40 | |
| May 13, 2022 at 17:05 | comment | added | dipetkov | And as a general advice, the more plots the better. You can also make a y-vs-f(x) plot for each transformation f(x) you've considered with the regression line overlaid. | |
| May 13, 2022 at 17:04 | comment | added | dipetkov | Thanks for adding the plot. There seems to be an influential data point. You can see it to the right of the plot, perfectly interpolated by the regression line. If you exclude this one point, you might get better results without any transformations. Visually, there is nothing nonlinear. | |
| May 13, 2022 at 17:04 | comment | added | EdM | What are the actual scales/ranges of y and x? x must be strictly positive as you tried a log transform. | |
| May 13, 2022 at 17:01 | comment | added | Janilson | I edited the post to include the scatter plot for x and y as well as the line obtained by the regression. I have more variables but they mostly have either significantly lower correlation to y or high collinearity to x, so I've tried PCR and PLS as alternative methods and they yielded slight improvements but not to a huge degree (r^2 went from 0.43 to 0.47), while also showing the same pattern on the QQ plot. | |
| May 13, 2022 at 16:55 | history | edited | Janilson | CC BY-SA 4.0 |
added 166 characters in body
|
| May 13, 2022 at 16:44 | comment | added | dipetkov | You have one predictor x and one outcome y. Have you looked at the plot of y against x. It will be esp. interesting to see this plot since the question is whether there is a non-linear relationship between x and y. | |
| S May 13, 2022 at 16:31 | review | First questions | |||
| May 13, 2022 at 16:56 | |||||
| S May 13, 2022 at 16:31 | history | asked | Janilson | CC BY-SA 4.0 |