Timeline for Predicting Y using X for the following data
Current License: CC BY-SA 3.0
16 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jan 10, 2012 at 18:58 | comment | added | celenius | Ah yes - operator error unfortunately. Fixed the ordering of histograms now. | |
| Jan 10, 2012 at 18:57 | history | edited | celenius | CC BY-SA 3.0 |
reorder histograms
|
| Jan 10, 2012 at 18:24 | comment | added | whuber♦ | Well done! Please note that in the last graphic the plots are out of order (evidently the software ordered them alphabetically rather than by X, which is strange). The graphics reveal fairly complex behavior. Have you held out a sizable portion of your dataset to test predictions? If not, now would be a good time to do so. | |
| Jan 10, 2012 at 17:36 | history | edited | celenius | CC BY-SA 3.0 |
deleted 2 characters in body
|
| Jan 10, 2012 at 17:17 | comment | added | celenius | I updated the recent plots based on your suggestion, @whuber. I agree with your point about plotting Y.SD agains X.mean; I think I wanted to do that originally and got a little confused. | |
| Jan 10, 2012 at 17:15 | history | edited | celenius | CC BY-SA 3.0 |
updated plots based on whuber's suggestions
|
| Jan 10, 2012 at 16:49 | comment | added | whuber♦ | Good images in that edit. I recommend normalizing the histograms so they can be compared: don't show frequency, show relative frequency. Also, make the slices thicker: 8 per slice leaves a lot of scatter. You can afford slices containing hundreds of values. Doing that will clarify the SD. Indeed, plot the Y SD vs. X mean, not vs. X SD: you're interested in how the Y distribution varies with the location of X; the SD of X is meaningless. Finally, it's looking like you should be using the log of X in these plots rather than X itself. | |
| Jan 10, 2012 at 16:05 | history | edited | celenius | CC BY-SA 3.0 |
edited body
|
| Jan 10, 2012 at 16:04 | answer | added | yellowcap | timeline score: 1 | |
| Jan 10, 2012 at 15:35 | history | edited | celenius | CC BY-SA 3.0 |
added more illustrations of the data
|
| Jan 10, 2012 at 4:44 | comment | added | jbowman | For a plot you can do whatever you want, of course. As whuber points out, it's not at all clear that the relationship is heteroskedastic. But even if it is, if there isn't much heteroskedasticity, a linear fit will be quite good, especially with so much data. Otherwise, there are various methods you can use to (largely) correct for heteroskedasticity, e.g., generalized linear models. | |
| Jan 10, 2012 at 3:10 | answer | added | whuber♦ | timeline score: 8 | |
| Jan 10, 2012 at 1:54 | comment | added | celenius | Thanks for the suggestion @jbowman. If I plot the median (or variant) for each grouping can I use that value if I know that the standard deviation varies for each observation? I thought that was violating the heteroskedastic assumption required for a linear model (but I'm not certain if there is a way of dealing with this). | |
| Jan 10, 2012 at 0:31 | comment | added | jbowman | It looks like you have a lot more observations with low X than with high X, which in your plot obscures the relationship. You might want to try, for an initial exploration, grouping X, say by 500s, and plotting the mean or median of Y for each X group, and also the standard deviation or a robust version thereof (hubers from the MASS package will give you robust location and scale parameters.) This would give you some idea of what sort of relationship there is embedded in E(y|x) and SD(y|x), both useful when you move on to generalized linear models (glm or gam.) | |
| Jan 9, 2012 at 23:27 | history | tweeted | twitter.com/#!/StackStats/status/156517435883786241 | ||
| Jan 9, 2012 at 21:56 | history | asked | celenius | CC BY-SA 3.0 |