Skip to main content
added 330 characters in body
Source Link
Tim
  • 141.2k
  • 26
  • 270
  • 512

It is not that much about independent and identically distributed random variables, because you can assume that the prices are not independent, but rather auto-correlated over time. In many cases, you would be interested in meeting the looser criterion of exchangability. Since it changes over time, you would rather be concerned if the underlying process is stationary.

But to answer this question you don't really need complicated statistical terms. In general, you want to have relevant data, that well represents the underlying distribution. Using irrelevant data is unlikely to help, it can even hurt the performance of the model. In some cases, you may be able to transfer what you've learned in one dataset to another (e.g. the seasonality in prices changes) so it still might be useful to include such data. But the general answer is that you should have good reasons to use the data if you know that it follows a different distribution than the prediction-time data. There is nothing bad in throwing away data that is irrelevant to your problem. As discussed by Xiao-Li Meng in the Statistical paradises and paradoxes in Big Data talk, you usually don't need more data, but better data.

It is not that much about independent and identically distributed random variables, because you can assume that the prices are not independent, but rather auto-correlated over time. In many cases, you would be interested in meeting the looser criterion of exchangability. Since it changes over time, you would rather be concerned if the underlying process is stationary.

But to answer this question you don't really need complicated statistical terms. In general, you want to have relevant data, that well represents the underlying distribution. Using irrelevant data is unlikely to help, it can even hurt the performance of the model. In some cases, you may be able to transfer what you've learned in one dataset to another (e.g. the seasonality in prices changes) so it still might be useful to include such data. But the general answer is that you should have good reasons to use the data if you know that it follows a different distribution than the prediction-time data.

It is not that much about independent and identically distributed random variables, because you can assume that the prices are not independent, but rather auto-correlated over time. In many cases, you would be interested in meeting the looser criterion of exchangability. Since it changes over time, you would rather be concerned if the underlying process is stationary.

But to answer this question you don't really need complicated statistical terms. In general, you want to have relevant data, that well represents the underlying distribution. Using irrelevant data is unlikely to help, it can even hurt the performance of the model. In some cases, you may be able to transfer what you've learned in one dataset to another (e.g. the seasonality in prices changes) so it still might be useful to include such data. But the general answer is that you should have good reasons to use the data if you know that it follows a different distribution than the prediction-time data. There is nothing bad in throwing away data that is irrelevant to your problem. As discussed by Xiao-Li Meng in the Statistical paradises and paradoxes in Big Data talk, you usually don't need more data, but better data.

Source Link
Tim
  • 141.2k
  • 26
  • 270
  • 512

It is not that much about independent and identically distributed random variables, because you can assume that the prices are not independent, but rather auto-correlated over time. In many cases, you would be interested in meeting the looser criterion of exchangability. Since it changes over time, you would rather be concerned if the underlying process is stationary.

But to answer this question you don't really need complicated statistical terms. In general, you want to have relevant data, that well represents the underlying distribution. Using irrelevant data is unlikely to help, it can even hurt the performance of the model. In some cases, you may be able to transfer what you've learned in one dataset to another (e.g. the seasonality in prices changes) so it still might be useful to include such data. But the general answer is that you should have good reasons to use the data if you know that it follows a different distribution than the prediction-time data.