Skip to main content
deleted 50 characters in body
Source Link

If you have enough data, I would use a conformal prediction-inspired approach for this task.

Use time series cross-validation to backtest how large your error typically is when predicting out N days, aggregating the predictions, and comparing against the aggregated actuals.

Let's say you're making daily predictions and interested in 30-day aggregated predictions and prediction intervals. In detail, the process would be:

  1. Set aside a 30-day validation period completely after a training datasetdataset.
  2. Train your RF on the training data.
  3. Predict on the 30-day validation period, aggregate to a single value.
  4. Compare to the aggregated actual values and calculate the error
  5. Step forward k days and repeat steps 1-4 as often as your data allows. k should ideally be 30, but if you don't have a lot of data, perhaps allowing some overlap and using 10 or 15 would be fine.
  6. Take the absolute values of all the errors, and use quantiles to calculate prediction intervals. For example, if you wanted a 95% prediction interval, you would take the 0.95 quantile of the absolute errors, add and subtract that from the predicted value to get upper and lower prediction interval values.

If the target variable is skewed, it may work better to not take the absolute value of the errors, and instead take the 0.025 and 0.975 quantiles of the signed errors to produce a 95% prediction interval.

If you want to read more about conformal prediction generally, I found this resource helpful: https://www.stat.berkeley.edu/~ryantibs/statlearn-s23/lectures/conformal.pdf

If you have enough data, I would use a conformal prediction-inspired approach for this task.

Use time series cross-validation to backtest how large your error typically is when predicting out N days, aggregating the predictions, and comparing against the aggregated actuals.

Let's say you're making daily predictions and interested in 30-day aggregated predictions and prediction intervals. In detail, the process would be:

  1. Set aside a 30-day validation period completely after a training dataset.
  2. Train your RF on the training data.
  3. Predict on the 30-day validation period, aggregate to a single value.
  4. Compare to the aggregated actual values and calculate the error
  5. Step forward k days and repeat steps 1-4 as often as your data allows. k should ideally be 30, but if you don't have a lot of data, perhaps allowing some overlap and using 10 or 15 would be fine.
  6. Take the absolute values of all the errors, and use quantiles to calculate prediction intervals. For example, if you wanted a 95% prediction interval, you would take the 0.95 quantile of the absolute errors, add and subtract that from the predicted value to get upper and lower prediction interval values.

If the target variable is skewed, it may work better to not take the absolute value of the errors, and instead take the 0.025 and 0.975 quantiles of the signed errors to produce a 95% prediction interval.

If you want to read more about conformal prediction generally, I found this resource helpful: https://www.stat.berkeley.edu/~ryantibs/statlearn-s23/lectures/conformal.pdf

If you have enough data, I would use a conformal prediction-inspired approach for this task.

Use time series cross-validation to backtest how large your error typically is when predicting out N days, aggregating the predictions, and comparing against the aggregated actuals.

Let's say you're making daily predictions and interested in 30-day aggregated predictions and prediction intervals. In detail, the process would be:

  1. Set aside a 30-day validation period completely after a training dataset.
  2. Train your RF on the training data.
  3. Predict on the 30-day validation period, aggregate to a single value.
  4. Compare to the aggregated actual values and calculate the error
  5. Step forward k days and repeat steps 1-4 as often as your data allows. k should ideally be 30, but if you don't have a lot of data, perhaps allowing some overlap and using 10 or 15 would be fine.
  6. Take the absolute values of all the errors, and use quantiles to calculate prediction intervals. For example, if you wanted a 95% prediction interval, you would take the 0.95 quantile of the absolute errors, add and subtract that from the predicted value to get upper and lower prediction interval values.

If the target variable is skewed, it may work better to not take the absolute value of the errors, and instead take the 0.025 and 0.975 quantiles of the signed errors to produce a 95% prediction interval.

If you want to read more about conformal prediction generally, I found this resource helpful: https://www.stat.berkeley.edu/~ryantibs/statlearn-s23/lectures/conformal.pdf

If you have enough data, I would use a conformal prediction-inspired approach for this task.
Source Link

If you have enough data, I would use a conformal prediction-inspired approach for this task.

Basically, useUse time series cross validation-validation to backtest how large your error typically is when predicting out N days, aggregating the predictions, and comparing against the aggregated actuals.

Let's say you're making daily predictions and interested in 30-day aggregated predictions and prediction intervals. In detail, the process would be:

  1. Set aside a 30-day validation period completely after a training datasetdataset.
  2. Train your RF on the training data.
  3. Predict on the 30-day validation period, aggregate to a single value.
  4. Compare to the aggregated actual values and calculate the error
  5. Step forward k days and repeat steps 1-4 as many timesoften as your data will allowallows. k should ideally be 30, but if you don't have a lot of data, perhaps allowing some overlap and using 10 or 15 would be fine.
  6. Take the absolute values of all the errors, and use quantiles to calculate prediction intervals. For example, if you wanted a 95% prediction interval, you would take the 0.95 quantile of the absolute errors, and both add and subtract that from the predicted value to get upper and lower prediction interval values.

If the target variable is skewed, then it may work better to not take the absolute value of the errors, and instead take the 0.025 and 0.975 quantiles of the signed errors to produce a 95% prediction interval.

If you want to read more about conformal prediction generally, I found this resource helpful: https://www.stat.berkeley.edu/~ryantibs/statlearn-s23/lectures/conformal.pdf

If you have enough data, I would use a conformal prediction-inspired approach for this task.

Basically, use time series cross validation to backtest how large your error typically is when predicting out N days, aggregating the predictions, and comparing against the aggregated actuals.

Let's say you're making daily predictions and interested in 30-day aggregated predictions and prediction intervals. In detail, the process would be:

  1. Set aside a 30-day validation period completely after a training dataset.
  2. Train your RF on the training data.
  3. Predict on the 30-day validation period, aggregate to a single value.
  4. Compare to the aggregated actual values and calculate the error
  5. Step forward k days and repeat steps 1-4 as many times as your data will allow. k should ideally be 30, but if you don't have a lot of data, perhaps allowing some overlap and using 10 or 15 would be fine.
  6. Take the absolute values of all the errors, and use quantiles to calculate prediction intervals. For example, if you wanted a 95% prediction interval you would take the 0.95 quantile of the absolute errors, and both add and subtract that from the predicted value to get upper and lower prediction interval values.

If the target variable is skewed, then it may work better to not take the absolute value of the errors, and instead take the 0.025 and 0.975 quantiles of the signed errors to produce a 95% prediction interval.

If you want to read more about conformal prediction generally, I found this resource helpful: https://www.stat.berkeley.edu/~ryantibs/statlearn-s23/lectures/conformal.pdf

If you have enough data, I would use a conformal prediction-inspired approach for this task.

Use time series cross-validation to backtest how large your error typically is when predicting out N days, aggregating the predictions, and comparing against the aggregated actuals.

Let's say you're making daily predictions and interested in 30-day aggregated predictions and prediction intervals. In detail, the process would be:

  1. Set aside a 30-day validation period completely after a training dataset.
  2. Train your RF on the training data.
  3. Predict on the 30-day validation period, aggregate to a single value.
  4. Compare to the aggregated actual values and calculate the error
  5. Step forward k days and repeat steps 1-4 as often as your data allows. k should ideally be 30, but if you don't have a lot of data, perhaps allowing some overlap and using 10 or 15 would be fine.
  6. Take the absolute values of all the errors, and use quantiles to calculate prediction intervals. For example, if you wanted a 95% prediction interval, you would take the 0.95 quantile of the absolute errors, add and subtract that from the predicted value to get upper and lower prediction interval values.

If the target variable is skewed, it may work better to not take the absolute value of the errors, and instead take the 0.025 and 0.975 quantiles of the signed errors to produce a 95% prediction interval.

If you want to read more about conformal prediction generally, I found this resource helpful: https://www.stat.berkeley.edu/~ryantibs/statlearn-s23/lectures/conformal.pdf

Source Link

If you have enough data, I would use a conformal prediction-inspired approach for this task.

Basically, use time series cross validation to backtest how large your error typically is when predicting out N days, aggregating the predictions, and comparing against the aggregated actuals.

Let's say you're making daily predictions and interested in 30-day aggregated predictions and prediction intervals. In detail, the process would be:

  1. Set aside a 30-day validation period completely after a training dataset.
  2. Train your RF on the training data.
  3. Predict on the 30-day validation period, aggregate to a single value.
  4. Compare to the aggregated actual values and calculate the error
  5. Step forward k days and repeat steps 1-4 as many times as your data will allow. k should ideally be 30, but if you don't have a lot of data, perhaps allowing some overlap and using 10 or 15 would be fine.
  6. Take the absolute values of all the errors, and use quantiles to calculate prediction intervals. For example, if you wanted a 95% prediction interval you would take the 0.95 quantile of the absolute errors, and both add and subtract that from the predicted value to get upper and lower prediction interval values.

If the target variable is skewed, then it may work better to not take the absolute value of the errors, and instead take the 0.025 and 0.975 quantiles of the signed errors to produce a 95% prediction interval.

If you want to read more about conformal prediction generally, I found this resource helpful: https://www.stat.berkeley.edu/~ryantibs/statlearn-s23/lectures/conformal.pdf