Return to Answer

If the distribution of the residuals are not normal, then you might want to consider other methods since the predictions and confidence intervals are likely to be misleading. Ease-of-computation doesn't seem like a good enough reason.

In terms of clock cycles, it's more expensive to create models than it is to make predictions from them. I would imagine that you'd create the model(s) relatively infrequently (but use them a lot), and so the model creation speed might be decoupled from your process.

To illustrate this, I created five toy models based on Hadley's fueleconomy dataset:

A couple of things stood out to me:

1. the models took much longer to create than they did to make predictions.
2. if speed is important, it's worth looking at the gputools R package. On my workstation, the GPU optimized linear regression executed about 100x faster than the standard one.

If the distribution of the residuals are not normal, then you might want to consider other methods since the predictions and confidence intervals are likely to be misleading. Ease-of-computation doesn't seem like a good enough reason.

In terms of clock cycles, it's more expensive to create models than it is to make predictions from them. I would imagine that you'd create the model(s) relatively infrequently (but use them a lot), and so the model creation speed might be decoupled from your process.

To illustrate this, I created five toy models based on Hadley's fueleconomy dataset:

A couple of things stood out to me:

1. the models took much longer to create than they did to make predictions.
2. if speed is important, it's worth looking at the gputools R package. On my workstation, the GPU optimized linear regression (gpuLm) was about 100x faster to create, and 10x faster to predict, than the standard R lm.

If the distribution of the residuals are not normal, then you might want to consider other methods since the predictions and confidence intervals are likely to be misleading. Ease-of-computation doesn't seem like a good enough reason.

In terms of clock cycles, it's more expensive to create models than it is to make predictions from them. I would imagine that you'd create the model(s) relatively infrequently (but use them a lot), and so the model creation speed might be decoupled from your process.

If the distribution of the residuals are not normal, then you might want to consider other methods since the predictions and confidence intervals are likely to be misleading. Ease-of-computation doesn't seem like a good enough reason.

In terms of clock cycles, it's more expensive to create models than it is to make predictions from them. I would imagine that you'd create the model(s) relatively infrequently (but use them a lot), and so the model creation speed might be decoupled from your process.

To illustrate this, I created five toy models based on Hadley's fueleconomy dataset:

A couple of things stood out to me:

1. the models took much longer to create than they did to make predictions.
2. if speed is important, it's worth looking at the gputools R package. On my workstation, the GPU optimized linear regression executed about 100x faster than the standard one.

If the distribution of the residuals is not normal, then you might want to consider other methods - particularly if you're planning to use forecasts or confidence intervals.

If you need to construct the model(s) quickly on a moderately sized dataset, you might consider performing the calculations on a GPU. R, for example, has a package called gputools that, for me, resulted in a 20x to 30x reduction in execution time for lm and glm. Once setup, it's just like the native lm function except that the call is made with the 'gpu' prefix, i.e. gpuLm(y ~ x1 + x2, data = df).

In terms of clock cycles, it's more expensive to create models than it is to make predictions from them. I would imagine that you'd create the model(s) relatively infrequently (but use them a lot), and so the model creation speed might be decoupled from your process.

You might try doing this in two stages: i.e. perform some sort of clustering to group similar records/transactions together, and then create a linear model for each cluster. You could try all sorts of things and cross-validate to figure our what works best.

If the distribution of the residuals are not normal, then you might want to consider other methods since the predictions and confidence intervals are likely to be misleading. Ease-of-computation doesn't seem like a good enough reason.

In terms of clock cycles, it's more expensive to create models than it is to make predictions from them. I would imagine that you'd create the model(s) relatively infrequently (but use them a lot), and so the model creation speed might be decoupled from your process.