The Stack Overflow podcast is back! Listen to an interview with our new CEO.

Questions tagged [forecasting]

Prediction of the future events. It is a special case of [prediction], in the context of [time-series].

Filter by
Sorted by
Tagged with
207
votes
3answers
19k views

How to know that your machine learning problem is hopeless?

Imagine a standard machine-learning scenario: You are confronted with a large multivariate dataset and you have a pretty blurry understanding of it. What you need to do is to make predictions ...
83
votes
1answer
85k views

How to apply Neural Network to time series forecasting?

I'm new to machine learning, and I have been trying to figure out how to apply neural network to time series forecasting. I have found resource related to my query, but I seem to still be a bit lost. ...
68
votes
10answers
45k views

What is wrong with extrapolation?

I remember sitting in stats courses as an undergrad hearing about why extrapolation was a bad idea. Furthermore, there are a variety of sources online which comment on this. There's also a mention of ...
47
votes
3answers
3k views

AIC,BIC,CIC,DIC,EIC,FIC,GIC,HIC,IIC — Can I use them interchangeably?

On p. 34 of his PRNN Brian Ripley comments that "The AIC was named by Akaike (1974) as 'An Information Criterion' although it seems commonly believed that the A stands for Akaike". Indeed, when ...
37
votes
2answers
7k views

Is it unusual for the MEAN to outperform ARIMA?

I recently applied a range of forecasting methods (MEAN, RWF, ETS, ARIMA and MLPs) and found that MEAN did surprisingly well. (MEAN: where all future predictions are predicted as been equal to the ...
37
votes
4answers
73k views

Difference between forecast and prediction?

I was wondering what difference and relation are between forecast and prediction? Especially in time series and regression? For example, am I correct that: In time series, forecasting seems to mean ...
35
votes
6answers
30k views

Best method for short time-series

I have a question related to modeling short time-series. It is not a question if to model them, but how. What method would you recommend for modeling (very) short time-series (say of length $T \leq 20$...
35
votes
1answer
40k views

Detecting Outliers in Time Series (LS/AO/TC) using tsoutliers package in R. How to represent outliers in equation format?

Comments: Firstly I would like to say a big thank you to the author of the new tsoutliers package which implements Chen and Liu's time series outlier detection which was published in the Journal of ...
30
votes
9answers
66k views

Why use vector error correction model?

I am confused about the Vector Error Correction Model (VECM). Technical background: VECM offers a possibility to apply Vector Autoregressive Model (VAR) to integrated multivariate time series. In the ...
25
votes
2answers
996 views

When is it appropriate to use an improper scoring rule?

Merkle & Steyvers (2013) write: To formally define a proper scoring rule, let $f$ be a probabilistic forecast of a Bernoulli trial $d$ with true success probability $p$. Proper scoring ...
24
votes
4answers
39k views

When to log transform a time series before fitting an ARIMA model

I have previously used forecast pro to forecast univariate time series, but am switching my workflow over to R. The forecast package for R contains a lot of useful functions, but one thing it doesn't ...
23
votes
1answer
2k views

Explanation of what Nate Silver said about loess

In a question I asked recently, I was told that it was a big "no-no" to extrapolate with loess. But, in Nate Silver's most recent article on FiveThirtyEight.com he discussed using loess for making ...
22
votes
1answer
17k views

How to decompose a time series with multiple seasonal components?

I have a time series that contains double seasonal components and I would like to decompose the series into the following time series components (trend, seasonal component 1, seasonal component 2 and ...
22
votes
3answers
2k views

AIC versus cross validation in time series: the small sample case

I am interested in model selection in a time series setting. For concreteness, suppose I want to select an ARMA model from a pool of ARMA models with different lag orders. The ultimate intent is ...
21
votes
1answer
88k views

How can I predict values from new inputs of a linear model in R?

I've created a linear model in R: mod = lm(train_y ~ train_x). I want to pass it a list of X's and get its predicted/estimateed/forecasted Y. I looked at ...
21
votes
2answers
20k views

Interpretation of mean absolute scaled error (MASE)

Mean absolute scaled error (MASE) is a measure of forecast accuracy proposed by Koehler & Hyndman (2006). $$MASE=\frac{MAE}{MAE_{in-sample, \, naive}}$$ where $MAE$ is the mean absolute error ...
20
votes
6answers
1k views

Is my weatherman accurate?

A question which bothered me for some time, which I don't know how to address: Every day, my weatherman gives a percentage chance of rain (let's assume its calculated to 9000 digits and he has never ...
19
votes
3answers
22k views

How to use DLM with Kalman filtering for forecasting

Could someone walk me through an example on how to use DLM Kalman filtering in R on a time series. Say I have a these values (quarterly values with yearly seasonality); how would you use DLM to ...
19
votes
3answers
1k views

How can we judge the accuracy of Nate Silver's predictions?

Firstly, he gives probability of outcomes. So, for example, his predictions for the U.S. election is currently 82% Clinton vs 18% Trump. Now, even if Trump wins, how do I know that it wasn't just the ...
19
votes
2answers
3k views

Why does minimizing the MAE lead to forecasting the median and not the mean?

From the Forecasting: Principles and Practice textbook by Rob J Hyndman and George Athanasopoulos, specifically the section on accuracy measurement: A forecast method that minimizes the MAE will ...
19
votes
3answers
557 views

How to tell if girlfriend can tell the future (i.e. predict stocks)?

My girlfriend has recently gotten a job doing sales and trading at a major bank. Buoyed by her new job, she believes she can predict whether stocks will be up or down at the end of the month greater ...
18
votes
2answers
27k views

VAR forecasting methodology

I am building a VAR model to forecast the price of an asset and would like to know whether my method is statistically sound, whether the tests I have included are relevant and if more are needed to ...
18
votes
1answer
3k views

Paradox in model selection (AIC, BIC, to explain or to predict?)

Having read Galit Shmueli's "To Explain or to Predict" (2010) I am puzzled by an apparent contradiction. There are three premises, AIC- versus BIC-based model choice (end of p. 300 - start of p. 301):...
18
votes
1answer
1k views

$ARIMA(p,d,q)+X_t$, Simulation over Forecasting period

I have time series data and I used an $ARIMA(p,d,q)+X_t$ as the model to fit the data. The $X_t$ is an indicator random variable that is either 0 (when I don’t see a rare event) or 1 (when I see the ...
17
votes
5answers
2k views

Can data cleaning worsen the results of statistical analysis?

An increase in the number of cases and deaths occurs during epidemics (sudden increase in numbers) due to a virus circulation (like West Nile Virus in USA in 2002) or decreasing resistance of people ...
17
votes
1answer
706 views

Stepwise AIC - Does there exist controversy surrounding this topic?

I've read countless posts on this site that are incredibly against the use of stepwise selection of variables using any sort of criterion whether it be p-values based, AIC, BIC, etc. I understand why ...
16
votes
2answers
3k views

Is it possible to automate time series forecasting?

I would like to build an algorithm that would be able to analyze any time series and "automatically" choose the best traditional/statiscal forecasting method (and its parameters) for the analyzed time ...
16
votes
3answers
16k views

ETS() function, how to avoid forecast not in line with historical data?

I am working on an alogorithm in R to automatize a monthly forecast calculation. I am using, among others, the ets() function from the forecast package to calculate forecast. It is working very well. ...
16
votes
2answers
15k views

stochastic vs deterministic trend/seasonality in time series forecasting

I have moderate background in time series forecasting. I have looked at several forecasting books, and I don't see the following questions addressed in any of them. I have two questions: How would I ...
16
votes
2answers
8k views

Getting started with neural networks for forecasting

I need some resources to get started on using neural networks for time series forecasting. I am wary of implementing some paper and then finding out that they have greatly over stated the potential of ...
16
votes
2answers
7k views

How to do forecasting with detection of outliers in R? - Time series analysis procedure and Method

I have monthly time series data, and would like to do forecasting with detection of outliers . This is the sample of my data set: ...
16
votes
1answer
1k views

Problem defining ARIMA order

This is a long post so I hope you can bear with me, and please correct me where I'm wrong. My goal is to produce a daily forecast based on 3 or 4 weeks of historical data. The data is 15 minute ...
16
votes
1answer
12k views

Multivariant time series in R. How to find lagged correlation and build model for forecasting

I'm new in the page and pretty new in statistics and R. I'm working on a project for college with the objective of finding the correlation between rain and water flow level in rivers. Once the ...
16
votes
1answer
957 views

Least stupid way to forecast a short multivariate time series

I need to forecast the following 4 variables for the 29th unit of time. I have roughly 2 years worth of historical data, where 1 and 14 and 27 are all the same period (or time of year). In the end, I ...
15
votes
3answers
36k views

Why use a certain measure of forecast error (e.g. MAD) as opposed to another (e.g. MSE)?

MAD = Mean Absolute Deviation MSE = Mean Squared Error I've seen suggestions from various places that MSE is used despite some undesirable qualities (e.g. http://www.stat.nus.edu.sg/~staxyc/T12.pdf, ...
15
votes
5answers
10k views

How to model prices?

I asked this question on the matemathics stackexchange site and was recommended to ask here. I'm working on a hobby project and would need some help with the following problem. A bit of context Let'...
15
votes
2answers
27k views

Estimate ARMA coefficients through ACF and PACF inspection

How do you estimate the appropriate forecast model for a time series by visual inspection of the ACF and PACF plots? Which one (i.e., ACF or PACF) tells the AR or the MA (or do they both)? Which part ...
15
votes
1answer
5k views

How to achieve strictly positive forecasts?

I am working on a time series whose values are strictly positive. Working with various models including AR, MA, ARMA, etc, I couldn't find an easy way to achieve strictly positive forecasts. I'm ...
15
votes
2answers
6k views

ARIMA estimation by hand

I'm trying to understand how the parameters are estimated in ARIMA modeling/Box Jenkins (BJ). Unfortunately none of the books that I have encountered describes the estimation procedure such as Log-...
15
votes
1answer
10k views

Time Series Forecasting with Daily Data: ARIMA with regressor

I'm using a daily time series of sales data that contains about 2 years of daily data points. Based on some of the online-tutorials / examples I tried to identify the seasonality in the data. It seems ...
15
votes
3answers
556 views

Determining whether a website is active using daily visits

Context: I have a group of websites where I record the number of visits on a daily basis: ...
15
votes
3answers
15k views

Using the R forecast package with missing values and/or irregular time series

I am impressed by the R forecast package, as well as e.g. the zoo package for irregular time series and interpolation of missing ...
14
votes
5answers
67k views

What is difference between “in-sample” and “out-of-sample” forecasts?

I don't understand what exactly is the difference between "in-sample" and "out of sample" prediction? An in-sample forecast utilizes a subset of the available data to forecast values outside of the ...
14
votes
4answers
2k views

Predictive models: statistics can't possibly beat machine learning? [closed]

I am currently following a master program focused on statistics/econometrics. In my master, all students had to do 3 months of research. Last week, all groups had to present their research to the rest ...
14
votes
1answer
8k views

How to calculate forecast error (confidence intervals) for ongoing periods?

I often need to forecast for future periods in monthly series of data. Formulas are available to calculate the confidence interval at alpha for the next period in the time series, but this never ...
14
votes
1answer
1k views

Dealing with missing data in an exponential smoothing model

There does not seem to be a standard way to deal with missing data in the context of the exponential smoothing family of models. In particular, the R implementation called ets in the forecast package ...
13
votes
2answers
10k views

Timeseries analysis procedure and methods using R

I am working on a small project where we are trying to predict the prices of commodities (Oil, Aluminium, Tin, etc.) for the next 6 months. I have 12 such variables to predict and I have data from Apr,...
13
votes
4answers
13k views

Are models identified by auto.arima() parsimonious?

I have been trying to learn and apply ARIMA models. I have been reading an excellent text on ARIMA by Pankratz - Forecasting with Univariate Box - Jenkins Models: Concepts and Cases. In the text the ...
13
votes
2answers
858 views

Using time series analysis to analyze/predict violent behavior

This is a bit of a flippant question, but I have a serious interest in the answer. I work in a psychiatric hospital and I have three years' of data, collected every day across each ward regarding the ...
13
votes
1answer
4k views

Calculating forecast error with time series cross-validation

I have a forecasting model for a time series and I want to calculate its out-of-sample prediction error. At the moment the strategy I'm following is the one suggested on Rob Hyndman's blog (near the ...