New answers tagged

0 votes

What are good ways to statistically test for noise?

Much of this answer is copied from an answer I have given in this related question. If you have data with seasonal effects in it, you would usually fit a model with seasonal terms and then extract ...
  • 102k
1 vote

Measuring events in sales time series

What you normally do in such situations is to apply randomized A/B testing. I.e., you randomly separate your customers into two parts $g_A$ (the control group) and $g_B$ (the treatment group), and ...
  • 8,694
1 vote

Is it possible for long-run variance to be negative or 0?

Remember that long-run variance is a limit, so it is a little bit different than what one regularly thinks of as variance. For simplicity lets assume $\mu=0$ $$\lim_{T\to\infty}\{\text{Var}[\sqrt{T}\...
  • 1,041
0 votes
Accepted

Testing the impact of a events on time series

Sounds like Google's CausalImpact method might be useful (python implementation here). Briefly, you use pre-sale data to train a model that forecasts impressions, then forecast them for the sales ...
0 votes

Principle Component Analysis for Feature extraction from Voltage and Current Signals

The answer is yes. One can use sequence or timeseries generated by PCA for fault classification in transmission lines. Here is the story. I generated a fault sample in Simulink/MATLAB, saved data to ...
1 vote
Accepted

How to calculate variance of AR(1) process

The first step is to write the difference in terms of the original model: $$ \Delta Z_t = Z_{t} - Z_{t-1} = (\alpha-1)Z_{t-1} + \nu_t $$ Since $Z_{t-1}$ and $\nu_t$ are uncorrelated by the white noise ...
1 vote

Dickey-Fuller test significant => series stationary?

Another aspect that I recently became aware of (see particularly this thread Is the Dickey Fuller test one-sided or two-sided?) is that the DF-test can be interpreted as one-sided in the sense that it ...
0 votes
Accepted

Minimizing an objective function with input variables from a correlated error term

You can do linear regression for $y = X\beta + \epsilon$ using a covariance matrix for $\epsilon$ that belongs to an AR(2) model, or to any ARMA(p, q) model for that matter. You can e.g. use the ...
  • 8,694
1 vote

Resampling a timeseries

These returns are not iid, because: they aren't identically distributed (have different variances, even if having the same absolute or scaled mean), nor independent (the latter because their ...
  • 714
0 votes

Daily Data Transfer Logs - Anomaly Detection

What you are proposing to create are called statistical control or Shewhart charts. There are many references, I would recommend Chapter 14 of Box's book "Statistics for Experimenters". In ...
2 votes

Sum of $I(1)$ and WN

It seems to me that $$ \begin{aligned} Y(t)&=X(t)+w_t, ~~~w_t \sim N(0,\sigma^2_w) \\ & = X(t-1)+e_t+w_t, ~~~ e_t \sim N(0,\sigma^2_e),\\ & = (Y(t-1)-w_{t-1})+e_t+w_t \end{aligned} $$ is ...
0 votes

Do I need stationary time series data for Isolation Forest Model?

Tree based algorithms may struggle with non-stationary time series data. Especially when there is trend present, the future values of the series may have a very different distribution. For seasonality,...
  • 53.2k
2 votes

Is the Dickey Fuller test one-sided or two-sided?

First, $\phi$ is the true coefficient of the process, not the test statistic. The test statistic that is something else than $\phi$, namely the default t-statistic for the hypothesis that the ...
0 votes

Can anyone point me towards tutorials describing how to use the Kalman filter for forecasting?

The best I found till now is this open source interactive book: Kalman and Bayesian Filters in Python Hope this helps !
0 votes

Creating synthetic data for time series, Hidden Markov Model

If you know the parameters $(\pi, A, B)$ of the system you want to simulate, then it should be a pretty straightforward process. Simply draw a uniform random number and compare it to the probabilities ...
  • 41
1 vote

State space models in python statsmodels: including lag state innovation in observation equation

The problem here is that the $\theta$ and $\sigma_\varepsilon^2$ parameters are not separately identified. In particular, if you work through the Kalman filtering equations, it can be shown that the ...
  • 1,288
0 votes

Interpret the Conditional Sigma (vs Realized Absolute Returns) of the DCC model

These seem to be the fitted conditional standard deviations (or perhaps variances) of four time series. On average, you would expect higher absolute returns when the distribution has a higher standard ...
4 votes
Accepted

Simulating AR(2) Process With Initial Conditions

Programming the simulation from scratch requires just a few lines of $\textsf{R}$ code: ...
  • 2,510
-1 votes
Accepted

Understanding the loss funcion in DeepAR

I have a better understanding of the training process now. I wrote it down and it is shown as images.
0 votes

Preventing information leakage when scaling a time series?

No... for training data for time-series data you cannot do whatever you like. The original reasoning is correct, you are introducing leakage into your data if you are not scaling with only past data. ...
0 votes

What do the "coefficients" in R's HoltWinters function represent?

Coefficients in Holtwinter's Method In the HoltWinter's model with seasonal= 'additive', I am able to produce the model output for coefficients manually. It seems like we need to use a[N-p],b[N-p] ...
  • 1
0 votes

Multiple Time Series Prediction (Python)

For Python - you can use https://pypi.org/project/keras-generators/ to generate multiple times-series input and target data for training. It performs normalizing/scaling of data, train/test splits and ...
  • 101
-1 votes

Is regression a good model for predicting disk usage growth?

If you want to use regression, I recommend using the scikit-learn library. It's quite straightforward: https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html <...
  • 269
2 votes
Accepted

Logit transformation of target values in regression

In what follows, I assume that the rate is bounded by 0 and 1. This would map the rate to the log odds space, wherein the argument (rate / (1-rate)) would not be constrained to the unit interval. ...
2 votes
Accepted

Deal with quarterly or monthly seasonality in forecasting a year ahead

In forecasting, one typically does not remove seasonality, but models it with a seasonal model. The tag wiki contains pointers to literature; I especially recommend the online textbooks by ...
1 vote
Accepted

Computing reconciled prediction intervals when forecasting logged outcome variable using fable

The latest development version of fabletools allows you to probabilistically reconcile any forecast if you use sample/bootstrapped forecasts. Try updating the ...
0 votes

Does Box-Cox need to be Reversed Prior to Running Model Evaluation?

The process for ARIMA is to transform de data. Train the model, make prediction and reverse transform the data that has been predicted. If any other transformation has been made, like differentiation, ...
2 votes
Accepted

How do we make predictions for future data when you have lagged dependent features used in training?

Assuming your MAPE is on a test set and not based on the fitted values: Yeah using features that are your lagged target variable is pretty common when doing time series forecasting with LightGBM ...
  • 1,405
2 votes
Accepted

How do we compare MAPEs?

Certainly you can say that. You could also say that M2 is $1-\frac{\text{MAPE}_{M2}}{\text{MAPE}_{M1}}=30\%$ better. It depends on what the denominator of your percentage calculation of "better&...
0 votes

Dickey-Fuller test for stationarity

It is because when $\phi > 1$ it's clearly non-stationary, as it has trend. In this case, the failure of stationarity can be detected by eye, so the ADF test is not necessary. More importantly, the ...
2 votes
Accepted

How to correlate on sharp data changes

The simplest way would be to take the absolute value of the difference of each pressure value to its last value $\Delta p(t) = p(t)-p(t-1)$ and then calculate the correlation between this finite ...
  • 713
0 votes
Accepted

Difference between seasonality-adjusted data and the trend component

An additive time series decomposition expresses a time series as $$y_t = T_t + S_t + R_t$$ where $T_t$ is the trend component, $S_t$ is the seasonal component and $R_t$ is the remainder. The ...
  • 53.2k
2 votes

AR And MA Order

From your plots it appears that you might have a series with a periodic signal in it. The best way to check this is to look at the Fourier intensity of the time-series, which you can do using the <...
  • 102k
0 votes

Rolling autocorrelation vs whole series autocorrelation

You should have a look at locally stationary processes. They are defined as typical time series models (e.g. ARMA) but where the parameters of model change smoothly over (unit interval) time. In ...
0 votes

Does it make sense to increment by 1 the numerator and denominator in the MAPE to avoid division by 0?

Well, you can do this. I would very much recommend you take a look at What are the shortcomings of the Mean Absolute Percentage Error (MAPE)? The MAPE elicits forecasts that can be quite far away from ...
0 votes
Accepted

Regression Analysis of Time as Independent Variable in Experiment

Running a simple linear regression model on the 300 pairs is a viable option. However, as you suspected, it would presume that there is no difference between the individuals. So the next possibility ...
  • 8,694
0 votes
Accepted

Choosing the best Time Series Decompositions technique for International Trade

All of these are well-established methods. If one of them were clearly superior across a wide range of datasets, it would have superseded the others. Thus, which one performs best will depend on your ...
1 vote

Goodness of fit for exponential model in lmfit

I only have one (depending on the perspective maybe 2) measurements per timepoint Having only one measurement per timepoint brings you in a difficult situation. This makes it difficult to analyze ...
1 vote

Goodness of fit for exponential model in lmfit

I'd leave $R^2$ out since your nonlinear models under comparison may have a different number of parameters. I would, instead, rely on one of these three approaches: (a) use of information criteria ...
  • 1,497
1 vote
Accepted

Does the Augmented Dickey-Fuller test only consider first-order effects?

It is related to the concept of stationarity of AR(p) processes, for which all roots of the characteristic polynomial $$ \phi(z)=1-\phi_1z-\ldots-\phi_pz^p $$ must be outside the unit circe, see e.g. ...
0 votes

Aggregated time series or regression approach?

There's a problem with aggregating data (Approach 2) in this context: the survival estimate for an "average" set of covariate values isn't necessarily the average of survival estimates among ...
  • 69.1k
1 vote
Accepted

Can I use interrupted time series analysis on irregular reported medical data?

Electronic healthcare records provide longitudinal data that don't have the regular observation times of panel or standard time-series data. Even if formal methods of interrupted (regularly spaced) ...
  • 69.1k
1 vote

How to account for the pandemic in time-series forecasting?

If your time series exhibits 'anomalous behavior' in more recent observations, you necessarily have to remove or at least 'adjust' the data. Otherwise, your model performance will suffer for future ...

Top 50 recent answers are included