0
$\begingroup$

Disclaimer: I checked some similar questions but I could not find anything in particular that would work for my case.

I am dealing with a time series going from 2015 to 2023. The data points are the results of an aggregation of scores calculated on a European country's financial news (simplifying: I take financial news per day, perform sentiment analysis, and get a score per sentiment (n. of pos, n. of neg and neutral per day) and then aggregate it to get one data point per day).

Now, the model I built would work pretty well, if it wasn't for the huge Covid outlier: my target is GDP, and it's well captured, but in 2020 my sentiment curve goes way deeper compared to GDP. In fact, the country I am working in was particularly hit by Covid, hence there was a huge mass of negative news in 2020. This means I can't just remove the outlier or use the traditional methods, cause I would lose information, i.e. part of that info is important, cause GDP decreases considerably in that period, but not that much...

Now, I have tried changing the aggregation techniques to better deal with that. More specifically, I tried the following:

  • np.log(n_positive_news + 1) / np.log(n_negative_news + 1)
  • np.log((n_positive_news + 1) / (n_negative_news + 1))
  • -1 * n_negative_news

I also tried applying exponential moving average, rolling mean, and rolling sum. But none of these techniques helped a lot. Exponential moving average and the third aggregation technique improved results, but slightly.

  • Any suggestions on other aggregation techniques or in general, on how to deal with that outlier?
  • Train/test split: what do you think would be the optimal split? One that would include or exclude Covid in the training period? and why? for the moment, I am including it in the training period (train 01/01/2015-31/12/2020, test 01/01/2021-10/01/2023

Plot looks something like this (that drop is 2020, and the red line is my forecast). enter image description here

Improve this question
$\endgroup$
2
  • 3
    $\begingroup$ I guess I get what you’re saying, but having a model that works except for when major events happen strikes me as being akin to, “But other than that, Mrs. Lincoln, how did you enjoy the show?” (Abraham Lincoln was shot and killed while he was attending a play with his wife.) $\endgroup$
    – Dave
    Commented Mar 15, 2023 at 11:38
  • $\begingroup$ My goal is not to know how to predict major unexpected events, cause that is almost impossible. I want to find a way to smooth an outlier without getting rid of it completely, capturing the drop better. Cause the model itself captures the trend and the Covid drop in the first place. $\endgroup$
    – duecci
    Commented Mar 15, 2023 at 11:50

1 Answer 1

Reset to default
1
$\begingroup$

One approach I've taken with sales data is to identify the range which was affected by the pandemic, remove the actuals, and create dummy variables within the range using an ARIMA model to interpolate as described here: https://otexts.com/fpp3/missing-outliers.html

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.