I have been collecting data of each tweet being sent out for each article. The data is like this.

created_at, user_id, article_id
xxx, x, x
y, y, y

When I plot the graph for each hour and minute I get non-linear graphs because of the nature of time-series

What I'm wondering is what algorithm should I use to try to fit in this kind of graph to predict number of tweets happening in an hour or minute interval.

enter image description here

enter image description here

Is this possible?

This is what I have done to produce the graph.

apple.created_at = pd.to_datetime(apple.created_at)
apple['created_at_minute'] = apple.apply(lambda row: row['created_at'].minute, axis=1)
apple['created_at_day'] = apple.apply(lambda row: row['created_at'].day, axis=1)
apple['created_at_hour'] = apple.apply(lambda row: row['created_at'].hour, axis=1)

Smoothing (loess, MA and variations) is one option of trying to make sense, as the other responder suggests.

Basic time series analysis (look up some basic econometrics material) could help too, if the data is autocorrelated (so the number of tweets in hour T predicts the number of tweets in hour T+1).

And it's highly likely that they are autocorrelated: an article hits Twitter in hour T, goes viral, which makes the number of tweets in hours T+k jump up as a consequence.

However, if all you have is data on tweets and nothing else, it may be hard to estimate one general model for the number of tweets en masse. As you can see on the graph yourself, it looks very very irregular. Few very big spikes, with most of the time being rather quiet.

If you could somehow group the articles into types/genres, that might help you condition the model and bring some order into it. Or bring in more data/variables.

Or, and now I am just wildly speculating, given the nature of the data, it could be an example of some highly skewed distribution (to the left, so most of the time, very few tweets are observed), but sometimes, a viral thing happens and it explodes. Which is realistic enough I guess. But you'd also need to model some autocovariance there, as one freak "draw" conditions the distribution in later times (so shifting it more to the right, making it more "normal"ish...).

It could be an interesting exercise, but I really don't know how you'd rigorously go about fitting such model (if it were, e.g. some sort of Pareto distribution or something).

Now that I think about it: logs! A log-normal distribution is skewed to the left, and if you take the log of it, you get a normal. So perhaps try taking logs of the counts (add 1 to cover 0 tweets) and fitting an autocorrelated normal distribution on the logs?

Sorry for the wild speculations :D


If you want to smooth out the graph, look into moving average, exponential moving average, etc.

If you want to make a prediction, the data should follow some underlying patterns(which it should, in this case). Then you can use ARIMA to find underlying seasonal/cyclical patterns.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.