Prove or disprove that rating worth in online chess depends on the time of the day

Question

Crosspost from Chess.SE.

TL;DR

I feel that ratings on a chess website are worth differently depending on the time of the day, and I want to validate the claim. I have a complete history of games played last several years. How should I formally state the hypothesis? What methods can be used to confirm or reject it?

Rating system

Various online chess sites use Elo or Glicko rating system. Briefly, each player has a numerical rating. The greater the difference in ratings of two players, the higher the probability of the higher-rated to win. The winner gains some rating points, and the gain depends on the rating of his opponent (win against the higher-rated opponent gives higher gain). Same for the loser. Exact formulas are available on Wiki pages by the link above, but they are not directly relevant.

Regarding the magnitude, here's the rating distribution for bullet (super-fast) chess at lichess.org: 600, 1175, 1450, 1775, 2800 for percentiles 0, 25, 50, 75, 100. Average rating gain after a game of two equally rated opponents is about 5 points.

Story

Once I noticed that my average performance varies depending on the time of the day: e.g. at the morning I can lose 50 rating points after a dozen of games, but I will easily regain it at the evening. One possible explanation is that my brain may work worse at the morning because of sleepyness or something. But can the reason be not me but my opponents?

Imagine two countries, A and B, in different time zones. Players from A and from B tend to play chess at the evening; let it be 00:00 UTC and 12:00 UTC respectively. And, for some reason, players from A are generally stronger than players from B. In the extreme example players from different countries never meet each other, so their ratings establish independently. So even if two players have similar strength, the player from B will have higher rating (because he is stronger among his population).

In this setting, if I play at 00:00 UTC I will meet opponents from A, and if I play at 12:00 UTC I will meet opponents from B. Players of my rating will have different absolute strength in these cases.

Formalization

I would like to formalize the claim and to find a method to validate it. Here's the best I could think of:

Assume that all players have some absolute strength, measured in the same units as the rating¹. The outcome of each game is determined with probability according to Glicko formulas with regard to players' absolute strengths. Such system is stable enough, i.e. the rating of each player always fluctuates near its expectation. However, since the graph is not complete, the rating doesn't necessarily equal the absolute strength.

Let $R_n(x, t)$ be the random variable — the rating of a player of absolute strength $x$ after playing $n$ games at time $t$ o'clock UTC, and $R(x, t) = E\lim\limits_{n \to \infty} R_n(x, t) / n$ — the expectation of player's rating at the infinity.

Hypothesis: $R(x, t) = R(x)$, that is, $R(x, t)$ does not depend on $t$.

Question 1: Is this wording good enough for a formal hypothesis?

¹ The absolute strength might be defined as the expectation of one's rating if every pair of players meets infinite number of times. However, I'm not sure whether this expectation depends on the order of games played so abstain from defining it this way.

Methods

I tried developing methods that can help rejecting the hypothesis; hardly any of them could confirm it.

• Clustering. We build a multigraph with players as vertices and played games as edges. Then we cluster it somehow: either with some well-known algorithms, or ad-hoc like, say, dividing the clock into 24 hours, computing for each player the frequent hour of play and putting him there. Next we use the approach from this answer: assume that ratings of each cluster are shifted by a constant $\theta_{cluster}$ and estimate these constants with maximum likelihood or something. More elaborated approach, as described in the same answer, is to assume that rating shift itself depends on rating, so is a function; these functions also seem estimatable.

  However, there is little hope that clustering works. Unsupervised approach may fail since the graph seems too tightly connected. Clock-wise approach does not consider the fact that players may tend to play more than once a day (e.g. at the morning and at the evening).

• Ordering. We somehow order all players in a circle such that in general everyone plays more games with neighbours than with distant fellows. Then we may introduce parameters similar to the previous approach: for example, $\theta_s$ is the rating shift for the $s$-th player in a row. Add some regularity ($|\theta_s - \theta_{s+1}| < \varepsilon$) and use MLE once again. Pitfalls are similar: no guarantee that reasonable circular ordering even exists. And if it does, it can be hard to compute.

• Person-wise analysis. Look at the history of a certain player. If average gain is greater at some point hour, then likely rating points worth less at that point. However, if we take the average gain for all players... we get zero, since Glicko is (almost) zero-sum. So, sounds good, doesn't immediately work.

Question 2: Could you help me formulate the hypothesis and suggest any methods to validate it?

Answer 1

While this might seem like a complicated problem, you can test this hypothesis in a pretty straightforward way using logistic regression. I'm assuming you do have access to the ratings of each player you face, and your own rating.

First, prepare your data. You'll need columns encoding the difference in ratings between yourself and your opponent for each match (let's call this $\delta$), the outcome (win or loss), and the time of day. Time of day should be binned. The size of the bins will depend on the data you have, but bins of 3 hours each sounds like a reasonable starting point.

Then, fit your baseline logistic regression model to this data:

$$ \text{Log-odds(Win)} = \text{Intercept} + \text{Slope} \times \delta $$

This means the log-odds of you winning a match depends on a) an intercept, which represents the log-odds when playing an opponent of the same ratings as you ($\delta = 0$), and the slope represents the change in odds as $\delta$ changes. This model is the null hypothesis: nothing is affected by time of day.

Next, fit the alternative model:

$$ \text{Log-odds(Win)} = \text{Intercept}_{\text{Time}} + \text{Slope} \times \delta $$

Here, the intercept term is estimated separately for each time bin, meaning that your chances of beating an opponent who has the same official rating as you can vary throughout the day - there is a main effect of time.

Finally, you can compare the two models. If the alternative model fits the data significantly better than the null model, you can reject the null hypothesis, and conclude that your hypothesis is correct.

In R code, this analysis would be something like the following:

null_model = glm(win ~ delta, data = your_data, family = binomial)
alt_model = glm(win ~ delta + time_window, data = your_data, family = binomial)
anova(null_model, alt_model)