-1
$\begingroup$

I have a dataset that tracks the time at which people go to sleep and when they wake up. This data is recorded daily for an entire year. So for each study participant, I have information for when they went to sleep on a given day, and when they woke up.

I am trying to find outliers in these data based on sleep schedule - which participants have sleep schedules that deviate from the norm? I would like to convert the sleep hours data into a distribution, so that I can see how their sleep schedule data is extreme or normal based on the distribution I construct - sort of like getting a Z-score.

However, I'm not exactly sure how to do this. My initial thought is to take the distribution of sleep start times and sleep end times for each day of the week and construct a distribution from that, but that feels inelegant. Is there a more general solution that you can help me identify that would tell me which participants have irregular sleep schedules?

$\endgroup$
9
  • 5
    $\begingroup$ You have a much richer dataset than a mere univariate distribution: you have an entire time series. Capitalize on that structure. $\endgroup$
    – whuber
    Apr 25 at 19:48
  • 1
    $\begingroup$ @whuber can you share some techniques that might be helpful? $\endgroup$
    – Daniel
    Apr 25 at 19:59
  • 5
    $\begingroup$ See time-series. It's a big subject. Sharing some typical data can help get people oriented to the kind of data you have and perhaps prompt useful suggestions. $\endgroup$
    – whuber
    Apr 25 at 20:01
  • $\begingroup$ “The” time at which people go to sleep does not sound like real data — real people often go to sleep, wake up, and go back to sleep. $\endgroup$
    – Matt F.
    May 4 at 13:08
  • 1
    $\begingroup$ We need to define our research questions. Short of that there are "no outliers". For example a person who sleeps on average 2.5h per night is an outlier in terms of duration; the same person though if they go to sleep at around 23:15 is not an outlier in terms of sleep "start", but someone who sleep 4:30-to-11:30 is. As people correctly mentioned your data also have a time-series structure as well as as other complex dependencies (e.g. a weekly effect should be there too, vacations, etc.). My advice: Forget the Stats for a moment. (cont.) $\endgroup$
    – usεr11852
    May 9 at 21:54

2 Answers 2

1
$\begingroup$

Strictly speaking (in my humble opinion), you cannot build up an empirical distribution from the data to then detect outliers in the data (sort of a chicken and the egg problem). You can evaluate the statistical leverage each observation has on an estimate to determine if it is influential.

However, as the comments pointed out, you have a nice matrix of time series data. How do you organize this unstructured dataset? Unsupervised learning! Personally, I would use the dtwclust package in R as seen in this wonderful tutorial here. Essentially, this package performs standard clustering methods but with a time series focused distance metric (as opposed to Euclidean). What this will pop out is groups of similar time series, hopefully with some luck and tweaking you find a nice decision boundary to identify what you call irregular sleep patterns.

$\endgroup$
0
$\begingroup$

Initial thought: Asleep versus Awake at time $t$ is a binary outcome.

Let $ x_{it} \mid \pi_t \sim \text{Bernoulli}(x_{it}\mid\pi_t)$, then have $\pi_t$ be a probit transformation of a Gaussian process (to take advantage of the time-dimension).

Then you have a series of $\pi$'s, and can caculate likelihood for an individual as the product of those bernoulli responses against that series of $pi$'s. Individuals with a low likelihood will be deviations from the norm.

Going one step further: You could cluster people by their time schedule. This could be having multiple GP's representing (probit) probability of being asleep, then people are assigned to one of those GP's.

Alternatively, if you just want to find outliers, you could easily just do k-means on the awake vectors.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.