Context and data

I am studying suicides among the military. I created a table that aggregates certain metrics (number of holidays, number of hours worked, etc...) for each officer, for each month preceding the suicide (up to 6 months). If the officer didn't commit suicide, I use the date of the suicide that took place in his company, so I can compare similar period of items. For companies that didn't suffer any suicide, I use an arbitrary reference date : July 1st 2018.

So it looks like : my data set

I created columns that contains the average of each numerical feature for the given period, both in the company where suicides took place, and both in the general population. That way I can detect cases where an officer worked a lot of hours while his colleagues were doing way less, or officers that were doing extra hours in a company where everyone was very busy, and the atmosphere may have been tensed.

So the question is , what test is more insightful for the task at hand :

  • a t-test to compare, for example, the average sick leaves amount of two independent populations : those who committed suicides, and those who didn't. The test could be on the average 6 months before the event, or 6 different test for each period.

  • a paired t-test on the population who committed suicide, where I would compare the difference between average sick leaves the month during the event, and 6 months before.

At the end of day, my objective is double :

  1. Identify risk factors for suicides
  2. Flag officers at greater risk of suicides.

My idea was to use t-test to detect which feature are significantly different from average population, and paired t-test to detect when the drift start to occur. Or put an other way, when could we be able to fire alarms.

Note : I think it doesn't matter for t-test but my 2 populations have big size difference. One is around 90 000 and the other around 100.

Because the population is very imbalanced, I am thinking I could obtain the same information as t-test by using the SMOTE algorithm to boost my minority class and then use a random-forest to study feature importance.

Am I making any sense ?

up vote 1 down vote accepted

If your goal is as stated, then I would use neither. The problem appears more complex, and I would not compromise it by using simplistic statistics.

  • you have repeated measures (multiple measurements for a given person)
  • it is a longitudinal study
  • there may be correlations across time series (e.g. number of work hours by end of December compared to number of work hours by end of January)
  • the assumption that only the period directly leading up to the suicide is predictive might be incorrect, as depression (which often causes suicide) is not an acute state

Given the last point, a t-test comparison between the two groups (no suicide vs suicide) seems a more reasonable approach, but you can't be sure whether the predictive power is hidden at earlier time points.

What are your group sizes? How many months of data do you have for each individual?

Moreover, before I would start doing anything with the data, I would remove a third of the samples and not touch them at all while you work on the remaining data set. Once you think you have figured out what your predictors are, you will use the test set to validate your findings, preferably in a blinded run. This is what we do in predicting disease outcomes.

I strongly suggest that you do that; it will strengthen your conclusions immensely.

POST SCRIPTUM:

We have a paper in press with a somewhat similar setup. The readout was a disease (about 100 cases and a couple of hundred people who remained healthy) and the predictors were molecular data collected during two years leading up to the diagnosis (or end of follow-up for the controls).

What we did was to use a machine learning approach (random forests), and treated each sample separately (i.e. sample from six months before was treated separately from sample 1 month before), and also tested the predictive power of different strata (for example, can we predict the onset based on samples collected more than half a year before the diagnosis?).

However, this may be dangerous (overfitting), so we took utmost care to avoid it. Among other precautions (like never splitting samples from the same person when doing k-fold) we evaluated our predictions on a blinded, separate validation test set.

Whatever else you plan, do yourself a favor and make this validation set, just make sure that it is stratified correctly (i.e. that there are no differences say in gender ratio or age between the training set and test set). You can always evaluate the full set (training + validation) post hoc.

  • 1
    In the last point it seems "wrong incorrect" may be a typo? Also why did you say "last month"? I thought OP's question hints that there are 6 months. Though I remain agreeing with you that even 6 month may not be enough. – Penguin_Knight Oct 11 at 15:22
  • Thank you @January for clarifying. Suicide /No suicide population size : 100 / 90 000 Like Penguin_Knight noted, I didn't assumed that the last month only is predictive, I am trying to know how early I can detect a change of pattern. Since multiple measurement at different times is not wise, then I can compare the 2 populations. I chose a 6 months window quite arbitrarily, I think I can go back 2 years. Should I compare over such a large period, or split that period in intervals and compare them ? So no oversampling necessary ? – dalyds Oct 11 at 15:45
  • I have added a description of my own experiences to the answer. – January Oct 11 at 16:15
  • 1
    To be sure I understood the setup of the experiment you described, I draw a schema Since you don't know where in time does the predictive power of molecule A lie, you insert a new row for each point of time. And since you don't know how early you can detect the disease, you try using only data from 18 months before, then another experiment with only data from 12 years before, etc . Am I correct ? Additional question, do you keep the period indicator as a feature ? – dalyds Oct 12 at 9:17
  • Yes, if I understand correctly, that is what we did. Basically, when doing k-fold, you must make sure that if you have a sample in fold-1 from patient N, all other samples from that same patient must be either removed from the remaining (k-1) folds and, possibly, added to the current fold. – January Oct 12 at 9:21

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.