Summary: The last formula boxed in red (which is a modified log likelihood from logistic regression) is a special non-differentiable loss function that is adapted to contain a Bonferroni correction in the loss function. The function is non-differentiable with respect to theta which is a problem for gradient descend and thus for using it to train regressors and neural networks. If it could become differentiable the function could optimise a classifier specifically designed to increase the power of multiple hypothesis testing procedures in the case there is useful additional information that goes with the tests and would then lead to more significant findings in cancer research. my question thus is how to make the formula boxed in red differentiable with respect to theta (preferably with some theoretical justification! ;P ).


For some research into the Biology of Cancer (details excluded for clarity sake) I devised a method to mitigate the issue finding important drivers of the process, which is in essence an issue of Multiple Hypothesis testing.
To do this I combine Machine Learning approaches with Multiple Hypothesis testing correction and aim to end up with valid/significant drivers of the cancer process.

The Challenge

Consider the following situation: One is supplied with a bunch of independent statistical tests that each have some additional information that comes with them as displayed in figure 1 below. For these statistical test the goals is to perform Multiple Hypothesis Testing (MHT) and call significance for all the 7 tests, using Bonferroni correction:

Figure 1: A bunch of independent statistical tests Figure 1: Seven independent statistical tests, with additional information

Confronted with the situation in figure 1 one would perhaps like to make use of the fact that all the tests with additional information 'A' attached to them have very low p-values and in a way pre-select them for Multiple Hypothesis Testing (MHT). In this example case the number of statistical test is small (7), but there are of course situations were one has to correct for many more (>10^6). The use of the additional information could in those cases really increase the overall power of the procedure.

Note that in this case the additional information is comprised of only one bit of information (A/B), but could just as well be a complex feature vector (this is where the Machine Learning comes in !)

Possible Solution 1

Given that the statistical tests are independent (note: in my case they actually aren't fully independent but that is perhaps something for another post) one could use Machine Learning algorithms to try to predict the -log10(p) value for for instance test 1 by training a linear regression model on tests 2 till 7. One could do this for all the tests, effectively a leave-one-out cross validation and arrive at an unbiased prediction of the -log10(p) and use this information to select the really juicy 'A' type tests over the 'B' type, thereby decreasing the number of test in the MHT, increasing the power.

--> In machine learning terms: The additional information becomes the feature vectors (x), The individual tests are the samples and the -log10(p-values) of the tests become the labels (y). After this the tests/samples can freely be distributed in training and test sets for cross validation because all the samples are stated to be independent.

There is only one issue with the approach above: It does not optimize for the desired end result being to find as much as possible significant results. In small sets like here this does not constitute a problem, but in large MHT problems with only a few small p-values this leads in effect to regression methods fitting the large 'uninteresting' part of the distribution, leading to poor performance on the small p-values it should actually focus on. A possible solution could be weighting of the samples, but this is problematic since it is not clear what an optimal weight should be.

Therefore another solution is needed that results in a model that actually optimizes the desired end result, being the largest possible number of significant results after MHT correction. This is the target of the next possible solution (2).

Possible Solution 2

To have a Machine Learning algorithm try to maximize the number of significant results on a test set, the method should be supplied with an appropriate cost function. If we assume for a moment that the dataset essentially consists of 2 classes being the significant tests and non significant test and given that these classes were known one could maximize the following log likelihood (cross entropy) as cost function to arrive at a model:

Cross Entropy

Which is a nice differentiable cost function that can be used for gradient descend. In this equation n is the total number of samples. yi represents the i'th example in the training set which can have the value 1 for the 'significant class' or 0 for the 'non-significant class' and htheta(xi) represents the output by the model, which has the following structure:

Hypothesis h_theta(x)/Model

With Theta being the model parameters and x being the feature vector. Now,.. If one reckons that the second term of the loss function (1st eq.) is concerned with assigning a cost to wrongly classified samples of the non-significant class removing it makes sense because the classification of non-significant samples is not of interest. One could also take the Bonferroni MHT correction into account in the cost function and one could modify the cost function into:

Modified loss function

Indeed the strikethrough is not part of the formula anymore. I{A} here is an indicator function that is 1 if A is true and 0 otherwise. In the brackets of the first indicator function above one can see the Bonferroni correction for the i'th p-value, in fact determining its class (sig. or non-sig). This class label also depends on m (the number of statistical test performed) which is determined by the sum of the next indicator function over the sample hypotheses. If htheta(x) reaches over 0.5 it is more likely according to the model that the sample is significant so it should thus be included in the final MHT correction. (note: The value of m/the cost function should eventually be corrected somewhat for the fact that the procedure excluded the one sample in the leave-one-out cross validation).


This last MHT cost function is a function that in fact optimizes the desired goal of finding the largest possible number of significant tests, but because of the indicator functions it is now non-differentiable, which is a big issue of gradient descent optimisation. The Final Challenge is to modify the MHT cost function to make it differentiable in such that it can directly be used for (stochastic) gradient descend (SGD). (One of the reason this is desired is because of the fact that that number of test is VERY large, but that SGD implementations on GPUs can crack the problem. Another reason is also the fact that it could be used to train a deep neural network, which has the benefit that it can model the highly non-lin representation that are likely to be found in the data.
I think that this cost function would really help to improve the number of discovered cancer drivers.

Hope you guys can help me out. Really curious though :) Perhaps there is even a better approach to this problem than my suggested path.

  • $\begingroup$ What's your question? Do you want to discuss your idea? As far as I understood your objective, you want to improve Bonferroni multiple testing by some machine learning stuff. This is not appropriate. First, for a lot of specific questions there are already adjustments for multiplicity that outperform Bonferroni in their respective setting. Secondly, for confirmatoric trials, the choice of a multiplicity adjustment procedure must not depend on so many arbitrary parameters as (explorative) machine learning procedures do. $\endgroup$ – Horst Grünbusch Aug 14 '14 at 11:50
  • $\begingroup$ (Made some edits to the post): My question is how to make the redboxed formula differentiable in a way that is theoretically justified. I see your point about the non-appropriateness of the machine learning stuff. I guess the important question is: is it baised. I say it is not, because all the test are independent, so one can really 'abuse' all the information of one test and use to make predictions about other tests. What is not allowed is to use the test's p-value information on itself! that is why the cross validation must be done. $\endgroup$ – MJW Aug 14 '14 at 15:03
  • $\begingroup$ Some nomenclature: A test is called biased iff for some parameter $\theta \in \Theta_0$ (Hypothesis) the rejection probability exceeds the one of a $\theta \in \Theta_1$ (Alternative). Another point is that a test might be liberal: Type-I-error exceeds predefined $\alpha$. This can happen if arbitrary parameters play a role. By the way, if all the tests are really independent, the best single step multiplicity adjustment is known as Sidák-correction. $\endgroup$ – Horst Grünbusch Aug 15 '14 at 10:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.