2
$\begingroup$

I have two processes, lets call them A and B. I want to infer whether their population means are equal for their experimental outcomes. Both of these processes generate 1s and 0s only. That is, the outcome is binary.

My situation is similar to using two different processes to flip a coin a few times. The idea is to compare the experimental outcomes of these two processes.

I will show two simple cases to illustrate the problem. In the first case, function t.test() of the R language's stats package runs just fine when t-testing these two outcomes:

> t.test(rep(1,5), c(rep(1,4),0))

    Welch Two Sample t-test

data:  rep(1, 5) and c(rep(1, 4), 0)
t = 1, df = 4, p-value = 0.3739
...

IN the second case, for some reason the t.test will not run, and an error is returned, when t-testing these two outcomes:

> t.test(rep(1,5), rep(1,5))
Error in t.test.default(rep(1, 5), rep(1, 5)) : 
  data are essentially constant

I will need a workaround to effectively run a t-test in case 2. I do not have a human available to always look at the data before deciding whether to run the t.test.

I have done some research and saw that some people write exception handling code in these situations. Do I really need to write my own t.test implementation, such as by wrapping exception handling code around the standard stats t.test and then doing some things depending on various things in the data (I don't know what exactly)? If there is a different implementation available somewhere then that might be the best solution. I am not confident in my ability to write new t.test implementations accurately.

These two cases represent entirely possible and realistic experimental data in my situation.

Edit: Additional background seems to be interesting to the community. Here you go.

Thanks to all for so many questions! Let me see if I can provide some of the context the community seems to be wanting. In the experiments in question, we might have a scenario where my users wish to infer whether technique A for studying for SAT exams is better than technique B. IN this case there will be (effectively) continuous outcomes in the form of the mean SAT score by groups A and B. In a different scenario my users want to analyze, there may be an AB test of a web site wherein the user wants to try to get more people to open the email and to do so the firm sends out an A subject on some emails, and a B subject on emails to different group of customers. Whether the customer opened the email is a success, encoded as 1, failure being the recipient did not open the email and this would be encoded most naturally as 0. A comparison of the group means would be interesting, as a significant difference in mean success rate of opening emails could cause the user's firm to change they way they write email subject lines. Hope this helps.

$\endgroup$
  • $\begingroup$ gung, Thank you helping me with formatting the code and the additional tags! $\endgroup$ – GeoffreyA Jan 14 '15 at 15:59
  • $\begingroup$ Although this question appears to be asking for code, the real problem is a statistical misunderstanding. I believe it is on-topic for CV. $\endgroup$ – gung - Reinstate Monica Jan 14 '15 at 15:59
  • $\begingroup$ Is there a reason that you're trying to do a t-test with binary results instead of doing proportion test? $\endgroup$ – Duncan Jan 14 '15 at 16:00
  • $\begingroup$ Duncan, yes there is a reason. The data will not always be binary, but sometimes it will. $\endgroup$ – GeoffreyA Jan 14 '15 at 16:01
  • $\begingroup$ gung, I need to decide if the means of two experimental outcomes are the same or not. $\endgroup$ – GeoffreyA Jan 14 '15 at 16:02
2
$\begingroup$

Based on your comments, I think I have an answer for you. It seems like for the processes you currently have, you have binary outcomes, but that there will be other processes which will not. You would like to use a t.test to analyze the results.

The issue that you're having is that the t.test is not designed for use on binary outcomes. It's designed for use on continuous outcomes. The formula for a t-test is:

t = (x1-x2)/(s*sqrt(2/n))

s represents the pooled standard deviation. You're encountering the error you have because the two samples in part 2 have no standard deviation - they're constant. This means that to generate the test statistics you're dividing by 0. Which, you know, makes the universe explode. If s is too small then the t value will be huge no matter what the difference between the two sample means (unless the sample means are identical, as in your example). The t-test becomes a moot point because either the sample means are identical, in which case the p-value would be 1, or the sample means are different by any amount, in which case the p-value would be so small as to be functionally 0.

In other words, you cannot effectively run a t-test in case 2. I'd be interested in seeing the sources you have found where people do workarounds for this.

Your examples at the end do help a lot. In the success/failure example (which is analogous to the examples you provided) you should use a proportion test instead of a t-test. In the SAT test, with continuous scores, you should use the t-test. It is highly unlikely that you would encounter a situation where there was so little variability in the SAT scores that you would see this error happen again. It's possible, but highly, highly, highly unlikely except with a very, very small sample size. In which case my advice is to increase the sample size :)

$\endgroup$
  • $\begingroup$ Interesting. However, one of the two samples in Case 1 is identical to those in the other case. A process particularly when run for just a few trials can easily output the same result. What I am saying is that your claim that the "two samples" have no standard deviation is interesting because the case that runs fine also uses ONE of those same "two samples" ... and it runs without error. $\endgroup$ – GeoffreyA Jan 14 '15 at 16:27
  • $\begingroup$ It is entirely possible that very small sample sizes will be the outcome in experiments which cost a lot of money or time to perform. THis needs to be handled in my case. $\endgroup$ – GeoffreyA Jan 14 '15 at 16:28
  • $\begingroup$ The pooled standard deviation is calculated using the following formula: s = sqrt(1/2*(s1^2+s2^2)) where s1 and s2 are the standard deviations for the individual data sets. So in the first example, the t-test will run because the pooled standard deviation is not equal to zero (the s2 isn't zero, so s isn't zero). In the second example since s1 AND s2 are zero, s will also be zero. $\endgroup$ – Duncan Jan 14 '15 at 16:59
  • $\begingroup$ Duncan, I am choosing to select your answer as the winning answer. Many people have given me assistance today, and I wish the web site would let me give thanks to everyone who took the time to supply advice, but the web site forces me to pick just one. Thanks for all your help everyone, and especially Duncan! $\endgroup$ – GeoffreyA Jan 15 '15 at 17:07
  • $\begingroup$ Could somebody please upvote my question at least one upvote? There was some interest today in my question. I am going to do my part by formally selecting the best answer. I hope someone thinks to do their part as well. Thanks everyone. It was great to work through your ideas. $\endgroup$ – GeoffreyA Jan 15 '15 at 17:09
4
$\begingroup$

A t-test is appropriate for continuous data that is normally distributed. If you have binary data, you need to do a proportion test or binomial test.

You point out in the comments that you don't know if the data are always binary or continuous. In R it is not complicated to write a code to automatically determine which type of data you have, and then run appropriate statistics.

For example:

group<-c(rep(0,100),rep(1,100))
continuous<-rnorm(n=200,mean=1+2*group)
binary<-rbinom(200,1,prob=.2+.3*group)


test<-function(data) {
  if (length(table(data[,1]))==2) {
    chisq.test(table(data[,1:2]))$p.value
  } else {
  t.test(data[,1]~data[,2])
  }
}

test(cbind(binary,group))
test(cbind(continuous,group))

I used a $\chi^2$ test not a proportion test because I always forget the prop.test syntax, but you can replace that easily. The results should be the same.

The case where you have no variability in the outcomes is going to be tricky, because in general the probability of a binary variable being 0 or 1 is undefined. In a small sample like you provided in the example, it is possible that you will have all 1's or 0's - unfortunately, I believe that you cannot calculate statistics from that. Someone smarter than me may give you better guidance on what to do with that, but you may need to do a simple proportion comparison without a statistical test.

$\endgroup$
  • $\begingroup$ Interesting. Some processes I must analyze might have relatively rare success rates, so there might actually be all 0s for outcomes in say 10 or even 100 trials. I would not be able to infer accurately in such imbalanced outcomes, whether the process was binary. However, in my case I am able to directly ask the user if the outcomes are binary or continuous. This direct input from the user will work fine. Thank you these ideas. $\endgroup$ – GeoffreyA Jan 14 '15 at 16:43
  • $\begingroup$ The appropriate data for a t-test are normally distributed, not student-t distributed. If you are estimating the SD from your data, then the test statistic will be distributed as t instead of z / normal, but this is different from saying your data should be distributed as t. $\endgroup$ – gung - Reinstate Monica Jan 14 '15 at 16:47
  • $\begingroup$ Correct, sorry about that, brain slipped. Edited. $\endgroup$ – robin.datadrivers Jan 14 '15 at 16:48
  • $\begingroup$ Re constant data in a chi-squared test, R will throw a warning, but the test will still execute. You really wouldn't need a test though. A simple logical detection rule, akin to what you already have, would be appropriate. $\endgroup$ – gung - Reinstate Monica Jan 14 '15 at 16:51
  • $\begingroup$ Gung, sorry, but logical automatic detection is not appropriate in my application, for the reasons I already stated, again, because there can be a rare success rate, that is, highly unbalanced outcomes, and furthermore, there can be small number of trials being performed by my users. $\endgroup$ – GeoffreyA Jan 15 '15 at 17:02
1
$\begingroup$

If the outcomes of both processes is binary, I'd recommend using prop.test to determine if the processes have significantly different outcome probabilities. The documentation for it can be found here: https://stat.ethz.ch/R-manual/R-patched/library/stats/html/prop.test.html

$\endgroup$
  • $\begingroup$ There will not be strictly binary process outcomes. The examples I gave are specifically binary however. $\endgroup$ – GeoffreyA Jan 14 '15 at 16:08
  • $\begingroup$ I'm sorry, I'm confused then. You state in your question that both processes generate only 1s and 0s, is that not the case? $\endgroup$ – Adam Bosen Jan 14 '15 at 16:12
  • $\begingroup$ Adam, the examples are outcomes of binary processes. The code will need to handle binary and continuous processes. I feel it is better to start small when solving problems. $\endgroup$ – GeoffreyA Jan 14 '15 at 16:17
  • 1
    $\begingroup$ Ah. In that case, I'd recommend having a conditional that switches between the two tests as appropriate. Could you include a check of whether or not length(unique(c(A,B))) == 2? to determine if the processes are binomially distributed, then run prop.test if it is, and t.test if it is not? $\endgroup$ – Adam Bosen Jan 14 '15 at 16:25
  • $\begingroup$ That's exactly what my code does in my post. $\endgroup$ – robin.datadrivers Jan 14 '15 at 18:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.