4
$\begingroup$

My problem is as follows:

I have 2 measuring devices, both using different measuring techniques, each trying to quantify properties of objects passing their measurement area. The devices measure the objects at the same time. For this particular problem I'm only interested in the number of objects each device is able to measure.

For each time unit, which is set to one hour, the devices summarize the number of objects they've measured or "seen", like the below example data:

Time    Device1 Device2
00:00   58      47
01:00   38      52
02:00   12      13
03:00   0       2
04:00   23      2
....    ..      ..

I want to compare the mean difference of these 2 datasets

The first thing that came to my mind was to use a 2-tailed t-test to determine if there is a significant difference in means between the two datasets. Using the following parameters in Excel:

Hypothetized Mean Difference = 0
Alpha = 0,05

Question: Is the t-test even suitable in this particular setup? And if it is, is my data paired or independent? If not suitable, what other method could be used?

I guess what confuses me is the fact that I have 2 measuring devices measuring the same event, instead of one device measuring different events. I haven't dealt with statistical problems in ages, so my knowledge is a bit rusty. Any help would be greatly appreciated.

Edit: As has been pointed out in the answers, the question that I should be asking is: do the devices produce equivalent measurements? For clarification, the example data provided is fake, but closely resembles the measured data. The measured data is not normally distributed.

|cite|improve this question
$\endgroup$
  • $\begingroup$ I would consider this paired. $\endgroup$ – Affine Jul 24 '14 at 14:59
  • 1
    $\begingroup$ I might be missing something here, but could you just correlate the counts from the two devices, and use that ($r$) as an indicator of how well they match, or badly they mismatch? $\endgroup$ – Eoin Jul 24 '14 at 17:14
  • 1
    $\begingroup$ @Eoin correlation is unsuitable as a measure of 'how well things match'. For example if the second machine always counted only 10% of the first machines values, they'd be essentially perfectly correlated, but very different in value. $\endgroup$ – Glen_b -Reinstate Monica Jul 25 '14 at 1:39
  • $\begingroup$ First thing: identify clearly the question you want to ask of the data. Second thing: these are count data - once you know what questions you want answers to, use a procedure that will be able to deal with the special features of counts (you might, for example, anticipate with counts that variability will not be constant as the mean changes). $\endgroup$ – Glen_b -Reinstate Monica Jul 25 '14 at 1:41
  • $\begingroup$ Some of the comments here seem to be relevant. $\endgroup$ – Glen_b -Reinstate Monica Jul 25 '14 at 1:47
4
$\begingroup$

The issue is less with can one perform the statistical test (paired t test) for difference?, and more with what is the inference you are drawing from the tests results?

From what you describe, both measurements are made on the same objects at the same time, your test is trying to support inference for whether or not the equipment are making different measurements. A few thoughts:

  1. If you are interested in whether or not the two equipments make equivalent measurements—a separate statistical question than whether they make different measurements—you should also be performing equivalence tests, and combining your inferences from both tests.

  2. If you are interested in a deeper understanding of how the two measures perform, perhaps a regression model, such as OLS regression, which will tell you trend and strength of association, as well as evidence of association existing.

  3. From the tiny picture you give, your data do not really look normally distributed, which is one of the t test's assumptions. Perhaps a distribution-free test, such as the sign rank test would be appropriate?

|cite|improve this answer
$\endgroup$
  • 1
    $\begingroup$ +1 though a couple of points that might bear a mention - OLS regression assumes constant variance, but we wouldn't usually expect to see constant variance in count data with changing mean. Non-constant variance is often more of an issue with normal theory procedures (like t-tests and tests of regression coefficients) than normality. It might be better to consider using GLMs (ones suited to counts, at least) for either a plain comparison of means or for a regression-like model. $\endgroup$ – Glen_b -Reinstate Monica Jul 25 '14 at 1:42
  • $\begingroup$ @Alexis thank you for a clear and concise answer! As I've thought about it, the question I mean to ask is if the measurements are equivalent. Also, I completely overlooked the fact that the data is not normally distributed and that it is a requirement for the t-test. I will update my question to reflect this. The sign rank test looks promising, so I will give that a try. $\endgroup$ – marb0 Jul 25 '14 at 8:38
  • $\begingroup$ marb0 An assumption of the signed rank test is that the data are continuous. Counts are not continuous. $\endgroup$ – Glen_b -Reinstate Monica Jul 25 '14 at 9:52
  • $\begingroup$ @Glen_b Where are you getting the continuity assumption for the signed rank test? $\endgroup$ – Alexis Jul 25 '14 at 16:25
  • $\begingroup$ @Alexis The derivation of the distribution of the test statistic is predicated on all permutations of the signs over the ranks 1:n being equally likely. Without it the distribution being continuous, the distribution of the test statistic is wrong. If there are few ties it's possible to make a reasonable approximation. Under heavy ties, such as may occur when you have count data with small expected values, this is not generally the case. If one performs a permutation test on the actual ranks, this is not a problem, but I've only ever see one person actually do that. $\endgroup$ – Glen_b -Reinstate Monica Jul 25 '14 at 17:48
3
$\begingroup$

Since you are counting events in fixed time periods, a Poisson model comes to mind. In turn, that suggests that a more suitable analysis might be obtained by square-rooting the measurements before you do the $t$ test. (The square root is the variance-stabilizing transformation for data where the variance is proportional to the mean, as is true of the Poisson.) And yes, it should be paired, because you are collecting those data in the same time periods.

However, is it also true that the Poisson rate parameter (expected number of events per unit time) is changing from one time period to the next? It sure looks like it from the example data. In that case, that $t$ test is based on an underlying distribution that is a mixture of several distributions with different scale parameters. That distribution is pretty heavy-tailed. That usually tends to make the $t$ test conservative. In this case, the square-root transformation is a less obvious choice, but it will still at least make the distributions more symmetric.

I appreciate @Alexis for pointing out the possibility of doing an equivalence test. It is the right way to establish with confidence that two things are the same (merely failing to find significance is the wrong way). I really wish that this topic was in the intro stats textbooks, as it isn't that hard to do. The hypotheses under test are $H_0: |\mu_d| > \delta$ versus $H_1: \mu_d \le \delta$, where $\mu_d$ denotes the true mean difference (e.g., of $\sqrt{y_1}-\sqrt{y_2}$), and $\delta$ is a specified threshold for equivalence. To get $\delta$, don't look at the data, and instead, think about how different two measurements can be and still be considered practically the same.

The procedure (due to Schuirmann) is to reject $H_0$ at level $\alpha$ if the $(1-2\alpha)$ confidence interval for $\mu_d$ lies entirely in the interval $(-\delta,+\delta)$. For example, for a .05 significance level, you would use a 90% CI. This is the same as saying that two one-sided $t$ tests against the upper and lower thresholds are both rejected at level $\alpha$. The heavy-tailedness will make this test conservative too.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ How about a paired t-test on the square-root transformed measurements? This will account for the non-stationarity of the distribution. $\endgroup$ – Andrew M Aug 20 '14 at 0:22
-1
$\begingroup$

why you do not use Bland-Altman plot? it plots the difference between two measures versus the mean of measures...

|cite|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.