How to test for statistical difference for discrete data

Question

I'm designing an experiment to evaluate reagents used to develop fingerprints. There will be a variety of objects with different surfaces, 2 different reagents, 3 different fingerprint donors (prints deposited intentionally on the object), and 3 human graders. Photos will be taken of the developed prints on the objects for the grader to visually evaluate.

I've created a 5 level visual standard, similar to a Likert scale, to use for grading both the print development and the background development. High contrast (strong print development with minimal background development) is the ideal situation. For the visual standard, there are separate scales (0-4) for the print and the background, with visual representations of each level for both.

From what I understand, the data generated from this experiment would be discrete. I'm unsure whether it would be considered ordinal, interval, or ratio. It seems like it would have more information than just a simple ranking (for ordinal) because of using the visual standard. I think it makes sense to say there is an “absolute zero” because there will be examples of no print development and no background development. Does this mean the data would be considered a ratio subtype? Or is there another datatype that I’m not aware of that is more appropriate?

I’m also unsure of how to treat the dual grading scale results (for the print and the background). This doesn’t seem like the classic “before” and “after” medical study scenario of paired data. But I’m not sure if the dual grades would be considered “paired” in a different sense. Perhaps I need to make a single “combined” grade using the ratings for the print and the background, but I’d like to keep them independent, if possible.

The most important question to answer for the experiment will be whether there is a statistically significant difference between the two reagents (which reagent produces the highest contrast prints). I’m assuming the data will probably not be normally distributed, so I’ll need to use a nonparametric test. I don’t have a statistical software package to use, but I do have Excel.

I’d appreciate hearing recommendations for the type of hypothesis test to use (with the consideration of doing the calculations with Excel), a rough estimate of how many replicates I’d probably need, how to interpret the results of the hypothesis test , and anything else that is important to consider to draw meaningful conclusions.

Answer 1

I would agree with most of what @Emma Jean says but a few extra points to consider:

1) Regression models are actually quite robust to things like the Likert scale, so you don't need to rule them out - see https://link.springer.com/article/10.1007/s10459-010-9222-y. However, I am not suggesting you use this.

2) What many in sensory science do is give people a line of set length with no number (see this). One end of this line would be perfect print development and the other end would be absolutely no print development. You then have the observer mark along this line where they think the current sample lies and repeat for your other measure background development. You can then simply use a ruler to measure how far along the line the value is, thereby creating a continuous, albeit bounded, measure which conforms (close enough) to the assumptions.

3) If you are going to have two measures (print and background development) there are two ways forward:

• i. You create two separate regressions as @Emma Jean suggested. If you do this you will want to correct for multiple comparisons to keep your family-wise error rate at what ever you designate as statistical significance (i.e. 0.05)- you can use false discovery rate, Bonferroni's least significant difference, Holm's, Tukeys, etc . This is quite straightforward to do in R, and some can even by hand. Which one you choose is dependent on your question and greater context, but probably a discussion for another thread. On that note, there is a R package for doing all of this and you could likely find loads of help on stackoverflow for carrying this out! It is a good learning opportunity and I suspect excel will struggle to carry out the kinds of tests you want. Especially, the mixed-effects models that Emma mentioned which I think you will probably want to use.
• ii. You carry out a multivariate analysis of variance - this is where you have two response variables and you produce p-values for both simultaneously, thereby reducing the number of p-values you produce, avoiding harsher corrections. The catch is that the interpretation is a little more opaque than with individual regressions. This would however probably be more meaningful for you given the aim of your study!

4) The final thing you need to consider is practical significance. Statistical significance is all well and good but it tells you nothing about whether the difference is actually meaningful in the context of the real world. This is something that you have to decide on using your subject matter knowledge to pre-define what "size" difference has practical meaning.

Once you have figured out the effect size you want to be able to detect you can use a power analysis (there are free online generators for this that you just enter the value you have!) and it will tell you how big a sample you need.

Answer 2

I wouldn't call myself an expert in this realm but since it's the internet, here's my 2 cents...

You have a somewhat complicated experiment and Excel will not cut it here. I recommend you download SPSS (if you are not very comfortable with programming) or R. SPSS is not free but many university libraries or labs will usually let you access it for free. R on the other hand is free but will require a bit of a learning curve.

I'm guessing that each rater may interpret results differently in which case you may be looking at a mixed effects model. This allows for adjustments based on rater differences (i.e. different starting points (intercepts), etc.)

The fact that there are potentially two responses makes this what we call a multivariate problem. These are much harder than your standard univariate (one response) problem. The easiest way, though probably not correct, way would be to consider fitting a separate model for each response.

I would strongly advise against using a Likert-type response for this type of problem. They are notoriously weird to work with and very rarely satisfactory. I would instead recommend that you use some sort of 0-100 or something along those lines. Not to mention that you are now facing some wacky potentially multivariate, multinomial mixed effects model if you do take this route. Do some quick Google searches about the analysis of Likert scales and you'll see what I'm talking about!

Answer 3

the data generated from this experiment would be discrete

Usually the term "discrete" would apply to a random variable (numeric rather than distinct categories of things), which took a countable number of distinct values. An example of a discrete variable is a count (counts clearly satisfy the requirement of being countable).

I'm unsure whether it would be considered ordinal, interval, or ratio.

The values taken by your Likert-item like variable would be ordered categorical (per Stevens' typology -- which you seem to be using someone's version of).

This contains much more information than nominal categories though not as much as if you had scores - i.e. values for the categories that reflected 'how different' each level was in a meaningful way (which would effectively make your variable interval on Stevens' typology)

It may be more-or-less reasonable to treat your variable as interval. However, the difficulty is likely to be convincing editors/reviewers/readers of this (I can't make that argument for you since I am not an expert in your area). If you can establish that an interval scale is reasonable, you might be able to then marshal some argument that 2 really is twice as good as 1 (and so forth), which would make it ratio.

I’m assuming the data will probably not be normally distributed,

If your responses are 5 values (even if they are interval), the population of values your sample is supposed to be a draw from certainly cannot be normal. It's not an interesting question though -- I doubt I've ever seen a real variable that was actually normal.

so I’ll need to use a nonparametric test

non-normal in no way implies a need for a nonparametric procedure (nor does "close to normal" imply you should use a parametric one). This confusion seems to stem from a common misrepresentation in many basic texts (particularly common in a variety of areas where stats is applied but not deeply studied) of what parametric and nonparametric mean.

It may be that some parametric model is suitable enough to use a test that assumes it (whether or not it's a normal model is a separate matter). However, even if a parametric model is reasonable, a nonparametric test may be a sensible choice.

The most important thing to start with is your research hypothesis and your understanding of the variable. If you can formulate what you want to know about the populations precisely enough, it may be easier to give good advice.

This doesn’t seem like the classic “before” and “after” medical study scenario of paired data. But I’m not sure if the dual grades would be considered “paired” in a different sense.

"Paired" does not imply "before vs after"; all manner of cases are NOT of that kind but are still paired. You do need that the pair-differences are meaningful (so they need to be measuring the same thing, for a start)

Your values definitely come in pairs, but they're different variables (so not 'paired' in the present sense). You have a multivariate response. You might wish to consider a multivariate comparison rather than two univariate comparisons

If you want sample size recommendations you're going to have to supply some effect-size and desired power.