0
$\begingroup$

Let me just start by saying that I am no statistician and might be misinterpreting some concepts, and appreciate all the help I can get with this problem. I creating a survey, in which participants are to evaluate short video clips (~5sec) generated by a machine learning algorithm. The videos show dashcam footage from a traffic simulator and are one of two possible image types (RGB images or semantically segmented images). I would like to investigate the effect of using one image type over the other when it comes to how realistic-looking videos the algorithm generates (i.e. evaluation of an unsupervised generative model).

So now comes the actual questions. My null hypothesis is that there is no difference in the degree of realism between the groups (image types). I am trying to figure out how many participants I need for my survey, which from my understanding is what's called the sample size. In order to calculate the sample size, I need to know the population size, but here is where I get confused. Who or what is my population? The only restriction I have on my participants is their age which must be 18-80 years. Or am I misinterpreting what a population is, maybe it should rather by my total distribution of videos? Have I already chosen a sample size, which is the number of videos on my experiment? Please let me know if I am on the wrong track here.

There are in total 104 videos in the survey, whereof 26 are generated RGB, 26 are ground truth (label) RGB, 26 are generated segmented and 26 are ground truth segmented.

At this point, my thoughts are to use a t-test with a significance level of 0.05, effect size Cohen's d of 0.8 and statistical power of 0.8. But of course, I am open to suggestions.

Please let me know if I should elaborate further on my problem. Thank you!

$\endgroup$
0
$\begingroup$

I think your video processing method is the intervention, and your population is "all the 18-80 year olds who drive". Statistical inference lets you take a sample from that population, run an experiment, and make a statement as to what might happen if this intervention were applied to the population. You really don't care about your sample. It's not your immediate question, but one of the biggest questions in interpreting statistical results is trying to work out how representative your sample is of the population. But I think I'm on solid ground here claiming that this is your population. You want to know if video processing method effects human perception?

I think where your question gets non-trivial is that you are then trying to compare RGB versus semantically segmented images over a random selection of images. Personally, I would treat this in a mixed effects model, where the treatment (RGB versus semantically segmented) is a fixed effect, and the actual image chosen in a random effect. I'd do the power calculations by simulation, but there are papers around that might help find sample size for mixed effects models in your field https://www.journalofcognition.org/articles/10.5334/joc.10/ Of course, this is statistics, so I'm sure other opinions are available, so I would be quite happy to discuss this with others.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you for taking the time to answer my questions. More than anything, you've helped me realize that I need to better establish my independent and dependent variables before I continue with the survey. Thank's for the insight. $\endgroup$ – Paddy Mar 13 at 13:30
  • $\begingroup$ I think you have a model of the form: Realism Score = Constant + Image Processing Method + Image Chosen + Individual Difference + Error. Image Processing Method is a 1/0 variable, with a single parameter. This is the only parameter you wish to estimate well. I think Image Chosen is a random effect where you anticipate that there could be differences in the results based on the image but you don't care (or can't care if you want to keep the sample size down)- you just want any differences out the noise so you can see the signal from your processing method. $\endgroup$ – Paul Hewson Mar 13 at 14:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.