I have this problem and more I read online, more confused I'm getting. So I have 4 (or more) groups with different users (for simplicity I only used small numbers, but I may have 50, 100, 150 and 200 users per group). After finding mean for each group, I want to be confident and back the results with some stats, but not sure what. Looking for a statistical metric telling me when I can disregard mean e.g. from Class1 due to not enough observations. I have measure SD, Variance, SE and Confidence Intervals (CI), but not one value to show relation to sample size (although CI does that partially). Appreciate any input.
An ANOVA would be my first choice. With an ANOVA you perform a global test of difference for all the groups. If you reject this hypothesis (of at least one difference in means) you can perform additional tests to test each pair of means, for ex. post-hoc Tukey HSD test. Both of these tests make no assumptions of group sizes.
Thanks for comments. Following suggestion from @user2974951, I think I found what I'm looking for. All I need to use is the Power Calculations with the following R formula:
pwr.p.test(h = 0.2, n = nr, sig.level=0.05)
While I have to understand what all parameters mean and read some literature, first preliminary results seem promising.