How to handle different sized experiment samples Imagine a 3 by 1 experiment. One group has 1,000 observations, one with 5,000 observations, and 4,000 observations in the last group. I'm trying to see whether the manipulation between groups in the experiment had an effect. Do I need to do anything about the fact that the groups are different sizes, for example by giving greater weight to the group with only 1,000 observations?
Imagine my model is something like.
lm(DV ~ 3by1Variable, data = df). 

Where the three by one variable is a factor for whether an observation was in group 1, 2, or 3.
 A: I would recommend oneway.test in R, which does not assume the
groups have the same variance. The allowance for unequal population variance as reflected in sample variances is much the same as in
a Welch t test; the residual degrees of freedom are smaller if
sample variances differ. The assumption that the three groups be from
normal populations remains.
Fictitious data along lines you describe are sampled in R as follows.
Of course, your data will be different. If your data should present
issues not covered here, please ask.
set.seed(507)
x1 = rnorm(1000, 50, 7)
x2 = rnorm(5000, 52, 8)
x3 = rnorm(4000, 54, 9)

Here we know that the data were sampled from normal populations.
But for your data it would be prudent to look at normal probability
plots of the three samples to see if their plots are substantially
linear. (As here, larger samples tend to give more-nearly linear plots than smaller ones; minor departures from linearity in the tails are nor worrisome.)
par(mfrow=c(1,3))
 qqnorm(x1, col="red"); qqline(x1)
 qqnorm(x2, col="green2"); qqline(x2)
 qqnorm(x3, col="skyblue2"); qqline(x3)
par(mfrow=c(1,1))


Boxplots give another chance to see whether the samples are roughly
symmetrical. However, with normal datasets of this size, multiple
moderate 'outliers' are unsurprising.
x = c(x1, x2, x3)
g = c(rep(1,1000), rep(2,5000), rep(3,4000))
boxplot(x~g, col=c("red", "green2", "skyblue2"))
abline(h = 50)


The `oneway test gives a P-value very nearly $0,$ so we reject the
null hypothesis that all population mean are equal. [In an ordinary ANOVA, assuming equal population variances, the denominator DF would be nearly 10,000, but is considerably
lower here on account of the different sample variances.]
        One-way analysis of means 
     (not assuming equal variances)

data:  x and g
F = 77.195, num df = 2.0, denom df = 2936.9, p-value < 2.2e-16

Ad hoc two-sample Welch t tests indicate that all three means differ.
Because of the tiny P-values involved here, it seems that any reasonable
method of protecting against 'false discovery' from multiple tests on
the same data would still find highly significant differences.
t.test(x1,x2)$p.val
[1] 8.844251e-08
t.test(x2,x3)$p.val
[1] 1.287326e-20

Plots of empirical CDFs (ECDFs) of the three samples show that
Group 3 ECDF (blue) plots to the right and below the others, and
hence tends to have higher values overall. Similar comparisons
can be made between other groups.

plot(ecdf(x3), col="skyblue2", main="ECDFs of Groups")
 lines(ecdf(x1), col="red")
 lines(ecdf(x2), col="green2")

