What is the correct way to test for the impacts of working with a reduced data set? I'm interested in learning about the potential impacts of working with a reduced dataset (while keeping a sufficiently high sample size, >30 obs). The data collection process can be expensive and I'd like to know how would one test if working with less data is just as good.
Specifically, is it wrong to compare a full data set with a reduced version of the same data set? I ask because comparing a full data set collected at one point in time against another, that is smaller in scale, could be different simply because of the factor of time. So, for example, if the following data were not randomly collected (the same locations "ID" are sampled each year and are independent of each other) and the dependent variable can be anything from salinity to dissolved oxygen - doesn't really matter), is this an improper way of approaching the problem?
df <- as.data.frame(seq(1:100))
names(df)[1] <- "ID"

set.seed(123)
df$dependent_var <- sample(16:35, size = 100, replace = TRUE)


df_reduced <- filter(df, !ID %in% c(1:50))


df$data_type <- "All"
df_reduced$data_type <- "Less"

new_df <- dplyr::bind_rows(df, df_reduced)

wilcox.test(new_df$dependent_var ~ new_df$data_type, mu = 0, 
            alt = "two.sided", 
            conf.int = T, 
            conf.level = 0.95, 
            paired = F)


# Wilcoxon rank sum test with continuity correction
# 
# data:  new_df$dependent_var by new_df$data_type
# W = 2401.5, p-value = 0.6954
# alternative hypothesis: true location shift is not equal to 0
# 95 percent confidence interval:
#   -2.000019  1.999979
# sample estimates:
#   difference in location 
# -1.629517e-05 

Is this a viable conclusion?: Half as many data points have a 70% probability (p-value) of sufficiently capturing the variability of a more "full" data set.
I realize some background knowledge on the dependent variable is necessary, but I'd like to know what are some things to consider when approaching this problem strictly from a statistics point of view, if possible. Please correct me wherever I'm making a mistake.
 A: This won't be a comprehensive answer, but I can offer a few thoughts.
I don't think you are interpreting the p-value correctly, and I don't think the Wilcoxon-Mann-Whitney test is telling you what you want to know.
Also, there's nothing magical about the n > 30 rule-of-thumb.  How many observations are sufficient for your purposes depends on how large of an effect you are trying to detect and the amount of noise in the data relative to the size of this effect.
Drawbacks to having a smaller sample size include less power in hypothesis tests. That is, a higher p-value for a given effect and sampled population. And less confidence in the precision of parameter estimates.  That is, a wider confidence intervals for, say, the calculated mean.
The following code should demonstrate these points for a simple data set.  I didn't want to cook the results by including a set.seed(), so you may want to run this a few times for the pattern to solidify.
The larger sample A has a mean of 5, and the one-sample t-test returns a p-value of c. 0.02 when the mean of A is compared to a null mean of 4.  The 95% confidence interval for the mean is (4.16, 5.84).
The results for B will vary since it is a random sub-sample of A.  But in general, the p-value for the one-sample t-test will be > 0.05.  And the confidence interval for the mean of B will be wider.
A = c(1,2,2,3,3,3,4,4,4,4,5,5,5,5,5,6,6,6,6,7,7,7,8,8,9)

t.test(A, mu=4)

B = sample(A, 10, replace=TRUE)

t.test(B, mu=4)

