3
$\begingroup$

I am working in polishing up work an undergraduate class did and the data set is raising a lot of red flags for me. Like to the point that if these concerns I have aren’t resolved by either redoing the sampling or explaining why my concerns aren’t valid, I’m going to walk away from the project. But before making that ultimatum, I want to make sure I’m not missing anything.

  1. The survey itself has 135 responses. Problem is that the group actually being studied only had 17 respondents total with some questions being much lower. There are basically two binary independent variable that define 4 groups. Student: y vs no, veteran: y vs no. The group that we care about is student veterans which the n=17.veteran non-students: n=5, Non-veteran students: n=46, non-v, non-s: n= 47 (combined total: n=115. Not everyone answered both questions).
  2. The sampling is non-randomized. Even if I were convinced that n=17 wasn’t too small for generalization to a larger population, it’s not Randomized anyway (most if not all responses came from snowball sampling). The other source was a link included on in a veteran center monthly email (double self-selection bias given how de veteran’s actually read it).
  3. Last issue: much of the data is ordinal data requiring non parametric analysis. While not normally a major issue, but given the other issues,it is just one more thing that undermines the validity of the data analysis.
  4. What is an acceptable margin of error for sample mean? Unlike margin of error for proportions, the moe for means isn’t a percentage. I’m guessing I can just divide that by the mean to get percentage, but I cannot find anything online specifying what’s acceptable margin of error for means.

So am I wrong in thinking a non-random sample of 17 prevents any meaningful generalization of data for research/policy interests.

Edit: I ended up not using the survey data for quantitative analysis but instead used this data to argue for conducting semi-structured interviews and then use those findings and the structured survey data to inform the design of a structured survey to be done later using proper sampling methods.

$\endgroup$
1
  • 1
    $\begingroup$ For 4) The uncertainty estimation of the estimated mean is given by the standard error. $\endgroup$
    – Ggjj11
    Commented Jul 3, 2023 at 10:28

2 Answers 2

3
$\begingroup$

For sample size, well, as you surely know, a small sample will lead to imprecise estimates of any parameter. How imprecise and estimate can you tolerate?

For nonrandomness - if you don't have a random sample, there may be problems with inference (or generalization) regardless of sample size. In 1936 in the United States the Literary Digest did a poll on the presidential election and, based on millions of responses, confidently predicted that Alf Landon would beat FDR in a landslide. In reality, FDR won one of the biggest landslides of any presidential election.

The key thing there is how biased the sample is, and how that might affect your conclusions.

Still, many, many studies can't have random selection. I don't think I've ever worked on a data set that was truly a random selection from a well-defined population. These studies can still be valuable.

For ordinal data I am not sure why you say that ordinal data requires "nonparametric" statistics. Some methods for ordinal data are quite parametric.

As to what is an acceptable MOE for a mean, the reason you can't find literature on it is that you have to decide (perhaps in consultation with substance matter experts) on what is acceptable or not. It depends entirely on the situation. Sometimes you need a very precise estimate, other times a WAG (wild ass guess) will do.

$\endgroup$
1
$\begingroup$

My concerns proved valid and I ended up not using the survey data for quantitative analysis but instead used this data to argue for conducting semi-structured interviews and then use those findings and the structured survey data to generate hypotheses to inform the design of a structured survey to be done later using proper sampling methods and hypothesis testing based on that data.

For anyone who might come across this post before drawing a sample for your own research, I do want to note that had I been allowed to choose the sampling method used originally, I would have used a proportional (or approximately proportional) stratification of the sample by gender while ensuring that the stratum for female veterans had an adequate sample size; and then either obtained a sample of male veterans large enough to ensure that the ratio of strata sizes is the same in the sample and the population; or, if not possible, collected that many responses, or collected adequate samples for each stratum and then used weighting to adjust the sample data. The method of weighting would depend on what you're researching and why you need to make adjustments. For my research, probability weighting or RIM weighting seems most appropriate as there is no interventional treatment group or control.

$\endgroup$
2
  • 1
    $\begingroup$ In terms of overall sample size inadequacy it suffices to show that the sample size is too small to ask the most simple question: what is the probability that Y=1 vs. Y=0 when there are no covariates? This requires n=96, i.e., you need n=96 to estimate just the intercept in a binary logistic model. See here. $\endgroup$ Commented Jun 3 at 13:56
  • 1
    $\begingroup$ In this case, it would have been a z-test. I guess I forgot to clarify that I was ultimately trying to determine if the difference between student veterans reporting a disability and student veterans reporting having a VA Disability rating and then compare if those differences are significantly different for male student veterans and female student veterans. Even if I opted to use regression, the dependent variable would be continuous and not binary. In this case, because, unlike SSDI, VA disability ratings are ordinal, it would have required ordinal logistic regression if I opted to regress. $\endgroup$ Commented Jun 3 at 15:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.