Is there a way of evaluating the quality of the result of a statistical test? Is there a way of evaluating the quality of the result of a statistical test, depending on the representativeness of a sample (or dataset) with respect to the whole population? For example, suppose that the data is not representative of the whole population, then the statistical test will not be very meaningful. Is there a way of detecting this?
 A: Try to get good data. There are some things you can do to help ensure that the data
fairly represent the intended population. Use a random sample not a 
'convenience' sample. Measure/evaluate subjects/items independently. Don't throw out 'outliers'
unless you can track them back to equipment failure, data entry
error, or other specific difficulty. 
Make sure the data aren't 'cherry picked' to be the one 'significant' 
set out of several dozen. Suppose a psychology professor runs six
elementary psych lab sessions a year, about 30 students in each. 
In every lab over four years,
 the prof uses students as subjects to test a favorite theory. 
Finally, in one of the labs during the 4th year, a P-value of 0.043 appears.
The prof exclaims, "I knew it all the time. Finally, vindicated. Now
I can publish." I would not want my name on that paper as statistician.
Check data for flaws. There are many possible things to check, and so I can only give a couple of examples. If you think data should be normal, check to see if that's believable. Are there unusual runs of high or low values? (You might google 'control chart' and look at some of the indications they can provide that values may not be independent or may be drifting in value over time.
Markovian data: For the $n = 100$ observations below, the data are consistent with normal, and there is nothing suspicious about the histogram or the boxplot, but a plot of the observations in sequence shows suspicious runs up and down, and suspicions are confirmed by the ACF (autocorrelation function). An ACF plot of an independent sequence of observations would have ACF values mainly within the blue dotted lines.) You can search to get
explanations about 'autocorrelation' and 'lags.' (Results from R.)
shapiro.test(x);  

        Shapiro-Wilk normality test

data:  x
W = 0.98427, p-value = 0.2812       # consistent with normal

par(mfrow=c(2,2))
 hist(x, prob=T, col="skyblue2")
 boxplot(x, col="skyblue2", pch=19)
 plot(x, type="l");  acf(x) 
par(mfrow=c(1,1))


Indeed, these data were simulated to be independent (as a random walk, which has Markovian dependence) as below:
set.seed(2019)
m = 100;  x = numeric(m);  x[1] = 100
for(i in 2:m){
 x[i] = x[i-1] + rnorm(1, 0, .2) }

Independent data: The same procedures for a truly independent data sequence are shown below:
set.seed(611)
x = rnorm(100, 100, 10) 
shapiro.test(x)$p.val  
[1] 0.8095263


Perhaps other users of the site will have additional suggestions for vetting
data.
