In connection with the question here I came upon a more interesting question. I believe the question is large and distinct enough to have it's own thread. Of course I might be mistaken, in that case I apologize for the clutter...
I have a dataset of experimental values, coming from an unknown distribution. The data is then used in an analysis, and as a measure of significance of the results of the analysis, I would like to sample random data from the same distribution and run my analysis on these datasets (say 1000 times the size of the original dataset) to see if the results from experimental values show significant deviation from randomized data.
I was thinking about drawing samples from a normal distribution, as it feels most natural that the samples come from some normal distribution. However I need to back up my assumption of normality somehow. I was originally thinking of using some sort of a normality test, but after some reading on the matter such as What is 'normality?' and Normality testing: 'Essentially useless?' threads, and of course the Wikipedia article on normality tests; I feel like these tests are not an accepted way of validating normality assumptions.
- How can normality be validated without using visual cues such as QQ plots? (the validation will be a part of larger software)
- Can a "goodness of fit" score be calculated?
EDIT: As requested, I'll try and add some details about the data. The data at hand are from a biological experiment, however the instrumentation has high variation between the runs. The analysis I've mentioned takes the measured values and using a mathematical model evaluates functional meaning of the measured data. To do so, I need to see how unrelated/uncorrelated, made-up data rates in the same analysis, hence the intent to model by randomized values. I hope this clarifies my point of view.
EDIT2: There has been a series of questions, asking for clarification on the question. Both here and in the comments below I tried to explain my situation to my best ability. It seems like we are suffering from a communication mismatch... I dunno how I can give an example without writing up a long table and complicating things further OR brutally simplifying the bigger picture.
I have no doubt that everyone who took their time and supply a reply has the best intentions, but I really appreciate if you could focus on the question at hand instead of inquiring further and further into the motivations behind why I need to do things this way and not another way.