testing significance from a non-normal simulated distribution with CIs

Question

Say we have a value B =-0.9631366 of a population level parameter from a real population. Then we use simulations based on some mechanistic knowledge of the population to create a range of 250 values for B

The simulated data re below and the real value is B = -0.9631366

library(ggplot2)
ggplot(www, aes(x=simB)) +  geom_histogram(alpha=0.6)

Is it correct to test significance of B as deviating from the simulations as:

U95 = mean(www$simB) + qt(.975, df=249) * (sd(www$simB)/sqrt(249))
L95 = mean(www$simB) - qt(.975, df=249) * (sd(www$simB)/sqrt(249))

and if B < L95 or B >U95 B is different from sims at alpha =0.05

or because of the non-normality of the simulated distrubution is more required?

www <- data.frame( simB= c(-0.855779034437112, -0.987178580538226, -0.897713186798834, 
-0.944427599586953, -0.896791768386997, -0.917051166714377, -0.899809973306369, 
-0.802414698833733, -0.939055326243634, -0.981593636749885, -0.89377603941762, 
-0.943135254742762, -0.961242483565258, -0.981683778625361, -0.916617370707248, 
-0.877299847611706, -0.984446442563942, -0.946332681825364, -0.902519876016483, 
-0.861779659282824, -0.970615934851319, -0.970220134564053, -0.989759102905799, 
-0.95638323732857, -0.776318290539656, -0.719531206835281, -0.925055342023467, 
-0.947438395358281, -0.958320110471624, -0.9752084281148, -0.906332835476218, 
-0.950874010361265, -0.945581574454779, -0.966574153874217, -0.950739379260704, 
-0.901269942269079, -0.986308236955983, -0.625839164839443, -0.915133893576611, 
-0.917493338282421, -0.780906920195035, -0.913649887015811, -0.933465705313089, 
-0.969223197379272, -0.937081349445786, -0.938991080904559, -0.965427219385943, 
-0.894484593271904, -0.984500091643262, -0.991103592246488, -0.789014341393232, 
-0.832591685794236, -0.971252900323587, -0.870122652474482, -0.930377922808605, 
-0.920207735717193, -0.940276948765526, -0.755889680161622, -0.881617684101453, 
-0.970416701415067, -0.984826842091866, -0.860596498053491, -0.919279397383687, 
-0.974765804331888, -0.900440062250957, -0.981686197984381, -0.96845630621896, 
-0.973337130599757, -0.923119205883875, -0.985575485466655, -0.89726170512267, 
-0.869268420691373, -0.898273738630049, -0.978579863169303, -0.943738482657564, 
-0.886716860559106, -0.95558017478473, -0.693383561097355, -0.956241400537855, 
-0.972326232835288, -0.964413393090554, -0.994447158961302, -0.978695347755226, 
-0.934416087535161, -0.918705072101488, -0.891231131143489, -0.956986966678055, 
-0.967758028969008, -0.804723930198671, -0.92595244242526, -0.934729986201244, 
-0.96975964206674, -0.917840906040743, -0.888580930571457, -0.939216641337341, 
-0.921824904094623, -0.897372583421359, -0.974879150323193, -0.963897764104796, 
-0.978597596314819, -0.932011183720776, -0.860159973381268, -0.918154629368016, 
-0.866596061658211, -0.940393436959039, -0.929530857388062, -0.909974013125193, 
-0.970067049749278, -0.838975615312944, -0.792860756627094, -0.990763836079547, 
-0.966083607487013, -0.948851800942443, -0.959843821788649, -0.923873876501207, 
-0.956018115995337, -0.952790783284228, -0.970263081709061, -0.989831391524988, 
-0.828582708736267, -0.905585632337764, -0.841297195919796, -0.953773904135005, 
-0.758986496939465, -0.948868903208259, -0.969452791564126, -0.713525902425064, 
-0.959229044813073, -0.88566857463148, -0.82378706740502, -0.945916915487615, 
-0.937437856775107, -0.955720106790172, -0.976344933874582, -0.892949228758341, 
-0.929209415152962, -0.370522593804931, -0.946692078453091, -0.968067797446262, 
-0.868288507529086, -0.920259678400992, -0.868543242032412, -0.80185732320003, 
-0.911870341102553, -0.785599829705549, -0.884106050413119, -0.907852887872512, 
-0.964592402423168, -0.914304718917662, -0.927335996192606, -0.974198235723978, 
-0.968659457359635, -0.946924652907109, -0.908896194277313, -0.971589633791525, 
-0.882792976343395, -0.752339843457605, -0.976508874877627, -0.905157982990083, 
-0.88109869010454, -0.855039100733371, -0.948563055751314, -0.753709408584966, 
-0.927748893919667, -0.982792464584162, -0.959268467938227, -0.923465690498043, 
-0.968782004445127, -0.912024720994946, -0.864035024354372, -0.93814353414198, 
-0.914691129370079, -0.919470306897394, -0.972026324855857, -0.881927915105332, 
-0.917889522496994, -0.890323355601995, -0.947669073979736, -0.938699015796664, 
-0.954424674670711, -0.960599541843406, -0.523893093274131, -0.945317705002649, 
-0.981846089629879, -0.972236668638128, -0.983747435212579, -0.878711335368713, 
-0.799951145047826, -0.971775584442765, -0.94828405016827, -0.935566809053414, 
-0.920101525796394, -0.982551218374477, -0.315090567525098, -0.978040110258354, 
-0.97224637662677, -0.91193289518175, -0.847858138045883, -0.906240064769445, 
-0.798861519921611, -0.776946303002482, -0.950204290694252, -0.876285389930142, 
-0.887801918597555, -0.982053433988483, -0.897616691131035, -0.856054260591338, 
-0.950241480352801, -0.892155039230019, -0.978770260513747, -0.930708701630615, 
-0.95772691096187, -0.87520108065953, -0.97342635046151, -0.898651301107369, 
-0.982776220293659, -0.991630276161649, -0.944077053889773, -0.954287000647911, 
-0.954681407984601, -0.970195561123288, -0.877525635225776, -0.901893897119648, 
-0.966372225121406, -0.939511185927264, -0.863117246569525, -0.930407950475305, 
-0.927895326407901, -0.964922059354554, -0.915023784105047, -0.973273033235832, 
-0.915670497459547, -0.885110524206701, -0.86886814753076, -0.930855012051049, 
-0.654459692684275, -0.880177654557235, -0.852491813934454, -0.895119979444544, 
-0.934424453914663, -0.953566575534681, -0.903500670064793, -0.963908059495309, 
-0.815739523046, -0.935760477591788, -0.833262784318341, -0.895437356039993, 
-0.948886505021108, -0.910343732776735, -0.925579671044067))

Answer 1

Does it make sense to think about the difference between your observation and the mean of 250 simulations? I doubt it does... after all why is 250 simulations is the magic number, why not 1000? As you increase the number of simulations the extent to which your observation falls outside the confidence interval of means of simulations will inevitability increase. That being said, if 250 is a number that is theoretically important and relevant, then maybe, just maybe, your significance test involving confidence intervals is reasonably okay. After all, because the central limit theorem really started to kick in once you had more than 20 or 30 simulations, you really can assume that the distribution of simulation means is reasonably normal regardless of the shape of the distribution of the actual individual simulated values.

If it doesn't make sense to think about the difference between your observation and the mean of 250 simulations, then what does make sense?

Does it make sense to think about the potential for error in your observation? If so, because you only made one observation and it came from the full population, I think that you are relatively out of luck. You could of course ignore that and start simulating some counter-factual populations, but I don't suspect that is really particularly theoretically sound.

My guess is/was that you are really interested in "how different is my observed value from the simulated values" (not the mean of a set of simulated values). Maybe that guess is wrong. However, if it is right, then you are hard pressed to do anything other than describe what you have: your observation is greater than 26% of the distribution of simulated and less than 74% of the distribution of simulated values. Sure, if you wanted to you could treat those directly as a type of one tailed p... which is easy because your proportions map onto Z directly, with qnorm(.26) being -.64 and qnorm(.74) being +.64. If you wanted to torture yourself (and the data) some, you could set up an indication of degree of absolute deviation of the simulations relative to the mean of the simulations:

dev <- www$simB - mean(www$simB)
dev <- abs(dev)
table(dev < abs(-0.9631366 - mean(www$simB)))

... you would find that your result deviates to a greater extent than the simulations from the simulation mean for 150 simulations. Divide that proportion by 2 and you'd get a proportion (p) of .3 and a corresponding Z of -.52. It seems to me that the *p*s and *Z*s above are meaningless relative to the 26% and 74%. However, the most data comes from visualization... and I think visualization is a sound approach because you have some prior assumptions about how the data is structured that you seem to hold pretty strongly. That is why I suggest/ed that you simply plot your observed result relative to your simulated results:

ggplot(www, aes(x=simB)) +  geom_histogram(alpha=0.6) + geom_vline(xintercept=-0.9631366)

... or if you have it out for histograms ...

ggplot(www, aes(x=simB)) +  geom_density() + geom_vline(xintercept=-0.9631366)

When I see this kind of problem, I'm left thinking about permutation methods, but since you sampled all of a population and don't have a comparison group I don't know how those could apply. If you'd sampled part of the population, then we could possibly generate a counterfactual distribution of what values you would have plausibly observed if you collected data again and compared that to your simulated values and been cooking with fire.