Setting up experiment for statistical analysis

Question

I am looking at setting up an experiment concerning a hobby of mine, basically measuring a variety of parameters 'before' and 'after' and see which one, if any, gives the most reliable prediction of a final parameter i.e. do they have a linear relationship, etc. The object being to save some time and effort later not sorting by the parameters that have little actual bearing on the final results, and to see if some of the more tedious sorting methods are actually as useful as thought.

The experiment is an extension of one I did for my final paper in my stats class a couple years ago: investigating the relationship between case weight & volume vs. muzzle velocity for centerfire rifle cases as used in modern target competition (in which I'm fairly active). For that paper I had to spend a fair amount of 'time' demonstrating various methods that didn't really pertain to what I wanted to know, but had to cover anyway to get an 'A' ;)

For this go-around... I'm looking at taking one hundred pieces of brass cases from one box, one lot. These are the 'non-consumable' parts, as they can and do get re-used and reloaded multiple times. Everything else - bullet, powder, primer - get shot down the barrel and cannot be re-used/re-measured. Specifically, the cases can and do become 'fire-formed' to the interior dimensions of the chamber under extreme pressure (50-70k psi). In my original experiment, I found a significant, but not especially strong, correlation between the initial case weight and the muzzle velocity... of the first firing. Case volume, on the other hand, which should be very strongly related to MV... was not, theoretically because the 'virgin' cases don't necessarily 'fit' the internal dimensions of the chamber and a certain amount of the energy generated is expended not just in heat, but in squashing the brass against the case wall during firing.

So again, I'm looking at taking 100pcs of brass, weighing them straight out of the box, taking various measurements (case wall thickness inside near the case web, case neck thickness near the mouth, weight, volume, etc.) in their 'virgin' untouched state, then performing several routine case prep steps (trim to length, chamfer/debur the mouth, debur the flash hole, uniform primer pocket) and repeat the measurements. Then load and fire the rounds as uniformly as possible - bullets sorted for maximum consistency, powder weighed on a milligram-capable analytic lab scale, etc. while controlling the rate of fire so as to regulate temperature rise along the barrel and measuring the muzzle velocity via a chronograph approx. 15 feet down range. Then I plan to completely clean the cases inside and out to remove any buildup of carbon or powder residue, and repeat the measurements and then load and fire again and then clean and measure one last time.

Part of what I want to see is how the distributions of some of the measurements change as the cases are prepped, then again as they are fired and formed to the chamber. After that... I want to see how much difference is there really between 'virgin' cases and fire-formed cases in terms of muzzle velocity, and finally... of all the tedious measurement steps mentioned above, which ones actually give a reasonable indication of consistent MV so I can successfully cull suspect pieces of brass during preliminary sorting rather than suffer lost points when using them on target in competition.

The single biggest problem in my mind, at least on the surface, is the chronograph. Getting anything approaching credible numbers for accuracy or % error from the vendors is somewhere between difficult and impossible. Given the nature of the device, it is extremely difficult to test on a consumer level - every round through it may (or may not) be just slightly different, so determining how much of the variation displayed on its screen is really the ammunition and how much is the variability of the instrument itself... has me scratching my head, to say the least.

As to why I was kind of vague before about the details of the experiment... well, sometimes people get kinda weird as soon as they realize something involves GUNS and won't touch it with a 10 ft pole regardless of how reasonable it may be.

Thanks,

Monte

Answer 1

Actually, people get paid big money for statistical guidance through experiments... If you're not too sure about it, I'd also advise to consult a statistician. An internet forum is not the best aid for complex analyses. Much of what is possible depends on the structure within the dataset: How are the variables distributed, how is the correlation structure,...

But I'll take a shot at it. So you have $m$ measurements at 4 different times, being:

• virgin state
• after preparation
• after first round (shot in "virgin state")
• after second round (shot in "non-virgin state")

Your response variable is $MV$. Let's assume the error of the chronograph is random and normally distributed, this makes life a lot more easy. Let's also assume that for all other measurements the assumptions of parametric models apply, i.e. that every measurement consists of $n$ indipendent and identically distributed random variables, being your $n$ tested cases.

A rather basic approach would involve

• using a Kolgomorov-Smirnov test to compare distributions (think about correction for multitesting)
• a model selection procedure to determine the best predictors for $MV$ using either set

So in theory, you could use a GEE with "round" as a factor, indicating whether the measurement was made in the first round or in the second round, thus allowing for different intercepts and coefficients for first and second round that can be compared and tested. You have to specify "case" as a random factor in the GEE to get correct estimates of your SE. Basically, if the main effect "round" is significant, there is a difference before and after in average $MV$. If the interaction "round":measurement is significant, the impact of the measurement is different between the two rounds. Now it boils down to use any sensible criterium to select the best model.

Without seeing the data, it's impossible to say whether this approach is actually valid. You will have to do some descriptive and exploratory analysis to really understand the underlying distributions and correlations. Otherwise you can never know whether you violated assumptions of any method.

On a sidenote: Give this dataset to 10 statisticians, and chance is big you get as many different models. You should keep in mind whether or not the model also makes sense in real life.

my 2 cents