1
$\begingroup$

I have 322 clothing items that have been ordered by our military from 2013 to 2018. Each military member is either in a Tier 1 or Tier 2 location. I ran Friedman tests on each individual item over the six year period and adjusted p values through Benjamini-Hochberg. My results look like this:

enter image description here

Because the UIC (Tiers 1 and 2) have more items that are significant (with p-values mainly in the order of 10^-16), I have highlighted those cells for further analysis. Someone has questioned why don't I consider MOS and Status as well, which I discount since there are more items that are not significant than are significant.

Am I correct in doing this?

$\endgroup$
6
  • 1
    $\begingroup$ Could you please say more about the purpose of this study, the questions you are trying to answer, and how you structured the Friedman tests? That information might help you get a better answer. $\endgroup$
    – EdM
    Commented Dec 18, 2019 at 20:44
  • $\begingroup$ The purpose of the study is to determine an optimized system for allocating points for clothing to each military member. The current system, a Tiered approach, is based on UIC, which refers to the unit where they work. The Friedman test looks at each item and how many were ordered in each year in each UIC (there are 2900 UICs). $\endgroup$
    – Angus
    Commented Dec 18, 2019 at 20:53
  • $\begingroup$ I should also point out that the number of p values in the order of 10^ (-15) were zero for both tiers 1 and 2 Status; 45 and 62 for tiers 1 and 2 respectively, MOS; and, 61, 109 for tiers 1 and 2 respectively , UIC. $\endgroup$
    – Angus
    Commented Dec 18, 2019 at 21:37
  • $\begingroup$ It's been a few decades since I had an MOS, so please remind me what the "points for clothing" allocated to each military member are supposed to represent. Also, I want to make sure that I understand how you did the Friedman tests. I understand that you did these tests separately for each item of clothing and separately for Tier 1 and Tier 2 locations. So in the terminology of the Wikipedia page did you use the 6 separate years as "blocks" to evaluate differences among MOSs, between genders, etc. as multi-category "treatments," or vice versa? $\endgroup$
    – EdM
    Commented Dec 19, 2019 at 1:16
  • $\begingroup$ It would help future visitors to the site if you would edit your question to include the information you provided in comments, as comments can sometimes get lost while there is a permanent record of all versions of questions and answers. It would also help if you could specify the numbers of categories of each of Status, Rank, MOS, and Environment, in addition to the 2900 UICs, as that information could be important in formulating an answer. $\endgroup$
    – EdM
    Commented Dec 19, 2019 at 1:24

1 Answer 1

2
$\begingroup$

If I understand your analysis correctly, much of what you are finding thus far might have a lot to do with the number of levels (categories) within each of the Factors that you have analyzed separately. Also, don't forget that each UIC level that ultimately gets adjusted for will affect fewer military members than will each MOS or Status level that gets adjusted for. It might be better to formulate your model differently, depending on the overall goal of this project.*

My understanding of the situation

As I understand it, the analysis was done separately for each of the 2 Tiers of Location and for each of the 322 clothing items. Within each of these 644 combinations of clothing and Tier, a separate non-parametric Friedman test was done for each of the 6 remaining Factors: Gender, Status, Rank, Unit Identification Code (UIC), military occupational specialty (MOS), and Environment.

For each combination of clothing item and Factor within a Tier, the Friedman test combined information among the 6 years of data to determine whether there were any significant differences in terms of orders for that clothing item among the levels of the Factor being examined. The Friedman test provides a p-value for whether any differences among the levels of the Factor are greater than would be expected by chance. As there are hundreds of comparisons being done overall, corrections for multiple comparisons were done to control false-discovery rate (FDR) via the Benjamin-Hochberg (BH) procedure.**

A difference for a clothing item within a Factor and a Tier was considered significant if the FDR was no greater than 5%.

UIC was singled out for further analysis as more than half of the items of clothing showed significant differences on that basis among UIC in both Tiers of location.

Prioritization

In terms of prioritizing further analysis, it might seem to make sense to start with the Factor that shows the most clothing items having significant differences, which is UIC. But your cutoff of 50% of items showing significance within a Factor/Tier combination is somewhat arbitrary. With almost 50% of items showing significant differences as a function of Status and about 40% showing differences as a function of MOS, it would seem to be unwise simply to ignore Status and MOS going forward.

Also, the numbers of military members associated with each level of a Factor will differ greatly among the Factors. With about 2 million active duty and Reserve US military members overall, adjusting for an individual UIC level will affect on average fewer than 1000 individuals, while each adjustment for an MOS will affect many more, and I suspect that an adjustment for a Status level will affect hundreds of thousands. So you might want to consider the number of individuals likely to be affected as you prioritize your further work.

One potential problem I see

There are 2900 levels of the UIC factor, about 25-30 levels of Rank, a few hundred levels of MOS, only 2 levels of gender, and probably only a few levels of Status (if Status means something like active/reserve). With at least 10 times more levels of the UIC Factor than for any other Factor, it might not be so surprising that more items of clothing have a "significant" difference for at least one UIC value (which is what the Friedman test reports) than you find for the other Factors.

Although the Friedman test takes into account the number of levels of the Factor being considered, I suspect that some levels of UIC/MOS/etc are substantial outliers for each clothing item, with the particular UIC/MOS/etc outlier value possibly different depending on the clothing item. For example, I was issued hospital whites to wear while performing my duties under my US Army MOS; I doubt that there is much call for such clothing in an Airborne unit.

So to my mind, finding 204 items of clothing having a "significant" difference with respect to UIC levels in Tier 2 isn't necessarily more important than finding 141 items with a difference based on MOS in Tier 2, when there are about 10 times as many levels of UIC as there are of MOS.

Alternative analysis approaches

Doing this type of analysis separately for each of the Factors and Tiers might not be the best approach. It's typically more powerful to consider many predictors together in a combined analysis; see this page for discussion in the context of regression. As your data are presumably units of clothing, you could consider a set of Poisson multiple regressions,*** modeling the numbers ordered of each clothing item as a function of all Factors together: Tier, Gender, Status, Rank, UIC, MOS, and Environment. If the model is adequate, you can test whether any of those Factors is significantly related to orders of the clothing item, and do further tests of which of the 2900 levels of UIC, 100-200 values of MOS, etc., are most responsible for those differences, potentially speeding up further analysis and evaluation.

You also might consider looking for clusters of clothing items whose orders tend to track together. Reducing the analysis from 322 individual items to a few dozen clusters of related items might simplify and better inform your analysis. Similarly, instead of examining all 2900 UICs individually you could consider grouping them according to shared characteristics based on the Table of Organization and Equipment. After the higher levels of organization are taken into account, specifics of particular UIC that don't fit well into the model could be evaluated.


*It's still not clear how this analysis will inform the "system for allocating points for clothing to each military member." Editing the question to make this clearer might allow a better answer.

**Probabilities from the BH correction are usually specified as "q" values, to distinguish the false-discovery rates that they represent from the corrected "p" values, but your meaning is clear from the question.

***If the order numbers are large, you could consider standard multiple regressions, possibly with some transformation (like logarithmic) of the order numbers.

$\endgroup$
2
  • 1
    $\begingroup$ @Angus I added a thought at the end about examining groups of related UIC together rather than individually. $\endgroup$
    – EdM
    Commented Dec 20, 2019 at 14:55
  • $\begingroup$ Thank you @EdM. Very insightful. $\endgroup$
    – Angus
    Commented Dec 20, 2019 at 18:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.