Skip to main content
clarified some potentially misleading statements
Source Link
EdM
  • 101.5k
  • 11
  • 102
  • 303

Shannnon entropy is well respected but has some potential problems in practice. First, as noted in the above-linked answer, the log term means you need to add some small number to each allele type to deal with populations that lack one or more allele types. Second, the plug-in formula that translates the $p_i$ values to an estimate of entropy has an intrinsic downward bias compared with the population value, depending on $k$ and the total number of alleles in the populationsample. The bias is particularly large if the number of total alleles in a population sample at a locus isn't much larger than the number of allele types, as seems to be the case in some parts of your data. There is also the possibility that you have completely missed some allele types so that your value of $k$ is an underestimate. These issues are noted, with links to further information, on this page.

You might also consider coming up with other measures that both make sense and might be easier to explain. For example: the number of allele types at a locus that contain some pre-defined fraction (say, 50% or 80%) of all alleles at that locus in your sample of a population. That would make sense if you were interested more in the distributions of more-frequent rather than less-frequent allele types. Or if you care more about infrequent allele types, you could come up with a measure that emphasized those (e.g., how many allele types contain the 20% of lowest-frequency allele types).

Shannnon entropy is well respected but has some potential problems in practice. First, as noted in the above-linked answer, the log term means you need to add some small number to each allele type to deal with populations that lack one or more allele types. Second, the plug-in formula has an intrinsic downward bias compared with the population value, depending on $k$ and the total number of alleles in the population. The bias is particularly large if the number of total alleles in a population sample at a locus isn't much larger than the number of allele types, as seems to be the case in some parts of your data. There is also the possibility that you have completely missed some allele types so that your value of $k$ is an underestimate. These issues are noted, with links to further information, on this page.

You might also consider coming up with other measures that both make sense and might be easier to explain. For example: the number of allele types at a locus that contain some pre-defined fraction (say, 50% or 80%) of all alleles in a population. That would make sense if you were interested more in the distributions of more-frequent rather than less-frequent allele types. Or if you care more about infrequent allele types, you could come up with a measure that emphasized those (e.g., how many allele types contain the 20% of lowest-frequency allele types).

Shannnon entropy is well respected but has some potential problems in practice. First, as noted in the above-linked answer, the log term means you need to add some small number to each allele type to deal with populations that lack one or more allele types. Second, the plug-in formula that translates the $p_i$ values to an estimate of entropy has an intrinsic downward bias compared with the population value, depending on $k$ and the total number of alleles in the sample. The bias is particularly large if the number of total alleles in a population sample at a locus isn't much larger than the number of allele types, as seems to be the case in some parts of your data. There is also the possibility that you have completely missed some allele types so that your value of $k$ is an underestimate. These issues are noted, with links to further information, on this page.

You might also consider coming up with other measures that both make sense and might be easier to explain. For example: the number of allele types at a locus that contain some pre-defined fraction (say, 50% or 80%) of all alleles at that locus in your sample of a population. That would make sense if you were interested more in the distributions of more-frequent rather than less-frequent allele types. Or if you care more about infrequent allele types, you could come up with a measure that emphasized those (e.g., how many allele types contain the 20% of lowest-frequency allele types).

Source Link
EdM
  • 101.5k
  • 11
  • 102
  • 303

You can think about each of your data points as a sample from a categorical distribution. That is, each of the two alleles for an individual at a locus will have one of $k$ possible allele types for that locus, with probability $p_i$ of having allele type $i$. You suspect that your 25 populations will differ in terms of their categorical distributions for any particular locus and want to compare those populations in terms of some measure of "diversity" at the locus, something like the variance of a univariate distribution. You also would like a similar comparison of diversity within each population among the 6 loci.

You have a choice of several measures of such diversity, with that choice depending on what aspect of diversity you are interested in and wish to explain to your audience.

The Shannon entropy, suggested in an answer to your related question, is a classic choice. For a particular combination of locus and population, it is $-\sum_{i=1}^k p_i \ln p_i$. You estimate $p_i$ from the fraction of total alleles at the locus in the population sample (total alleles = 2 times the number of individuals) having allele type $i$.

Shannnon entropy is well respected but has some potential problems in practice. First, as noted in the above-linked answer, the log term means you need to add some small number to each allele type to deal with populations that lack one or more allele types. Second, the plug-in formula has an intrinsic downward bias compared with the population value, depending on $k$ and the total number of alleles in the population. The bias is particularly large if the number of total alleles in a population sample at a locus isn't much larger than the number of allele types, as seems to be the case in some parts of your data. There is also the possibility that you have completely missed some allele types so that your value of $k$ is an underestimate. These issues are noted, with links to further information, on this page.

Another possibility is to take advantage of the already defined extension of variance to a multivariate categorical distribution, its covariance. A categorical distribution is a multinomial distribution with the same set of $p_i$ but only a single trial. The covariance of a categorical distribution is thus a $k$ by $k$ matrix with each diagonal element equal to $p_i(1-p_i)$ and each $i,j$ off-diagonal element equal to $-p_ip_j$.

One way to reduce the covariance matrix of a multivariate distribution to a single number is to compute its trace, the sum of the diagonal elements. For a categorical distribution, that is: $\sum_{i=1}^k p_i (1-p_i)$, of a similar form to the entropy but avoiding the problem of unrepresented allele types in a population.

You might also consider coming up with other measures that both make sense and might be easier to explain. For example: the number of allele types at a locus that contain some pre-defined fraction (say, 50% or 80%) of all alleles in a population. That would make sense if you were interested more in the distributions of more-frequent rather than less-frequent allele types. Or if you care more about infrequent allele types, you could come up with a measure that emphasized those (e.g., how many allele types contain the 20% of lowest-frequency allele types).

Whichever estimate of diversity you choose might have a bias from the population value, as for Shannon entropy. I recommend bootstrapping to estimate that bias. For testing hypotheses about differences of your diversity measure among loci or among populations, I recommend bootstrap-based confidence intervals based on a method that takes both bias and asymmetry of estimates into account, like the BCa method. I also recommend comparing two types of bootstrap resampling in this case: re-sampling among alleles, and re-sampling among individuals. If the results are not the same then your allele types within and among loci might not be in Hardy-Weinberg equilibrium for individuals or populations, a result that might inform your interpretation of your analysis.