4
$\begingroup$

I have done a GC content (counting C and G in genomes) and GC variation within genomes, in a slide-window (1000 pb-base pairs). I have each genomic data separated in tsv files. When I work with pandas and merge all data by columns, I got a lot of lines with NAs because each species has a different genome size. Pandas works well with this data, because I have means and all other statistics working fine, I can even plot the data with no problem. But my doubt is this, I can keep the data merged as it is or it is BETTER (statistically speaking) to drop all lines with NAs? I ask that because I would like to use R, but I cant merge the data because R complains about the different column sizes. Any tip or commentary about this subject would be very appreciate! Even a good reading about the subject. Thank you all by your time. Paulo

$\endgroup$
2
  • 1
    $\begingroup$ Why do you want to merge the species data? Do you want to analyze GC variation between species? Can the genomes of different species be meaningfully aligned (into columns of a data frame)? $\endgroup$
    – dipetkov
    Sep 2, 2022 at 6:52
  • $\begingroup$ yeah! I will compare variation within different whole genomes. $\endgroup$ Sep 2, 2022 at 23:42

2 Answers 2

5
$\begingroup$

Generally speaking, outside of extremely specific reasons, listwise deletion of missing values is not encouraged. It would be better if you could employ some alternative like multiple imputation (MI) or full information maximum likelihood (FIML). I have linked a couple very readible papers here that discuss missingness, modern techniques, and misconceptions. There is also a phenomenal book on MI using R that I can recommend called Flexible Imputation of Missing Data by Stef van Buuren.

The main reason you don't want to remove missing values willy-nilly is because you will lose a ton of valuable information. Normally listwise deletion deletes an entire row of data rather than just that unit that is missing. Regardless, this leads also to a loss of power. Additionally, there may be missingness patterns that you need to examine which predicate decisions to remove or impute values. Is the missingness due to chance? Is it systematic? Is there clustering involved? (i.e. a specific hospital constantly has missing values for medical exam data). These are also things to consider when dealing with missingness.

References:

$\endgroup$
3
$\begingroup$

Is this actually a missing data problem?

Technically, missing values are values we could have observed but didn't get recorded for one reason or another. (The reason for the missingness is crucial as data can be omitted in a biased way, eg, wealthier households may be less likely to report their income.)

However, in you case species have different genome size because they have evolved differently. (I have only basic knowledge of biology but I read a bit about genome size in Wikipedia.)

So if a species has lost or gained a gene (a chromosome?), the gene isn't really missing and there is nothing to be imputed. To align the genomic data for multiple species, you can replace the NA's with 0s or perhaps -1s.

The important question is how does the analysis take into consideration the fact that not all species have all the same genes. This is a science question (how to align and compare genomes with different size) or perhaps a data structure question (how to put together sequences of different length), not a statistics question.

$\endgroup$
4
  • $\begingroup$ Dear friend I am using a statistic in the case GC content variation within genomes, to compare this variation and understand how the total GC content and the GC chromosomal variation are associated with traits like phylum, habitat, etc...For fact GC content are very well associated with genome length in prokaryotes for example. The GC variation is calculated as GCVAR <- function(n, di){ log(1/n * sum(abs(di))) } , where di = GC_chr-window - GC_total. So if I discard rows my values get different because of 1/n. Then I think I will work with each genome separately. $\endgroup$ Sep 4, 2022 at 14:33
  • $\begingroup$ Sounds good. My understanding of your field of study is limited. But it seems the analysis you plan to do is well defined, so you face a data structure issue: a data frame might not be the right way to store genome data for different species. $\endgroup$
    – dipetkov
    Sep 4, 2022 at 14:48
  • $\begingroup$ My friend all files are tsv like: start position gc values (ex. '0' 0.43, '1000' 0.40....) I think data frames are easy to apply functions, because I keep the positions and the values, however I think I can use named vectors in R. Do you have a different option? Thank you. $\endgroup$ Sep 5, 2022 at 15:25
  • 1
    $\begingroup$ I'd say data frames are easy for applying many but not all functions. They don't seem the right choice for your case if applied naively. It seems more appropriate to have a long data frame with (species, position, CG content) triplets. Note that if your question is about data manipulation CV is not the place to ask it. Try Stack Overflow maybe? $\endgroup$
    – dipetkov
    Sep 5, 2022 at 15:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.