I have two variables where each observation represents counts at some point in a discrete 1D space (along an RNA sequence). The space is finite, and the counts are highly zero-inflated compared to a Poisson model, and are probably also over-dispersed.

I assume that each variable is a noisy estimator of some underlying function on the space (the affinity of the some protein factor for that part of the RNA sequence). I assume that the value of these functions at adjacent positions in space are not independent (that is the affinity of the proteins are for regions greater than a single base).

I want to know if one underlying function is predictive of the other, but I only have the very noisy estimates of them.

For example: Lets say I have the following data:

var1:     0 1 0 5 1 5 0 1 0 1
var2:     1 1 0 1 5 0 5 0 0 1
position: 0 1 2 3 4 5 6 7 8 9

Both datasets clearly have higher values in the center of the space than at either end, but and correlation between them would be very low because of the sparseness of the data.

Anyone have any ideas how to deal with this?


At the risk of making things more complex, I'm going to try an formalise the question a bit to make things clearer.

An RNA sequence is a string of $N$ bases. Each base $n$ has an affinity $\lambda_{X,n}$ for protein A, and affinity $\lambda_{Y,n}$ for protein B. I wish to know if $\lambda_{X,n}$ predicts $\lambda_{Y,n}$.

$\lambda_{X,n}$ and $\lambda_{X,n+1}$ are positively dependent.i.e. $\lambda_{X,n}$ is high, I expect $\lambda_{X,n+1}$ to be high. I expect that the affinity for most bases is 0.

I have measurements of the affinities in the forms of counts of the number of times each protein has interacted with each base. These counts are random variables $X$ and $Y$ such that

$$ X_n \sim ZIP(\lambda_{X,n}, \pi_X) $$$$ Y_n \sim ZIP(\lambda_{Y,n}, \pi_Y) $$

where $\pi_X$,$\pi_Y$ represent the level of zero-inflation, which is high, such that a standard correlation, such as spearman, would probably be low or 0.


In terms of pairwise dependence structures, the standard linear or monotonic correlations aren't appropriate but recent developments in nonlinear and complex dependence metrics may help. Columbia's Statistics department spent the academic year 2013-2014 focused on developing deeper understandings of dependence structures and ended it with a 3 day workshop and conference that brought together many of the top contributors in this field:


These contributors included the Reshef Brothers, now famous for a 2011 Science paper "Detecting Novel Associations in Large Data Sets"


which has been widely discussed (see AndrewGelman.com for a good overview of this debate, published simultaneously with the Columbia event:


In their presentation the Reshefs addressed all of these criticisms, as well as presenting a vastly more efficient MIC algorithm.

Many other leading statisticians also were in attendance including Gabor Szekely, now at the NSF in DC. Szekely developed his distance and partial distance correlations. Deep Mukhopadhay, Temple U, presented his Unified Statistical Algorithm -- a framework for unified algorithms of data science -- based on work done with Eugene Franzen


One of these approaches should help you elucidate the relationships. The sequencing of the RNA represents a subtle twist to this. Sequential analysis is another topic area receiving significant attention in recent years ever since Google began using sequential probability ratio tests (SPRT) as a way to economize on the information required for multivariate A/B type tests. Alexander Tartakovsky's 2014 book Sequential Analysis is probably the single best treatment of the field. In addition, Columbia Statistics (again) held a workshop on this in June of 2015. Clicking on the "Program and Speakers" tab will take you to links to the individual program abstracts, presentations and papers.


Another approach involves tensor regression for massively multiway contingency tables. David Dunson, Duke, is among those developing methods for this. See his paper Bayesian Tensor Regression on his website for an introduction and overreview of this field:


  • $\begingroup$ Thanks. I can see how MIC might be helpful, but is it not a ranking method? The way it is presented is that it is used to rank all pairwise associations in a large set of variables. I only have two variables. Also I worry that the zero-inflation of the data will be a problem. I can't see where you are coming from with the Baysian Tensor regression. I think perhaps I need to think about it some more. I don't think the "sequence" in my "RNA seqeuence" is referring to the same thing as the "sequential" in SPRT. $\endgroup$ – Ian Sudbery Nov 11 '15 at 16:54
  • $\begingroup$ Tensor regression assumes a massive number of gene sequences. If that's not an analytic challenge you're facing then based on your comments this response is largely unhelpful, beyond clarifying what it is you are really focused on. $\endgroup$ – Mike Hunter Nov 11 '15 at 17:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.