Relationship between two zero-inflated counts varying in space

Question

I have two variables where each observation represents counts at some point in a discrete 1D space (along an RNA sequence). The space is finite, and the counts are highly zero-inflated compared to a Poisson model, and are probably also over-dispersed.

I assume that each variable is a noisy estimator of some underlying function on the space (the affinity of the some protein factor for that part of the RNA sequence). I assume that the value of these functions at adjacent positions in space are not independent (that is the affinity of the proteins are for regions greater than a single base).

I want to know if one underlying function is predictive of the other, but I only have the very noisy estimates of them.

For example: Lets say I have the following data:

var1:     0 1 0 5 1 5 0 1 0 1
var2:     1 1 0 1 5 0 5 0 0 1
position: 0 1 2 3 4 5 6 7 8 9

Both datasets clearly have higher values in the center of the space than at either end, but and correlation between them would be very low because of the sparseness of the data.

Anyone have any ideas how to deal with this?

EDIT:

At the risk of making things more complex, I'm going to try an formalise the question a bit to make things clearer.

An RNA sequence is a string of $N$ bases. Each base $n$ has an affinity $\lambda_{X,n}$ for protein A, and affinity $\lambda_{Y,n}$ for protein B. I wish to know if $\lambda_{X,n}$ predicts $\lambda_{Y,n}$.

$\lambda_{X,n}$ and $\lambda_{X,n+1}$ are positively dependent.i.e. $\lambda_{X,n}$ is high, I expect $\lambda_{X,n+1}$ to be high. I expect that the affinity for most bases is 0.

I have measurements of the affinities in the forms of counts of the number of times each protein has interacted with each base. These counts are random variables $X$ and $Y$ such that

$$ X_n \sim ZIP(\lambda_{X,n}, \pi_X) $$$$ Y_n \sim ZIP(\lambda_{Y,n}, \pi_Y) $$

where $\pi_X$,$\pi_Y$ represent the level of zero-inflation, which is high, such that a standard correlation, such as spearman, would probably be low or 0.

Answer 1

In terms of pairwise dependence structures, the standard linear or monotonic correlations aren't appropriate but recent developments in nonlinear and complex dependence metrics may help. Columbia's Statistics department spent the academic year 2013-2014 focused on developing deeper understandings of dependence structures and ended it with a 3 day workshop and conference that brought together many of the top contributors in this field:

http://datascience.columbia.edu/workshop-and-conference-nonparametric-measures-dependence-apr-28-may-2

These contributors included the Reshef Brothers, now famous for a 2011 Science paper "Detecting Novel Associations in Large Data Sets"

http://www.uvm.edu/~cdanfort/csc-reading-group/reshef-correlation-science-2011.pdf

which has been widely discussed (see AndrewGelman.com for a good overview of this debate, published simultaneously with the Columbia event:

http://andrewgelman.com/2014/03/14/maximal-information-coefficient).

In their presentation the Reshefs addressed all of these criticisms, as well as presenting a vastly more efficient MIC algorithm.

Many other leading statisticians also were in attendance including Gabor Szekely, now at the NSF in DC. Szekely developed his distance and partial distance correlations. Deep Mukhopadhay, Temple U, presented his Unified Statistical Algorithm -- a framework for unified algorithms of data science -- based on work done with Eugene Franzen

http://www.fox.temple.edu/mcm_people/subhadeep-mukhopadhyay/

One of these approaches should help you elucidate the relationships. The sequencing of the RNA represents a subtle twist to this. Sequential analysis is another topic area receiving significant attention in recent years ever since Google began using sequential probability ratio tests (SPRT) as a way to economize on the information required for multivariate A/B type tests. Alexander Tartakovsky's 2014 book Sequential Analysis is probably the single best treatment of the field. In addition, Columbia Statistics (again) held a workshop on this in June of 2015. Clicking on the "Program and Speakers" tab will take you to links to the individual program abstracts, presentations and papers.

https://sites.google.com/site/iwsm2015/committees-and-sponsers

Another approach involves tensor regression for massively multiway contingency tables. David Dunson, Duke, is among those developing methods for this. See his paper Bayesian Tensor Regression on his website for an introduction and overreview of this field:

https://stat.duke.edu/~dunson/