1
$\begingroup$

As a heads up, my knowledge of statistics and probability is minimal at best but I need help on something for an application I’m building.

Ok I have a sequence of values either 0 or 1 that might look something like this, 0010111010 (can be 1000’s of digits in length). I currently have a program that looks at that sequence and counts the number of occurrences of any given pattern up to 6 digits. For example in the sequence; 0010111010 there are five 0’s, five 1’s, one 00, three 01, three 10, two 11, …, zero 000000, zero 000001. So it looks for how many occurrences there are for all 126 possible patterns and counts each one.

So my question is, What would be the best way (if there is a way) to check for a biased/irregular number of occurrences of a given pattern? For example in the sequence mentioned above there are 9 2 digit patterns, that means on average there should be 2.25 occurrences of each possible 2 digit pattern (I’m not even sure on that because each digit effects multiple patterns results), if it turns out the sequence has a biased count of any given pattern it needs to detect it. I’ve tried applying the chi-squared test and then getting a P-value for it but I got some weird results that would suggest even a random sequence of 4000 numbers is biased towards certain patterns and not random like I would have thought. Either way could someone tell me if what I’m looking to do is possible and if it is, the easiest method as well as how to I’d do it.

Summary: I need a way to know if a pattern of say 000 is more or less frequent in a sequence than it would in a random sequence. And the patterns can over lap so 0000 has 2 occurrences of 000 not 1.

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ Is your sequence only thousands of digits long? Or could it be billions or more? One's approach to this problem would differ in those circumstances. $\endgroup$
    – whuber ♦
    Commented Jul 2, 2021 at 18:54
  • 1
    $\begingroup$ @whuber I'd never need it to process billions or even millions. The minimum sequence length would be around 300 digits and the maximum length would be around 400k digits in the sequence. The only thing that might increase is the pattern length from 6 to potentially 10 or even a little higher depending on the speed of processing. $\endgroup$
    – Seamus Mckee
    Commented Jul 2, 2021 at 19:34
  • $\begingroup$ What are all tose 01's representing? Are you looking for some specific lind of pattern? Looking for all pattern (in subeq's up to some length) will present multiplicity problems, which might explain why you detected on simulated random data. $\endgroup$
    – kjetil b halvorsen ♦
    Commented Jul 3, 2021 at 19:59
  • $\begingroup$ @kjetilbhalvorsen They could represent anything that can be converted into 1 or 0 value. I’m not looking for a specific pattern, I simply want to see if any patterns are more/less frequent than they would be in a statistically random sequence. I feel my approach/math is wrong; Ideally there’s some equation that I put some values into and it outputs a p-value for each pattern and this value represents the probability that the number of occurrences of that pattern are abnormal. Honestly I’ve spent longer than I’d like to admit on this so I might just except its currently out of my skill set. $\endgroup$
    – Seamus Mckee
    Commented Jul 3, 2021 at 23:12
  • $\begingroup$ What is the marginal probability of 1's? Your assumption that all patterns should be equally likely is only reasonable if that is 0.5, else you must reformulate. If it is $p$ ad else the bits are independent, for patterns of length 2 the four pattern 00, 01, 10, 11 should have probabilities $(1-p)^2, p(1-p), p(1-p), p^2$ respectively --- could p change with time? ... You need to formulate mathematically what is your "null" model $\endgroup$
    – kjetil b halvorsen ♦
    Commented Jul 4, 2021 at 0:16

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.