1
$\begingroup$

Background: I am performing a pattern matching on some data which is being generated real time. I perform the pattern matching on this newly generated data at every 1 second. Every time a match is performed by the pattern matching algorithm, it gives me the value of best match along with some statistics on how good the match was. There are around 8-10 such performance statistics which are being provided by the matching algorithm each time it outputs the best possible match. Now the match given by pattern matching might be a true positive or a false positive. The problem statement here is to figure out whether the match performed was an actual match or not. To determine this, I can use the 8-10 matching quality assessment parameters. One single parameter is not sufficient to determine this. Now given there are only two possibilities: the match given by algorithm is the most optimal match or it is not. I have performed a logistic regression on this data. The pattern matching algorithm gives me the best match, I compare it with what actual match should have been, and assign a flag 1 if it was correct or 0. I run a logistic regression on this where the 8-10 matching quality assessment parameters are the predictor variables. Now I can get a value between 0-1 which would represent the probability that the matching algorithm worked.

Question: As it turns out the time series has a very peculiar quality that whenever there is a good match performed by the pattern matching algorithm it happens in sets i.e. if at t=0 there was a good match, at t=1,t=2,t=3 also there would be good matches. And the probability of match (coming from equation of logistic regression) first increases and then decreases so there's a local maxima every time a good match happens. This makes sense because I am dealing with a continuous real time data, there won't be abrupt changes in my pattern. Observing this I feel I can leverage this quality of my time series. So are there any statistical methods where if event at t=t1 has probability p=p1, and t=t2 has p=p2 and so on and its known that locally the behavior is first the probabilities keep on increasing and then they decline there can be a model developed around it. I was reading about bayesian time series modeling but at first look it seems it might not fit here. In markov chains the currents state is independent of past state, so I think I am dealing with the opposite, there seems to be a good correlation between current state and past state (but only when I am getting a good match otherwise not).

PS: the pattern matching is happening in a way (and on such a data) that actual matches are very less, most of the data generated real time doesnt match with anything in my base database. So current state = 1 happens with a very less frequency.

$\endgroup$
  • $\begingroup$ Yes, According to Markov property future is independent of past but hidden markov models give you transition between hidden states derived from entire output sequence given in training of models. So i feel in your case, HMM would definitely work. $\endgroup$ – Arpit Sisodia Feb 26 '17 at 12:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.