1
$\begingroup$

I have a time-series dataset that is poisson-distributed, where each day I get a new additional datapoint. If I input all the data into a HMM (I am using code I found from hmmlearn in python) it does a very good job at estimating the hidden (binary) states on the older historic data. Unfortunately it does a very poor job at classifying states (and its changes) in the most recent data-points, which is actually the data that I am most interested in classifying.

Edit: The issue is that the predicted state for an observation (e.g. observation number 100, made on day 100) may be incorrectly set to state 0, and will be corrected as more information is available (e.g. observation 100 will be changed from 0 to 1, on day 105). But then it is too late. I would like to know if there are better methods/improvements to get a state estimation for observation 100, on day 100.

If I want to do a (binary) state classification, where the most recent data-point(s) is also the most important, are there any methods that are better suited for this than HMMs? I.e. are there any algorithms for classifying states in live data that are suitable here?

I am relatively new at this so any help/advice is appreciated.

'*POISSON* emissions, 2 states'

from hmmlearn import hmm
import numpy as np
import pandas as pd

df = pd.read_csv('.../Desktop/Data.csv') # path to data 

df_inpt_col0 = df['Data'].copy() 
np_inpt_col = df_inpt_col0.to_numpy() # convert to array

data = np_inpt_col.astype('int') # set as integer type
scores = list()
models = list()

for idx in range(10):  # ten different random starting states
    # define our hidden Markov model
    model = hmm.PoissonHMM(n_components=2, random_state=idx,
                            n_iter=10)
    model.fit(data[:, None])
    models.append(model)
    scores.append(model.score(data[:, None]))
    print(f'Converged: {model.monitor_.converged}\t\t'
          f'Score: {scores[-1]}')

# get the best model
model = models[np.argmax(scores)]
print(f'The best model had a score of {max(scores)} and '
      f'{model.n_components} components')

states = model.predict(data[:, None])
np_states_mat = model.lambdas_[states]            

#to normalize states
np_states = np_states_mat[:,0] 
np_states[np_states == np_states.min()] = 0
np_states[np_states == np_states.max()] = 1

df_states = pd.Series(np_states,index=df.index)

plt.plot(df['Data'])
plt.plot(df_states)
$\endgroup$
2
  • 1
    $\begingroup$ Can you be explicit about what is poor about the predictions? Does it have to do with the plot your code renders? If so, please edit to include this information. $\endgroup$
    – Sycorax
    Sep 20 at 13:07
  • $\begingroup$ Not specifically with the plot. The issue is that the predicted state for an observation (e.g. observation number 100, made on day 100) may be incorrectly set to state 0, and will be corrected as more information is available (e.g. observation 100 will be changed from 0 to 1, on day 105). But then it is too late. I would like to know if there are better methods/improvements to get a state estimation for observation 100, on day 100. $\endgroup$
    – litmus
    Sep 21 at 8:14

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.