I have a time-series dataset that is poisson-distributed, where each day I get a new additional datapoint. If I input all the data into a HMM (I am using code I found from hmmlearn in python) it does a very good job at estimating the hidden (binary) states on the older historic data. Unfortunately it does a very poor job at classifying states (and its changes) in the most recent data-points, which is actually the data that I am most interested in classifying.
Edit: The issue is that the predicted state for an observation (e.g. observation number 100, made on day 100) may be incorrectly set to state 0, and will be corrected as more information is available (e.g. observation 100 will be changed from 0 to 1, on day 105). But then it is too late. I would like to know if there are better methods/improvements to get a state estimation for observation 100, on day 100.
If I want to do a (binary) state classification, where the most recent data-point(s) is also the most important, are there any methods that are better suited for this than HMMs? I.e. are there any algorithms for classifying states in live data that are suitable here?
I am relatively new at this so any help/advice is appreciated.
'*POISSON* emissions, 2 states'
from hmmlearn import hmm
import numpy as np
import pandas as pd
df = pd.read_csv('.../Desktop/Data.csv') # path to data
df_inpt_col0 = df['Data'].copy()
np_inpt_col = df_inpt_col0.to_numpy() # convert to array
data = np_inpt_col.astype('int') # set as integer type
scores = list()
models = list()
for idx in range(10): # ten different random starting states
# define our hidden Markov model
model = hmm.PoissonHMM(n_components=2, random_state=idx,
n_iter=10)
model.fit(data[:, None])
models.append(model)
scores.append(model.score(data[:, None]))
print(f'Converged: {model.monitor_.converged}\t\t'
f'Score: {scores[-1]}')
# get the best model
model = models[np.argmax(scores)]
print(f'The best model had a score of {max(scores)} and '
f'{model.n_components} components')
states = model.predict(data[:, None])
np_states_mat = model.lambdas_[states]
#to normalize states
np_states = np_states_mat[:,0]
np_states[np_states == np_states.min()] = 0
np_states[np_states == np_states.max()] = 1
df_states = pd.Series(np_states,index=df.index)
plt.plot(df['Data'])
plt.plot(df_states)