I would like to use (hidden) Markov Chain to predict $X[t+1]$ stock price. Historical data for top biggest 500 companies for last 10 years will be used for training.
The error would be sum of $PredictedX[t+1]-Actual[t+1]$ over 500 companies, given the previous $Actual[t, t-1, ..., t-n]$.
The problem - what if there are groups with different behaviour in those top 500 companies? Markov Chain would average it and the specific group behaviour would be lost.
First approach - we can group companies manually - for example into sectors, like Tech, Services, Auto, etc. and feed the
groupId into Markov Chain.
But, what if there are other, better ways to group? By better I mean - the prediction error would be less. Maybe grouping by sector is not really helpful, maybe the trend for last year, or last month could be better grouping criteria, or if there was or wasn't a significant drop in the last 30 days, or how many years company exists etc.
One approach - try to came up manually with bunch of ways to group companies. Try all of them and find the best one.
I wonder - if there are some ways to find the groups automatically? Let the Markov Chain to decide what it needs. Something like autoencoder?