0
$\begingroup$

I have been trying to create a Transition Matrix using the data from 2000 entities over 40 observations (Years). I have ranked the data into percentiles, for example the highest value entity in a given year being in percentile 1 and the lowest value entity in 0.01. I want to see the transition probability of an entity in say percentile state 0.99 to transition to any other state X such as 0.98.

After that I used this code in R from the 'markovchain' package

FITTED_DATA <- markovchainFit(data=Entity_DATA[,2:41],name="DATA_FITTED")

There are 41 columns with the first indicating the entity ID and the following columns representing the observed years. I get a DTMC object with a 100x100 transition matrix with each row summed up equal to 1. Yet some transitions are 0.

I did the same thing with the data ranked in deciles not percentiles (from 0.1 to 1) and get no zeroes in the transition matrix, but I fear I give up a considerable deal of accuracy doing it this way. This way I end up with a 10x10 transition matrix.

Is the reason I get zeroes in the transition matrix when I use percentile ranked data, because some transitions are not in the data i.e. the transition from say 0.98 to 0.97?

Is there a better way to come up with a sensible transition matrix? I appreciate any kind of help or hints one can give me.

Improve this question
$\endgroup$
4
  • $\begingroup$ What makes the zeros not "sensible"? What criteria do you offer for recognizing a "sensible" transition matrix, given that many real-world transition matrices are chock full of zeros and given that your estimates are highly uncertain due to the relatively small amount of data? $\endgroup$
    – whuber ♦
    Commented Jan 20, 2022 at 15:54
  • $\begingroup$ Thank you for the comment! As far as I understand the zeroes imply that from a given state there is 0 chance/probability of a transition. Yet there are no such restrictions on the entities in my data. Any transition from any state is possible, thus if a transition is very unlikely I would expect a small value, but not outright zero. I suppose the problem I'm facing is due to the limited breadth of my data? Unless I get richer data I would have to make due with the centile rank transition matrix? $\endgroup$
    – Hermann Josef
    Commented Jan 21, 2022 at 5:00
  • $\begingroup$ You over-interpret the zeros. They only mean that such transitions are sufficiently rare that they weren't detected in your small dataset. Given that there are $100^2=10^4$ possible transitions between percentiles, you would need an enormous dataset to detect every transition. Your problem ultimately concerns how to work with these uncertain estimates of transition probabilities. Perhaps you could describe the actual statistical problem you are trying to solve with this technique, for then we might be able to suggest effective solutions. $\endgroup$
    – whuber ♦
    Commented Jan 21, 2022 at 14:31
  • 1
    $\begingroup$ Thank you! Your comments have already helped me a lot to gain a better understanding. I'm essentially trying to replicate what is done in this paper by Henderson et al. 'How long must a firm be great to rule out chance? Benchmarking sustained superior performance without being fooled by randomness' (2011). More specifically I want to use the transition matrix to simulate the lifespans of the real entities. I was stumped, because in their transition matrix there seems to be no zeroes, but their data is much richer as they have 20,000 firms while I have a mere 2000 entities. $\endgroup$
    – Hermann Josef
    Commented Jan 23, 2022 at 7:37

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.