In one of the easy cases of inverse reinforcement learning, we try to derive the unknown reward function assuming that an optimal policy is known and MDP is completely known.
Let $P_{a_1}$ be the probability transition matrix from one state to another, following the optimal policy $\pi(s) = a_1$. Assuming $n$ states, then this matrix is $n \times n$. On the other hand, $P_a$ is the transition matrix of all other policies that are not $a_1$. Still $n \times n$.
If $I$ is the identity matrix and $\gamma$ is the reward discount factor (scalar), what is the dimension of $R$ in the follow rule? (from Algorithms for inverse reinforcement learning, by Ng and Russell, 2000)
$$(P_{a_1} - P_a)(I - \gamma P_{a_1})^{-1}R \geq 0 $$
Is $R$ a vector? or $n \times n$ matrix? And what would this mean?