In OpenAI Gym Taxi-v2 there are totally 500 states. But that's too big considering 5*5 matrix.
I think I'm missing few things?
Taxi-v2 This task was introduced in [Dietterich2000] to illustrate some issues in hierarchical reinforcement learning. There are 4 locations (labeled by different letters) and your job is to pick up the passenger at one location and drop him off in another. You receive +20 points for a successful dropoff, and lose 1 point for every timestep it takes. There is also a 10 point penalty for illegal pick-up and drop-off actions