Say your state space is $\Omega$ and your process is $X_{t}$. Consider now a new state space - $\Omega \times \Omega$. Then $Y_{y} := (X_{t-1}, X_{t})$  is a Markov process on $\Omega \times \Omega$. Now, you can use the ergodic theorem, provided you know the invariant distribution of $Y_t$. This is a distribution of pairs $(X_{t-1}, X_t)$ and we may write it as the joint distribution $\pi( x_{t-1}, x_t)$. By laws of probability, 
$$
\pi( x_{t-1}, x_t ) = \pi (x_{t-1} ) p( x_t | x_{t-1} ).
$$  

Thus:

\begin{align}
   \lim _{T\to \infty} \frac{1}{T} \sum_{t=1}^{T} \log p(x_t | x_{t-1} ) &= \mathbb{E}_{\pi( x, y )} [ \log p( y | x ) ]\\
&= \sum_{(x,y) \in \Omega \times \Omega } \log p( y | x ) \pi(x,y) \\
&= \sum_{(x,y) \in \Omega \times \Omega } \log \frac{\pi(x,y)}{\pi(x)} \pi(x,y) \\
&= \sum_{(x,y) \in \Omega \times \Omega } \log \pi(x,y) \pi(x,y) -\log  \pi(x) \pi(x,y) \\
&= \sum_{(x,y) \in \Omega \times \Omega } \log \pi(x,y) \pi(x,y) 
 -\sum_{x \in \Omega  } \log  \pi(x) \pi(x) \text{ marginalized in } y \\
&= H(X_{t-1}) - H(X_{t-1},X_t) \\ 
&= H(X_{t-1}) - H(X_{t-1},X_t) \\ 
&= -H(X_t | X_{t-1} ).
\end{align}

$H$ is the entropy function(al) and $H(X|Y)$ is the conditional entropy. According to Wikipedia: conditional entropy (or equivocation) quantifies the amount of information needed to describe the outcome of a random variable Y given that the value of another random variable X is known. 

So I think maybe you should consider the negative of the above quantity.
Regarding your last question - you can apply the same trick from above to $(X_{t-L} ,..., x_{t})$.