I'll rewrite the cross entropy in terms of distributions $q$ and $z$, so we can save your notation $t$ and $p$ for explicitly talking about classifiers. The cross entropy is:
$$H(q, z) = -\sum_{x} q(x) \log z(x)$$
The information theoretic meaning of the cross entropy is this: Say $q$ is a distribution that generates some data. We want to encode the data using a set of symbols (e.g. to transmit it over a channel, or store it). We'd prefer to use short codes, because they require fewer resources to transmit/store. There's a fundamental relationship between code length and entropy. It turns out that, for the optimal code, the average number of bits per symbol is given by the entropy of the distribution. No encoding can be shorter than this without destroying information. The intuition is that we should assign short codes to high probability events (because they'll occur more frequently) and longer codes to low probability events. Looking at the definition of entropy:
$$H(q) = -\sum_x q(x) \log q(x)$$
This is the expected value of $-\log q(x)$, which is the optimal code length for event $x$ (given in bits if the log is base 2, or nats if it's the natural log).
But, say we don't know $q$, and instead we have some 'proxy' distribution $z$. We can design an optimal code for $z$, even though the data are generated by $q$. In that case, how many bits/nats per symbol will we use on average to encode the data? The answer to this question is given by the cross entropy $H(q, z)$. Looking at the definition of the cross entropy, we can see that it's the expected value of $-\log z(x)$ with respect to distribution $q$. Here, $-\log z(x)$ is the optimal code length for event $x$, using the code based on $z$. The expectation is taken over $q$ because that's what generated the data. As our proxy distribution $z$ becomes closer to the data-generating distribution $q$, our code will be more efficient, and the cross entropy will be lower. It's minimum possible value is $H(q)$, when $z$ = $q$.
Bringing things back to classification, we have $t$ as the true/observed distribution of class labels, and $p$ as the classifier's estimated distribution over class labels. Plugging these concepts into the description of cross entropy, it makes sense to use $H(t, p)$ because $t$ is the data-generating distribution and $p$ is the 'proxy distribution'.
Here's another argument. When using hard class labels, people treat distribution $t$ as the empirical distribution, which assigns probability $1$ to the true/observed class label $i$, and $0$ to all others. In that case, the cross entropy reduces to: $-\log p(i)$. Summing over all data points, this is just the negative log likelihood. In that case, minimizing the cross entropy loss is equivalent to maximum likelihood estimation.
