There are several words that can have different meanings in statistics and other fields or every day life in your question. For instance fair in the context of throwing a die means that all sides have the same probability of occurring. The same word has a different meaning in game theory, where fair price means a price you'd pay for a lottery which eliminates the profit from the payoffs.
The word expected in statistics means this:
$$E[x]=\sum_{i=1}^6p_ix_i=\frac{1}{6}\$1+\frac{1}{6}\$2+\dots+\frac{1}{6}\$6$$
where $i\in{1,2,3,4,5,6}$ - an index of the side, $p_i$ the probability to get this side, and $x_i$ - the outcome of a trial (payoff). If the payoff was in dollars and the die is fair you could easily see how this is equal to $3.5.
The outcome (payoff) that happens most often is called a mode. In the case of a fair die, all outcomes are equally probable, so the mode is not an interesting measure.
UPDATE:
So, if you think of a fair dice as a tool which generates payoff randomly depending on which side was tossed, then yes, $3.5 is a fair price you'd pay for this tool assuming there's no time value of money. There are complications, into which you can dig in if interested, things like ST. Petersburg paradox etc.
UPDATE 2:
@hxd1011 asked whether there's a physical meaning to that referring to his Example 2 with the possible outcome to be animals.
Right, the expected value is of the values of a random variable. The random variables in statistics are defined as some - usually real - values that are linked to events from the event space. Do not mix them with indices. For instance, in your example 2 let's denote the events with indices $j={1,2,3}$, then we can enumerate all possible events: $\omega_1=\text{cat},\omega_2=\text{dog},\omega_3=\text{pig}$.
Let's say we have the associated probabilities $p_1=p_2=p_3=1/3$.
If you did not define the random variable $x_i$ yet, then there's no point in talking about the expected value at all. Expected value of what? Of index $j$? It doesn't have any physical meaning as you wrote.
Let's now define a random variable $x$ on the probability space as follows (suppose we have some hypothetical game that if you guess correctly you will get some money):
\begin{align}
x&=\$10,\quad\text{if }\omega=\text{cat} \\
x&=\$20,\quad\text{if }\omega=\text{dog} \\
x&=\$30,\quad\text{if }\omega=\text{pig}
\end{align}
Now we can talk about the expected value of $E[x]$. We can calculate it easily using the equation above, it's $\$20$ of course.