I think the concept of 'fair' is hard to define. Since a given roll of the die will produce a deterministic result (in other words, physics determines what the result is) we can't really say that there is a certain 'probability' of rolling a one. This relates to the mind projection fallacy, which essentially says that probability is a property of one's state of information of a phenomena, not a property of the phenomena itself. Relating to the roll of a dice, the result is not just based upon the die, but also the method in which it is rolled. If we 'know' enough about a given roll (the die's material composition, it's initial orientation, the forces applied to it, the environment it will land in, etc.) we can (theoretically) model all of the motion that occurs in that roll with arbitrary accuracy and instead of finding a 1/6 'probability' of landing on a given side, we will be near certain that it will land on some side.
This all is very unrealistic of course, but my point is that the method of rolling is as important as the die's physical makeup. I think a good definition of a 'fair' die would be one in which under reasonable constraints (on computing power, time, accuracy of measurements) it is not possible to predict the result of a roll with some level of confidence. The specifics of these constraints would be dependent on the reasons you are checking if the die is fair or not.
Suppose I tell you I have an 'unfair coin' and I will give you a million dollars if you can correctly guess what side it will land on. Do you choose heads or tails?