Can we say there is High Bias if we have high training error due to small training data size? If for a very small dataset we have a high training error, can we say that we are underfitting or have a high bias because of the low amount of training data?
Or do we use these terms (underfitting  and high bias) only for referring to issues due to low model complexity?
 A: The bias and variance are not connected to your specific sample, but to the estimation method and the underlying distribution. The variance is most often connected to the sample size, however. High training error does not automatically signify high bias, high training error can be the result of low sample size.
If you have a point estimator, say the sample mean $\bar{x}$ for the distribution mean $\mu$, then the bias is the difference between the expected value of $\bar{X}$ and $\mu$. It can be shown that the expected value of $\bar{X}$ is $\mu$, and hence, the sample mean is an unbiased estimator of the distribution mean. If you are not used to providing mathematical proofs, you can think of it this way: What happens if your sample size grows towards infinity? Will the difference between your estimator and the parameter you are trying to estimate eventually become 0?
The variance of your estimator says something about how much you can expect your estimator to vary from sample to sample, and this is often used to say something about the uncertainty of your results. Back to $\bar{X}$ as an example: Say you have a sample that gave you $\bar{x} = 13.5$. If you took another sample of the same size, how much would you expect $\bar{x}$ to differ?
Because bias and variance are connected to the underlying distribution, in most cases you cannot know their exact values, but the theoretical results can help you choose an estimator with low bias and variance.
If you have a model that you are training for classification/prediction, the same principles apply.
Some models will give you zero training error if you add enough parameters (e.g. an n-1 polynomial for an n-sized training set), at the cost of huge variance. An unbiased model could give you training error if you simply do not have enough data to train the model (e.g. $n<p$), and you have to use some default parameters to fill in.
There are no universal results you can lean on, since "model" does not have a specific definition.
