I know that we can use training data error to know if the model is underfitting, but it tells us more about the model than the data. How to know if the data is just hard to model? Are there any metrics to assess how hard the data is to model?
I don't know of any metrics, exactly, but there are a bunch of indications.
Substantive knowledge. What usually happens in your field? Way back when I was in grad school, I was at a lunch for fellowship winners. A psychologist and a physicist were talking and the physicist said "if we get R square of 0.9, we are upset that it's so low" to which the psychologist said "if we get R square that high, we know we did something wrong!"
Graphs. Graph your data in all sorts of ways. Do the relationships fall on nice straight lines? Quadratic curves? Or what?
Measurement error - How reliable are your measurements? Models work better with precisely measured variables. Latent variables can be particularly hard to measure accurately.
Also, how valid are your measurements? Measuring height is pretty straightforward. Measuring (say) strength of a national economy, not so much.
Interactions. Interactions are notorious. And the reliability of an interaction between two variables that are not very reliable is even less reliable.
Clarity and strength of underlying theory. This is related to the first point, but, how clear is your theory? Are you doing highly exploratory work, or are you confirming a well-known model for a particular population?
I am sure there are more such things, but this is a start.
Set aside some of your training data as a hold-out dataset. Fit your model on the rest. As you make your model more complex, does its performance on the hold-out dataset consistently improve? If it does, your model may be under-fitted. If not, it may already be close to optimal, or possibly over-fit.
Benchmark with models of different size
A common practice for developing machine learning models is to not only look at the best model you have, but also measure and visualize the relationship of model performance (on some validation set) with respect to model size to observe any trends visible. If a slightly smaller models than the largest one show weaker performance, then that suggests that further increases are likely to fit data better; but if your performance has reached a plateau, this indicates that the data might just be fundamentally hard to model.
The same applies to other aspects of model structure complexity - if simpler models reach similar results as complex models, that's an indication that what you get might be a fundamental limitation of data, but if more complex models perform better, then that's an indication that perhaps even the most complex model you tried is "too weak" to properly model the data and going even further could improve things.
Benchmark against expert human performance
Many tasks try to automate something which can also be performed by human judgement. Having a validation dataset processed by expert humans provides a benchmark of what is definitely possible to extract from that data (because you observed a "system" doing that), so a sufficiently powerful model should be able to perform at least as well. If the gap between model performance and human performance is large, that indicates that a more powerful model (or, potentially, more training data or more external world knowledge) might get better results, but if the gap is insignificant then that suggests that perhaps the problem fundamentally can't be modeled better.
But, as you might notice from my wording, there generally is no certainty. There is an active (and important!) niche of research on trying to provide formal guarantees, but (as far as I know) it has not yet resulted in generally applicable techniques to determine these limits for practical problems before actually trying to solve the particular task.