Is it possible to evaluate a given model without having access to its fit method? I have a data set with one real-valued feature and a real-valued target. Someone has used this data set to fit a model (a regression). I get a results of this fit, which is a single function mapping from the feature to the target.
My question is if it is possible to estimated the level of over-fit (or under-fit) just by looking (in a broad sense) at this single function and the given data set?
I guess that it should be possible because by a visualization one could potentially see if the function has some systematic deviations from the data (for example data points assume a quadratic dependency and the model is just a linear function). So, in this case, we can conclude that we have an under-fit.
Similarly, in some cases we could see that the function is too sensitive to noise and do not reflect real dependency. So, in this case, we can conclude that we have an over-fit.
However, I would like to have a more rigorous (quantitative, automatic) method instead of a manual visualization. Moreover, if we have several features instead of one, a visualization becomes problematic.
I would like to emphasize again, that the only thing that I have is a model (function mapping from features to targets) and a data set that was used to find this function (so I have only in-sample results). In other words, I cannot retrain a model and, let's say, run a K-Fold cross validation.
 A: 
 My question is if it is possible to estimated the level of over-fit (or under-fit) just by looking (in a broad sense) at this single function and the given data set?

Consider: you can design an experiment that collects data suitable for evaluation of a particular model performance aspect, which you then measure by comparing predictions with reference for the gathered samples/cases/specimen and summarizing these differences in a suitable fashion. E.g., as RMSE. This needs only predicted value and reference, your model can be entirely a black box.
Some kinds of evaluation of prediction quality are possible, others not. Which, will depend on how the data set was put together.
If you can only use the data that was already used for training (in addition to the restriction on the one given (even black box) model), this will severely hamper your ability to spot overfitting.
How useful any figure of merit is if calculated on training data as opposed to an independent set of cases (and there may be various "levels" of independence) is a matter that you need to carefully judge. In my field, they can serve only as a stepping stone on the way to figures of merit based on independent cases.
In addition, an experiment (and in consequence, the data set) may be designed so that various influencing factors can be measured, or cannot be measured.
Regularization will often shrink the predictions towards the mean target during training. You can visualize this (in analytical chemistry, the so-called calibration plot plots prediction over reference and will clearly show this) or you could write down e.g. the regression function target = f (reference) and thus check for additive or multiplicative bias.
Also, this allows you to measure certain aspects of bias. The "diagnosis" of over- or underfitting in addition requires you to weigh this against variance-type error. I.e., whether the model found an acceptable compromise or not, and acceptable will depend on application as well as e.g. what training data was available (the best possible compromise on a small data set may have considerably more systematic and more random error than the best possible compromise on a larger data set).
Or, you may measure a certain aspect of overfitting by looking into repeated measurements of the same case which differ by some measurement noise. You may say the model is overfit if this measurement noise is amplified by the model and leads to unacceptably large differences in the predicted values. Whether you do this graphically or write down e.g. the variance is again your choice.
This is even possible for trainig data if the training set was designed accordingly. If it wasn't: no chance.

If you can get new data and get it predicted by the one model that is available, more is possible. I.e., you could design validation experiments that allow you to measure various aspects of performance/the influence of various factors on the prediction quality. This would be very hard to automate, since it requires substantial knowledge on potential influencing factors.
You could also do what I as a chemist would call perturbation experiments with your black box regression model to measure sensitivity to particular influencing factors, while machine learning folks may call the same approach augmentation. Or dummy samples (@Mayeulsgc).
Yet another data set may be representative for the expected application distribution of cases, and thus allow you to measure expected total error.
If you want to estimate performance for future samples, you need a data set with future samples. So for example, this would not be possible for the data set you describe.
And so on.
A given data set may be perfectly suitable to check one or more such aspects of predictive performance.

If you have the model as function and coefficients, and just lack the training algorithm that produced this model, and you have additional knowledge about the data at hand, you may be able to go a bit further. E.g. for certain types of data and models I frequently work with, I can spot certain aspects of overfitting (visually) by looking at the coefficient pattern. (For these data, there should be a certain smoothness for physical and mathematical reasons, if the coefficients look noisy, they are :-). The converse is not true: they may look smooth but still be overfit)
A: Suppose for example your data has 5 points and the fitted function is a 5-th degree polynomial that exactly passes through all of them. Can you say just by looking at the data if this is an over-fit ?
It could be, for example, that those points represents precise measurements of some physical system (such as the trajectory of a planet) which has a theoretical model that is known to be a 5-th degree polynomial, where the coefficients are free parameters. Then 5 data points are enough to determine all the unknown parameters, and the fit is perfectly valid.
On the hand it could be of course that such a model is not known a-priory, or that the measurements have large uncertainties, and in that case the same function would be completely over-fitted and meaningless. (Conversely,  someone could fit a linear function when the model is known to be 5-th degree,  and the result might look visually reasonable, but still would be a severe "under-fit").
So the bottom line, as this example shows, is that you need to have some assumptions about the inherent uncertainties in the data (namely, a statistical model) and what class of models are reasonable for your data. (Note that some models, such as a simple linear or quadratic function, are more easily recognized by a human brain than others, so when you think in terms of patterns that are visually clear to you, you are in a way preferring this particular class of models).
If you can make such assumptions then you can essentially fit the same data to different models and apply some model selection criteria, which will tell you if a different model (either simpler or more complicated) describes the
data better. Note that statistical goodness-of-fit tests are always relative, either explicitly or implicitly, to some alternative models. That is, they tell you whether some alternative model describe the data better, and they can differ by which classes of alternative models are effectively considered.
