You can calculate an F statistic for any data set. It can always be used as an informal "explained-variance-to-unexplained-variance" ratio.
However, unless you are using a linear model and also meet all the assumptions, there's no way to argue that the F statistic "should" follow the F-distribution under the null hypothesis. As such, any p-value calculated from the F-distribution is basically meaningless and provides no basis for accepting or rejecting any hypothesis.
Even closely related models like Ridge/LASSO/ElasticNet (l1/l2 regularization), robust regression (Huber loss), and GLMs don't technically meet the assumptions of the F test and therefore the F statistic isn't generally used in these cases. So of course neural nets are right out.
There are alternatives to relying on the theoretical F-distribution, such as bootstrapping and cross-validation. While these techniques are not often used with the F statistic, they are generic enough that there's no reason why they couldn't be applied to the F statistic. For example, to apply the bootstrap, take a bootstrap sample of your original data, fit both models, and calculate the F statistic. Repeat that process 1000 times. You now have 1000 F statistics in a vector. Sort it. The 50th and 950th entries are respectively the lower and upper bounds of a non-parameteric 95% confidence interval. If that interval includes 1, you cannot reject the null hypothesis. If the interval does not include 1, you can reject the null hypothesis with an alpha of .05.
Cross validation does a more complicated dance of splitting training and validation data along folds, but the final result is also a estimate of the confidence interval of the given statistic. (The benefit of cross validation over the bootstrap is that cross validation measures out-of-sample error which is considered a more reliable estimate of a models generalization performance on new data it hasn't seen before. NN are prone to overfitting and CV helps control that risk.) However, in the interest of full disclosure, I have never seen anyone use the F statistic as a metric when using the bootstrap or doing cross validation. Nor do I think I would recommend it in practice, although it is an interesting idea to discuss. There are other metrics which are more commonly used and probably easy to interpret.
The standard way to compare two blackbox models such as NN is to perform cross-validation and to estimate the mean and standard deviation for some loss metric. MSE (Mean Squared Error) being most often used for regression. Let's say we did 5-fold CV: because we took five measurements, we can estimate the mean and standard deviation of MSE, and if one model's loss (averaged over all 5 runs) is several standard deviations lower/better than the other's then we say that the first model is the better.
(Obligatory joke: why is a weevil like a loss function? Because you should always take the lesser of two weevils!)
If you only have one model in the running and want to know if it did better than random chance you compare it to a degenerate linear model with no predictors and just a single intercept term - in other words, a model which always emits the mean of the dataset as its prediction for any input.
I would recommend Applied Predictive Modeling for a solid practical introduction to the topics of model selection, or Elements of Statistical Learning, Chapters 2 and 7 for a more theoretical point-of-view on the bias-variance trade-off and why cross-validation "works."