How do I know a simple validation result is statistically significant when comparing machine learning algorithms? In computer science literature, we always see different algorithms are trained with a lot of data (n=100,000), and then they are tested on a test set (n=10,000). Then, often,if one algorithm NUMERICALLY is slightly bigger than another algorithm in term of accuracy (e.g., 79.1% vs 78.1%), then the author will claim his/her method is better (beat the state of the art). However, how do we know the difference in term of accuracy is significant enough? 
In computer science literature, they often don't do statistical test. So, if I am in this situation, what should I do to compare the significance of the difference between two algorithms and what tests should be used?
 A: You're using two different meanings of significance here.

how do we know the difference in term of accuracy is significant enough?

Here you're talking about practical impact, or effect size. What effect size is important? That really depends. If your financial model gives better results to the extent that you're earning an extra 100.000€/year, but switching models costs 1.000.000.000€, the 100.000€/year is not sufficiently better. If your machine learning system saves the lives of 1% of the treated people (by detecting the disease early) and the disease affects 1.000.000 people/year, it probably is an important effect.
Note that you're talking here about degrees of significance; something can be more or less significant than something else.

statistical test

Now this is a completely different question. Typically, statistical tests give only binary results, they're only compared to a pre-defined threshold (e.g. the famous p < 0.05). And crossing that threshold does not mean that surely, practical significance will be achieved. Instead, it just means that the signal you are observing is so much stronger than the background noise that you can with confidence assume there is a signal.
This signal may have zero practical importance, but it seems to be there.
Now you can do significance tests to compare classifiers and machine learning models. This depends on the type of model, and by what measure you want to evaluate them. Very often, bootstrapping is indeed used here. However, make sure you're truly interested in significance in the statistical sense, not the practical, effect size sense.
