I lightly rewrite this already outstanding /r/eli5 comment, to simplify it further. I changed "New Yorker to "Utahn" as the latter's shorter.
Suppose you want to show that, say, Texans eat more than Utahns do. What you're really trying to prove is that Texans do not eat the same or less amount as Utahns do. This statement ("Texans and Utahns eat the same amount") is something called your "null hypothesis". Hypothesis testing has the goal of disproving the null hypothesis to prove what you're trying to show.
The idea of statistical testing is to say "well, assuming that Texans and Utahns did eat the same amount, how likely would we get the data we did? The chance of getting the data you got if, in fact, they did eat the same amount is called a p-value. For instance, if we say that p = 0.05, we mean that if Texans and Utahns ate the same amount, there'd be a 1/20 chance to observe the kinds of results we did observe. The lower the p-value, the less likely your null hypothesis is true, and the more confidence you have that, in fact, Texans do eat more.
Significance is the lowest p-value you'll accept as "strong enough" evidence. Lower significance thresholds decrease your chances of a false positive (i.e., finding that Texans eat more when in fact they don't), but increase your chances of a false negative (concluding that you don't know that Texans eat more, when in fact they do). Usually 5% is the weakest significance anyone takes seriously, but for situations where there's extreme cost to a false positive, you may choose a much lower number like 0.1%.