If you try a real thoroughly won't a computer find all sort of silly patterns?
Yes. (With emphasis on "silly") this is often referred to as overfitting.
If you have a huge (and growing amount) of information, would things start to match in a meaningless way?
The answer depends on how exactly the amount of information is growing, and how exactly you look for matches.
Data is usually regarded with respect to cases and variates. Variates are properties, and cases are sets of observed properties that belong together.
Example: patients are cases, and the variates are e.g. expression level of a number of genes or height, weight, eye color, shoe size, blood pressure, etc.
- If "more information" means that you measure more variates, e.g. expression levels for more genes of the same patients, then chances increase to find chance patterns across the constant set of patients.
- If "more information" means that you measure the expression levels of the same variates, e.g. a constant set of genes for more patients, then chances decrease.
This is known as the curse of dimensionality, and the Elements of Statistical Learning give a very nice explanation in chapter 2.
How much of a problem this is also depends on what kind of match you are are looking for:
- finding a distinction (classification problem) between groups of cases is easier (more degrees of freedom; thus more prone to overfitting!) than finding a function through all points (regression)
A different formulation:
- If you look for any match (any two persons at the party having the same birthday) chances increase with having more information/cases. If you ask for matches between all cases (everyone having the same bithday), chances decrease with adding new cases. But the chance to find (spurious)
What safe-guards do you put in place to get rid of these associations?
Possibly the most important safe-guard is you can test (validate) the found pattern (or predictive rule) on completely new cases (preferrably in a blinded or even double-blind way).
Unfortunately, this is very costly and so it is done far too seldom (look up debates about reproducibility in current biomedical research).
You can also calculate confidence intervals for the predictions (before going for an expensive validation study). If they are ridiculously wide (which they are if you calculate them honestly and you have too few cases), this means that you need to tone down your conclusions accordingly.
Another safeguard during the model set-up is that you restrict the model to the low complexity you can afford given the always too small* number of cases you have available.
You can and should do sanity checks based on the knowledge you have about the application and data. Even if the ability to predict correctly and the possibility to interpret the model do not always come together, it is IMHO often an important safeguard to keep to an interpretable model if you are in the risk of overfitting. If then the trained model contradicts basic e.g. physical properties, that can be a symptom of overfitting.
Example: I work with vibrational spectra of biological tissues. The spectra have the physical property that they should be smooth (and I can be very certain of that because there are 100 years of physics and chemistry both in theory and experiments behind that expectation - even if my measurements of the spectra are noisy). Linear models produce coefficients for each of the dimensions. If those coefficients are noisy (= not smooth) this is a sign of overfitting: the coefficients picked up noise from the measurements, my training algorithm failed to separate the true signal from the noise.
A completely different kind of safeguard is that you have to be very clear in thinking: It is often easy to get the right answer to the wrong question. When doing statistical hypothesis tests, it is often easy to find out how likely I am to falsely conclude that there is some pattern given that there is none or how likely I am to overlook a pattern given that there realy is some. However, what I want to know is the inverse: given that I found a pattern, how likely is this to be true? (this is close to @Ben Ogorek's false discovery rate). The link between these inverse questions is the percentage of true patterns among all possible combinations (hypotheses) I look at. And unfortunately, that is unknown. If I'm a good researcher, I'll have a better "nose" for true patterns, and this ratio will be large in the small number of tests I conduct. Blindly testing all possible combinations will have a very small ratio of true hypotheses among all tested hypotheses.
To stay in your bible example:
- As example for an informed and careful formulation of hypotheses ("good nose"), assume I predict from the old testament that the ten commandments are important, I can "validate" this "prediction" against the new testament. Conclusion would be e.g. success for the no murder, adultery, stealing/kidnapping commandmentments, mixed evidence on the sabbath.
- If on the other hand I blindly test all possible combinations of characters from the old testament against all known messages from ET (plus possibly all kinds of quotes from Shakespeare over Gödel to Steven King and Barack Obama) I'll likely find some elaborate method to produce some such quote(s) from the old testament. But the proportion of true pattern producing rules among all possible transformations of characters is minute. I therefore expect that "sucesses" are likely false positives. I still expect that
"validating" the transformation rule that produced the pattern for the old testament will fail to produce sensible patterns on the new testament (showing that the patterns were false positives and the pattern-producing transformation was overfit).
I'd say it is so unlikely that if it is found to work also on the new testament I have to rule out dishonesty (e.g. sneak preview on the new testament during model parameter tuning) before accepting that validation (compare to plausibility in one of @Ben's links).
* I'm told there are fields where enough cases are available. However, personally I have not yet had that pleasure.
You may be interested in reading up about no free lunch theorems. One consequence of the NFL-theorem(s) is that the important "ingredient" that makes one algorithm (heuristic) more successful than another is that is better matched to the problem. Thus, including (correct) knowledge about the problem/application and the type of data into the algorithm* can make a difference. However, these choices also include hyperparameters that tune or steer the overall behaviour of the training algorithm. Good choice there will be possible depending on the experience of the data analyst with the method in question.
* or choosing an algorithm according to its suitability to this type of problem and this type of data (and with which the data analyst has sufficient experience to conclude a good set of hyperparameters).