What does the true level of significance mean in data mining? There is a formula α*=1-(1-α)^c/k, 
a* - true level of significance.
a - nominal level of significance.
c - the number of candidate regressors. 
k - the number of finally selected regressors. 
I don't quite understand how the true significance gets calculated and the mechanic behind it. Hope someone can explain to me, thank you so much!
 A: Here is the quotation:

When a search has been conducted for the best $k$ out of $c$ candidate explanatory variables, a regression coefficient that appears to be significant at the level $\hat \alpha$ should be regarded as significant at only level $$\alpha = 1 - (1-\hat\alpha)^{c/k}.\tag{2}$$

A footnote explains its provenance:

This rule of thumb is obtained by equating the probability of accepting the null hypothesis for all $k$ explanatory variables in the absence of search, the tests being conducted at level $\alpha,$ with the probability of accepting the null hypothesis for all $c$ potential candidates when the test is conducted at claimed significance level $\hat\alpha;$ i.e., $\hat\alpha$ achieves $(1-\alpha)^k = (1-\hat\alpha)^c.$

Strong unstated assumptions are needed to justify this:

*

*All null hypotheses are true.


*The actual false positive rates equal their nominal rate $\alpha.$


*The tests must be (statistically) independent.
The first assumption is required for assessing the Type I (false positive) error rate.  When working with multiple hypotheses, it's convenient to compute the chance of accepting a true null hypothesis: by definition, this is at least $1-\alpha.$
Assumption (2) is needed to replace this by its bound $1-\alpha.$  (This assumption tends to be valid for simple hypotheses with two-sided alternatives based on continuously distributed sample statistics, but otherwise might be far wrong.)
Provided the tests are independent (assumption (3)), we may multiply those chances to obtain the chance of accepting all $c$ null hypotheses, yielding $(1-\alpha)^c.$
The idea, then, is that $p = p(\alpha,c) = 1 - (1-\alpha)^c$ is the greatest chance you are willing to take of making at least one incorrect decision among those $c$ tests.  Your restriction to $k$ candidate variables results in you conducting only $k \le c$ tests.  Presumably $p$ has not changed, whence you should be conducting your tests at a level $\hat\alpha$ for which $p(\alpha,c) = p(\hat\alpha,k).$  Equating their complements gives
$$(1-\alpha)^c = 1 - p(\alpha,c) = 1 - p(\hat\alpha,k) = (1-\hat\alpha)^k$$
whose solution is
$$\alpha = 1 - \left(1 - \hat\alpha\right)^{k/c}.$$
In practice, you ought to think hard about assumptions $(2)$ and $(3),$ especially how independent those original $c$ tests might be.  With some analysis you might be able to estimate an effective number $c^\prime$ of "degrees of freedom" among those tests and replace $c$ by $c^\prime$ in formula $(2).$
Reference
Lovell, M. (1983). Data Mining. The Review of Economics and Statistics, 65(1), 1-12. doi:10.2307/1924403
A: If we do a single test (on a single regressor or other case) with significance level $\alpha$ where the null hypothesis is true and all the assumptions hold, then the probability of a type I error is $\alpha$.  But what if we do 2 tests?  Each one has a probability of type I error of $\alpha$, so our overall chance of having made at least one type I error is greater than $\alpha$.  As we do more tests, the chance of at least one type I error increases, and that is what the formula is trying to capture (but note that there are certain assumptions behind that formula as well).
Edit to expand based on comment
I am not familiar with the exact formula that you show (not sure exactly how they are having $k$ play its role).  But I can show a variant.
The family wise error rate (or true rate of significance) is the probability of making at least one type I error given that all the null hypotheses being tested are true.  The probability of making at least one type I error, falsely rejecting at least one true null, is one minus the probability of not rejecting any true nulls.
The probability of not rejecting the first true null is $1-\alpha$ and the probability of not rejecting the second true null is $1-\alpha$ (and the same for the rest of them).  If the probability of not rejecting the first and the probability of not rejecting the second are independent then the probability of not rejecting both (not rejecting 1 and not rejecting 2) is $(1-\alpha) \times (1-\alpha) = (1-\alpha)^2$.  So the probability of not rejecting $c$ null hypotheses given that all are independent is $(1-\alpha)^c$ and so the probability of rejecting at least 1 is $1-(1-\alpha)^c$.  Note that this depends on the assumption of independence, so this is often considered an approximation (how good the approximation depends on the level and type of dependence).
