While reading a few papers, I came across the term "gold set" or "gold standard". What I don't understand is what makes a dataset gold standard? Peer acceptance, citation count and if its the liberty of the researcher and the relevance to problem he is attacking?
Say that you want to measure a certain polluant in drinking water, the golden standard will be the method which detects it with the highest sensitivity and accuracy. Any other method can then be compared to it, knowing that -under certain conditions- the golden standard is the best (e.g.: if you need to measure the polluant on site, the golden standard will not be any method that require huge machinery to be used, as you will not be able to bring it with you).
I think the Wikipedia article explains it quite well:
In medicine and statistics, gold standard test refers to a diagnostic, test or benchmark that is the best available under reasonable conditions. It does not have to be necessarily the best possible test for the condition in absolute terms. For example, in medicine, dealing with conditions that require an autopsy to have a perfect diagnosis, the gold standard test is normally less accurate than the autopsy. Anyway, it is obviously preferred by the patients.
I have observed the term "gold standard" in quotes more times than not, so I take it to mean something that is highly subjective. Even in the Wikipedia article some paragraphs refer to it in quotes. The OP is also referring to a "gold standard dataset" which I take it to mean a "Gold Standard" for descriminant Analysis, as in Fishers' Iris dataset being the "Gold Standard". But I am not 100% sure usage is consistant.
The term "gold standard" has been used a lot with respect to No Child Left Behind. One important component of the legislation is that it established the need for the education field to move towards interventions and practices that have been demonstrated to be effective in rigorous studies. In the NCLB materials, research designs have been classified into three categories based on the strength of the conclusions they warrant. Randomized control trials (i.e. randomized or true experiments) are the only designs that provide "strong evidence" of effectiveness. Thus they have been dubbed the "gold standard" in educational research. Well-designed true experiments are given higher precedence in the awarding of federal research grants.
This question relates only figuratively to gold.
A “gold standard” is an accepted standard that people can look to as an accurate and reliable reference. In medicine, for example, researchers often refer to blood assay as a gold standard for checking patients’ medication adherence. As with many such standards, however, because it is expensive and time-consuming, researchers search for quicker and less expensive, but still consistent ways of achieving comparable results. They gauge the value of their methods by comparing them to those achieved using the so-called gold standard.
In the case of a dataset, a gold standard would be one accepted as the most accurate and reliable of its kind, which could be used as a measure of those qualities in other datasets.