While reading a few papers, I came across the term "gold set" or "gold standard". What I don't understand is what makes a dataset gold standard? Peer acceptance, citation count and if its the liberty of the researcher and the relevance to problem he is attacking?
Say that you want to measure a certain polluant in drinking water, the golden standard will be the method which detects it with the highest sensitivity and accuracy. Any other method can then be compared to it, knowing that -under certain conditions- the golden standard is the best (e.g.: if you need to measure the polluant on site, the golden standard will not be any method that require huge machinery to be used, as you will not be able to bring it with you).
I think the Wikipedia article explains it quite well:
In medicine and statistics, gold standard test refers to a diagnostic, test or benchmark that is the best available under reasonable conditions. It does not have to be necessarily the best possible test for the condition in absolute terms. For example, in medicine, dealing with conditions that require an autopsy to have a perfect diagnosis, the gold standard test is normally less accurate than the autopsy. Anyway, it is obviously preferred by the patients.
A gold standard is a standard accepted as the most valid one and the most used. You can apply the expression for everything... But you can always accept or critic the standard, especially in the case of a dataset.
In a challenge, this usually mean the answer to the test set, previously hidden from participants.
The term "gold standard" has been used a lot with respect to No Child Left Behind. One important component of the legislation is that it established the need for the education field to move towards interventions and practices that have been demonstrated to be effective in rigorous studies. In the NCLB materials, research designs have been classified into three categories based on the strength of the conclusions they warrant. Randomized control trials (i.e. randomized or true experiments) are the only designs that provide "strong evidence" of effectiveness. Thus they have been dubbed the "gold standard" in educational research. Well-designed true experiments are given higher precedence in the awarding of federal research grants.
I have observed the term "gold standard" in quotes more times than not, so I take it to mean something that is highly subjective. Even in the Wikipedia article some paragraphs refer to it in quotes. The OP is also referring to a "gold standard dataset" which I take it to mean a "Gold Standard" for descriminant Analysis, as in Fishers' Iris dataset being the "Gold Standard". But I am not 100% sure usage is consistant.
Golden standard or Gold standard is a term used to describe a collection of tags that can be gathered from experts. This corpora is costly because you have to get the expert skills to work in tagging, they are high-quality and accurate.
The gold standard is randomization of your assigning of sample into different groups for your experiment and a double blind.