4
$\begingroup$

In my book (Introduction to data mining by Tan, Steinbach and Kumar - chapter 3), in the section about visualization, it's written:

Another general motivation for visualization is to make use of the domain knowledge that is "locked up in people's heads." While the use of domain knowledge is an important task in data mining, it is often difficult or impossible to fully utilize such knowledge in statistical or algorithmic tools. In some cases, an analysis can be performed using non-visual tools, and then the results presented visually for evaluation by the domain expert. In other cases, having a domain specialist examine visualizations of the data may be the best way of finding patterns of interest since, by using domain knowledge, a person can often quickly eliminate many uninteresting patterns and direct the focus to the patterns that are important.

I don't fully understand it. What's meant by domain knowledge? Who is a domain expert or specialist? Could someone explain the paragraph in simple terms?

$\endgroup$
1
  • 6
    $\begingroup$ "domain knowledge" is presumably whatever background knowledge people with data have about their problem. Managers, botanists, plumbers, lawyers should know much about management, botany, plumbing, law that is relevant to interpreting their data. In that strong sense they are experts in their field in a way that a statistician or data miner may not be. The passage you cite appears just to be saying that people in any field know a lot that they can use when looking at a visualization. (Good idea always to give the reference with a question like this.) $\endgroup$
    – Nick Cox
    Commented May 29, 2013 at 6:08

2 Answers 2

6
$\begingroup$

I think the key is the last clause, which suggests the author means a domain expert to be someone who has internalized important data patterns enough to quickly recognize them (or their absence) in a visualization. To get to that point, the expert must understand the normal relations between relevant variables so that unusual relations stand out.

Someone who's not an expert in the problem domain can spot patterns like correlation and outliers but not readily know if such patterns are important for the given variables.

$\endgroup$
6
$\begingroup$

As @NickCox explained a “domain expert” is typically someone who knows about the field or issue at hand. The implicit point is that the statistician/modeler/data miner and the domain expert are both experts but not in the same thing. One knows about data analysis and the other about whichever process/phenomenon the data pertains to. Simply calling them “experts” would be confusing.

As far as I know, the phrase comes from software engineering because to build a computer system, you need to understand the process it is supporting (which people need access to what, where does which information come from, which reports/output should be produced for whom, etc.) Here process can be a manufacturing process, an administrative process, a medical diagnostic, anything really. For example if we are talking about managing the performance of a chemical process, domain knowledge would be knowledge of chemistry.

It is generally difficult for people who are used to deal with the process everyday to describe it formally or to list all the things they know about it. Some things that are obvious to them might even fail to come up in the discussion precisely because they are obvious. There are organizational ramifications as well, people might be unwilling to reveal how they do their jobs (for example if they are “cutting corners” or if their expertise provide them with some career advantages). Often, what actually goes on on a factory floor or in some administrative process is very different from what management think is happening or what official rules imply. This is why it is a challenge to extract relevant information and people have been worrying about tools to extract that information from those who have it, the “domain experts”.

In your paragraph, the idea would be to use visualization to communicate preliminary results (e.g. relations between different variables revealed by the data mining effort) to people who know the process but might not be familiar with data analysis and numerical summaries. They would then presumably use their domain knowledge to suggests improvements, interpretations or new hypotheses for the data miner to explore.

Incidentally, many machine learning efforts (e.g. Kaggle, “data hackathons”) seem to proceed, rightly or wrongly, from the assumption that much can be achieved without access to domain knowledge by modeling away with a set of standard techniques and a lot of cleverness to fiddle with the tools and put it all together.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.