What function of distance for the questionnaire data? I have data from questionnaire from school. First question is study program (only 2 programs) and next 35 questions are various questions (influence of friends etc.)
Possible answers for 35 questions are "definitely yes", "mostly yes", "mostly no" and "definitely no". Some of data contains missing values.
I want to do hierarchical clustering in R (hclust).
First of all I merged answers "definitely yes" with "mostly yes" and "mostly no" and "definitely no". So now I have "yes", "no" and missing values.
For "yes" I assigned value 1, for "no" -1 and for missing values 0.
My questions are:
Is my categorization of data good?
What function of distance is suitable for this type of data? hclust in R usually use Euclidean and Manhattan distance, but I think these distances are not suitable for my type of data (because of missing values).
Thanks
 A: Note that on binary data, there is little difference between the various distance functions.
For example Euclidean and Manhattan distance: Since $(\pm 1)^2 = 1 = |\pm 1|$ and $0^2=0=|0|$, the Euclidean and Manhattan distances will produce the exact same ranking.
So you might want to look into set metrics, starting with Hamming, Jaccard, Tanimoto etc.
As for handling missing values, 0 will not do the trick. You really should treat the value as unknown/missing, not "half positive, half negative".
I'd suggest that you don't completely drop the distinction, but instead assign the values "0, 0.2, 0.8, 1" to the four choices, and N/A to missing values.
Then for distance functions, you can probably just use Manhattan or one of the set based, but make sure to add proper handling of missing values - for example by ignoring that attribute altogether.
The worst case then is that two records do not have any answered question in common. Say one answered only the first question, the other answered only the second. You just cannot compare these two! With your current approach (assigning 0 to "not answered"), you make these two almost identical (which the aren't, IMHO)!
A naive but probably quite reasonable approach would be to just use
$$\frac{\text{Number of questions answered with the same trend}}{\text{Number of questions answered by both users}}$$
or
$$\frac{\text{Number of questions answered with the same trend}}{\max\left\{\#\text{Q. answered by first user},\#\text{Q. answered by second user}\right\}}$$
A: First question should not be used in clustering, since it is only assignment to any student program.
I think that it is not a good idea to transform original variables to binary type (loss of information). You can use original format with missing values and 4 point scale.
Note that R is capable to compute distance also with missing values (using interpolation technique) so do not be afraid and use standard dist() function.
