I'm new to datascience (from a medical/medical science background).
My supervisor (social sciences background) asked me to assist in rewriting a paper where we do a cluster analysis for a questionnaire (unvalidated) with 63 questions relating to attitudes and behaviours and then to give insights about this data using an ANOVA on outcome variables (also related to attitudes and behaviors, basically an average of the responses to questions that seem related to a specific theme) based on questions from within this dataset. A previous paper used the questionnaire and stopped at defining the clusters ("we found these clusters, this is how each of the clusters answered each of the questions"). My supervisor wanted to gain additional insights to find determinants of the outcome variables of interest.
I ended up doing a PCA on the 63 questions and using those for a cluster analysis. I spend quite a bit of time in the results section defining the clusters based on their similarities/differences, but then in the next results section rather than doing an anova (or in my case multivariate kruskal wallis tests), on the exact same data i used to make up the clusters (which seems super not valid), I decided to do a hierarchical multiple regression on various demographic factors that were not included in the cluster analysis, and then in the final mode I added the clusters. Obviously, the clusters are most predictive of the outcome variables (because they use the same data).
I'm now trying to write my discussion and i need to say that what i've done is not valid, but then suggest a more useful way to perform this, and the usefulness of cluster analysis for describing behaviour/attitudes of a population in general. Is it reasonable to suggest that clustering be done on either demographic variables or on the outcome variables, and then compare clusters to the variables not included in the analysis? Is that valid?
What analyses can be validly performed after clusters have been defined, and how would those analyses be useful?
I've spent months trying to teach myself responsible statistics, and any help would be hugely appreciated.