1
$\begingroup$

*Background:* I am doing an academic study that is more exploratory in nature (my advisor did not want me to develop any hypothesis). So I read books and conducted interviews to try and gather a large list of items that people feel are important and influential to the topic being studied. From that list, I developed them into questions to ask in a survey which I got a good number of responses (n>500). The 50 questions survey include a list of 35 items that people think are important for the topic and 15 demographic questions about the respondent.

*My Problem:* Now I have a problem on choosing the correct analyses. What I want to achieve is through the analyses of the data, come up with a model that consists of all the items that people find as important for the topic being studied. But at the same time, my advisor doesn't want me to reduce the items from the survey too much because he feels all the items in the survey are important, but rather, based on the items respondents indicated as important (have high mean score?), group them together to form factors in a model (without reducing too much data) and look for any relationships. I am not sure which type of analyses would be suitable for this.

So far I think I chose the wrong type of analysis so I appreciate some help. After reading up on potential analyses that are suitable for studies that do not have hypotheses, I conducted a Principal Confirmatory Analysis on the non-demographics questions and formed them into 4 factors. But in doing so, there were about 10 non-demographic items that were deleted. My advisor is now asking me if PCA is the best analysis to choose because he saw that most of the mean scores for all the questions were high (at around 3-4 out of 5) so he assumed those are all important and should all be included into a model. I believe what he wants is all the questions with a high mean score which indicate that the respondent felt was important should really be kept in the model and should not be deleted from the model.

Ultimately, I am looking for suggestions as to what analyses would be suitable for the outcome I am trying to achieve...to develop a model that consists of all the items that people find as important from the survey (and if the mean score is any indication, a very high number of the items are score 3-4 out of 5 in the likert) I would appreciate any suggestions.

Improve this question
$\endgroup$
1
  • $\begingroup$ There is a fundamental contradiction in any instruction to "group [items] together to form factors in a model (without reducing too much data)." Grouping items into factors is data reduction. It will by definition reduce the amount of information as compared to using all 35 opinion items. (Though there are often good reasons for doing so, as the factor analysis literature explains.) Otherwise, it'd be helpful if you'd state the topic and list a sample opinion item including its response options. $\endgroup$
    – rolando2
    Commented Jan 30, 2012 at 4:12

1 Answer 1

Reset to default
2
$\begingroup$

PCA probably isn't what you're after here. You probably want Factor Analysis, which is typically used to investigate the latent factors supposedly underlying a measure.

However, it seems from your question that you want to reduce the number of items on your scale (which is a laudable goal too often forgotten in social science), so I would suggest that you investigate item response theory. Item response theory is a method which allows you to decompose the scores on your measure into two parts; person abilities and item difficulties.

It was originally developed for answers with a true/false answer, but has been generalised to personality/achievement tests since then. I personally use R for my statistical analyses, and there are three packages that would probably help with this kind of modelling; namely mokken, eRm and ltm.

A suggested workflow might be the following:

use the mokken package to test the assumptions required for IRT. Following this, attempt to fit a rasch model (the simplest form of IRT) using eRm. eRm has an extremely good vingette which you can get access to by calling vignette(eRm). If these models do not fit, then you can use the ltm package, which can fit two and three parameter models.

Your problem is actually quite interesting, in that you are aiming to discern which items people find most important. This can be done using an IRT approach by looking at which items are perceived as the least difficult, these will be the items that most people consider important.

One major advantage of IRT is that you can use it to select questions which will assess importance more quickly, by only keeping those which are most informative. HTH.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Hi Richie, so would exploratory factor analysis be more suitable for my scenerio (to find the items that are deemed as important by the respondents without reducing the items too much to form the model as what happened when I did PCA)? I may have chosen the wrong analysis with PCA in that it is more a data reduction method. However I $\endgroup$
    – Pete
    Commented Jan 27, 2012 at 15:52
  • $\begingroup$ However I am unsure if EFA is a data reduction method as well so I am not sure if its suitable for my case. Thanks for your suggesting for IRT, I will read more on that as well, but you also mentioned it helps in selecting questions with importance more quickly by keeping those that are most informative. Would that be a problem in my case, where my advisor wants me to keep the questions that respondents answered as important as much as possible? $\endgroup$
    – Pete
    Commented Jan 27, 2012 at 16:00
  • $\begingroup$ I agree that item response theory is the next step to move through. @Pete, the answer to your question is in the second-to-last paragraph in the answer above - you keep the "easiest" items as these are the ones with the highest scores. "Least difficult" = highest mean scores. $\endgroup$
    – Michelle
    Commented Jan 27, 2012 at 18:39
  • $\begingroup$ @Pete as Michelle noted, the items with the highest mean scores will (at least in rasch and one parameter models) have the lowest estimated difficulty. If you want a good measure though, you'll need to keep some of the difficult ones too. $\endgroup$
    – richiemorrisroe
    Commented Jan 28, 2012 at 8:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.