8
$\begingroup$

Given the following experiment, what is the correct statistical method to answer the question below:

A participant is shown pictures consecutively and is required to respond whether she saw an object or a face after each picture. In each trial (picture presentation) the presented picture (either 1 of 210 individual faces or 1 of 210 individual objects) is superimposed with a certain amount of random noise (between 5% and 98%). The presented picture in each trial is rather small, so each trial has also a background. The background can either be black, a large object or a large face. The individual pictures are matched, meaning each individual picture is presented 3 times in total, 1 time with a black background, 1 time with a large object as background, and 1 time with a large face as background. The amount of random noise superimposed on an individual picture is kept constant over the 3 different background conditions. The object in the large object background does not change and is not included in one of the 210 individual object pictures presented. Similarly, the face in the large face background does not change and is not included in one of the 210 individual face pictures presented. No noise is added to any of the backgrounds.

The question I'd like to answer is whether the perception of either faces, objects or both significantly differs between the 3 different background conditions. See question 5 down below for more details on the question I'd like to answer

So in the end, I have a data table, looking like this:

+-------------+-------------+-------------+-------------+-------------+-------------+
| Participant |  Category   |  Pic ID     | Noise level |  Background |  Response*  |
+-------------+-------------+-------------+-------------+-------------+-------------+
|  1          |  0          |  1          |   5%        |  1          |  0          |
|  1          |  0          |  1          |   5%        |  2          |  0          |
|  1          |  0          |  1          |   5%        |  3          |  0          |
|             |             |             |             |             |             |
|  1          |  0          |  2          |  24%        |  1          |  0          |
|  1          |  0          |  2          |  24%        |  2          |  1          |
|  1          |  0          |  2          |  24%        |  3          |  0          |
|             |             |             |             |             |             |
|  1          |  0          |  3          |  80%        |  1          |  1          |
|  1          |  0          |  3          |  80%        |  2          |  0          |
|  1          |  0          |  3          |  80%        |  3          |  1          |
|             |             |             |             |             |             |
|  ..         |  ..         |  ..         |  ..         |  ..         |  ..         |
+-------------+-------------+-------------+-------------+-------------+-------------+
|  1          |  1          |  211        |  12%        |  1          |  1          |
|  1          |  1          |  211        |  12%        |  2          |  1          |
|  1          |  1          |  211        |  12%        |  3          |  1          |
|             |             |             |             |             |             |
|  1          |  1          |  212        |  20%        |  1          |  1          |
|  1          |  1          |  212        |  20%        |  2          |  0          |
|  1          |  1          |  212        |  20%        |  3          |  1          |
|             |             |             |             |             |             |
|  1          |  1          |  213        |  75%        |  1          |  0          |
|  1          |  1          |  213        |  75%        |  2          |  0          |
|  1          |  1          |  213        |  75%        |  3          |  1          |
|             |             |             |             |             |             |
|  ..         |  ..         |  ..         |  ..         |  ..         |  ..         |
+-------------+-------------+-------------+-------------+-------------+-------------+

where Category is face (0) or object (1) and Response is also face (0) or object (1). The participant's response is the dependent variable. Dichotomous with an underlying continuum. Since every participant is measured in all 3 background conditions, it is a dependent design. Since, for one individual picture, I keep the noise constant over the 3 background conditions, it is somehow paired or matched.

First I thought about calculating the biserial correlations and comparing them based on the t-statistic but then I saw logistic regression which seemed to fit my data structure better. But I still feel that the matched samples and dependent design should be incorporated in the analysis somehow. So when I searched for that, Conditional Logistic Regression popped up.

The problem is, in Conditional Logistic Regression, the matching is done on the dependent variable. They usually match a 1 in the dependent variable with one or more 0 samples. I didn't match on the dependent variable but on the independent variables (same pictures with same noise level in each background condition). So I don't think I can use Conditional Logistic Regression for this data but I couldn't find anything else that fits.

Could someone with more experience in statistics explain to me what the correct way is to answer the above question whether the perception of either faces, objects or both significantly differs between the 3 different background conditions.

Thank you for your help.

[Experimental procedure]

The experiment has 1260 trials in total. Made up of 210 individual faces and 210 individual objects presented 3 times each (one time with each of the 3 different backgrounds). Trial order is randomized with the constraint that in the first, the second and the last block of 420 trials, each background is presented exactly 140 times and each individual object and each individual face is presented exactly once. Most but not all of the different individual faces and objects have a different amount of noise added to them, but the noise for an individual face or object is kept contant over the 3 different background conditions it is presented.

[Questions & Answers]

1. How many participants? There are 5 participants in total.

2. Are there any bounds on noise? The noise is discretized in 0.5% steps and in the range [5%, 98%]. The noise is randomly drawn from a noise vector (without replacement) and assigned to a picture. This vector includes a noise distribution (210 entries for each category) that does not include every possible value between 5% and 98% in 0.5% steps but skips some of those values and includes some other values up to 3 times (answer to question 3). This ensures that each participant experiences the same noise levels (though not likely for the same pictures since the noise levels are randomly assigned to the individual pictures at the beginning of the experiment) and that there is a good coverage over the whole range but the focus is on noise levels near the threshold at which (for our setup) the participants can recognize the picture in about 50% of the time. This threshold was found in a preliminary study with other participants using the same pictures presented on a black background. Thus the black background is the default background in this experiment.

3. Is it possible that two or more pictures could be presented with the same level of noise? Yes, this will happen several times and include up to 3 individual pictures for the same level of noise, but not more than 3.

4. Can you confirm you are not interested in the association of noise with the response? This question is hard to answer for me. It is to be expected that the effect of different backgrounds is (if at all present) most prominent if the pictures are harder to see i.e. there is more noise present. So I want to consider the noise in the analysis, but I don't necessarily require the analysis to tell me anything about the association of the noise with the response. I'm only interested in detecting any kind of difference between the background conditions with as much power as possible. At first I wanted to fit 2 psychometric curves for each of the 3 different background conditions (probability to respond with the respective category vs noise level) and then compare the shifts of the psychometric fits to check for differences in the background conditions. However, a bootstrapping analysis revealed that the variance of the fitting procedure is too large to be able to detect shifts in the range I am expecting them to be. So I assume information about the association of noise with the response might decrease the power of other kinds of analyses as well. If this is the case, I don't need it.

5. What do you mean by 'perception' and 'both'. What do you actually want to know? By 'perception of [category]' I don't mean percent correct but '[category] responses'. The assumption I have (and I'd like to test that) is that a face-background would influence a participant to respond with face BUT an object-background would NOT influence a participant to respond with object (this assumption probably doesn't make any sense to you as a reader but that's what I need to test). What I mean with 'both' is that should it be the case that a face-background influences a participant to respond with face AND an object-background influences a participant to respond with object, my assumption that only the face-background has an effect on perception would be false. The different noise levels were included because the chances to influence perception towards one of the categories should be greater when the pictures are harder to see/recognize. So if there is a background dependent effect on perception for any of the categories it is unlikely to show in the e.g. 5% - 20% noise range but rather in the higher noise range.

Please let me know if you need further information.

$\endgroup$
  • 2
    $\begingroup$ You said "every participant is measured in all 3 background conditions" - so how many participants are there ? $\endgroup$ – Robert Long Aug 5 '16 at 13:05
  • 2
    $\begingroup$ Is that going to be the total number or are you adding more ? Do you need to analyse the data with just the current 5 ? $\endgroup$ – Robert Long Aug 5 '16 at 13:12
  • $\begingroup$ If possible I'd like to analyze it with the current 5. Due to certain circumstances I can only measure about 6 people per year. So while in theory I could measure more, I would like to avoid it since it would cost a lot of time. $\endgroup$ – Simeon Aug 5 '16 at 13:16
  • 2
    $\begingroup$ Are there any bounds on noise (100% noise would not make sense, but what about 99% for example) ? Can you confirm you are not interested in the association of noise with the response ? Also, each picture is unique, but is it possible that two or more pictures could be presented with the same level of noise (since noise is randomly assigned) ? $\endgroup$ – Robert Long Aug 5 '16 at 13:48
  • $\begingroup$ Yes, there are bounds on the noise. But I don't have enough space in this comment, so I add the information to my post. Thank you for being so thorough. $\endgroup$ – Simeon Aug 5 '16 at 15:20
4
+100
$\begingroup$

First:

"Remember that all models are wrong; the practical question is how wrong do they have to be to not be useful."

Box, G. E. P.; Draper, N. R. (1987), Empirical Model-Building and Response Surfaces, John Wiley & Sons.

Thus there is no one and only correct way to model your data.

The research question is broadly inquiring about the association of background type on the perception of category, however there are 2 distinct questions:

  1. Is background=3(face) associated with response=0 (face) (research hypothesis: Yes)
  2. Is background=2(object) associated with response=1 (object) (research hypothesis: No)

Your setup is factorial. You are varying the levels of factors (Category, Noise and Background) and measuring the response for different combinations of these. Your particular interest is in the association of background (a 3-level factor) with the response (a binary variable), therefore a logistic regression analysis would seem to answer the questions.

  • The estimate for background=2 answers the question: what is the difference in the log-odds of responding with object when the background is an object compared to when the background is black. This answers research question 2. To be consistent with the research hypothesis, this estimate should be small and/or not statistically significant.
  • The estimate for background=3 answers the question: what is the difference in the log-odds of responding with object when the background is a face, compared to when the background is black. The negative of this estimate is therefore the difference in the log-odds of responding with face when the background is a face, compared to when the background is black. This answers research question 1. To be consistent with the research hypothesis, this estimate should be small and/or not statistically significant.

However, that is not the end of the story....

Obviously you have repeated measures on participants, and this needs to be controlled for, since the responses of one participant will be more like other responses of the same participant, than those of other participants (that is there is likely to be correlation of measurements within each participant). This can be controlled for by including random intercepts for Participant or by including Participant as a fixed effect. 5 is considered by many as the minimum number of levels for a factor to be used as a random effect and since you intend to add more participants to the study, this would be my recommendation. Either method controls for repeated measures so you could run both models and I will present both below.

You also have repeated measures on each picture, where each picture is measured 3 times. Thus there may also be correlation within each picture. Since you have 420 different pictures, it would not be a great idea to include picture as a fixed effect to control for this, so a random intercept is appropriate. So, my starting model would be a mixed effects model with random intercepts for Picture_ID and Participant, with fixed effects for Category,Background and Noise (with noise being coded as numeric). Participants are not nested within pictures, and pictures and not nested within participants so these are crossed random effects.

In R using the lme4 package, this would be specified as:

glmer(Response ~ Category + Background + Noise + (1|Participant) + (1|Picture_ID), data=dt, family=binomial(link=logit))

Due to the small number of participants, an alternative model is:

glmer(Response ~ Category + Background + Noise + Participant + (1|Picture_ID), data=dt, family=binomial(link=logit))

The analysis can be extended to allow for:

  • interactions between the fixed effects

  • non-linear association between the response and Noise (by including quadratic and possibly higher order terms for Noise)

  • the association of Noise to vary between participants and/or picture (by including random coefficients for Noise)

The above is based on contrasts of the desired background with the black background - that is face vs black and object vs black. If face vs object is required this can be handled by recoding the factor or specifying the reference level directly. If face vs not face or object vs not object is required then this can easily be accomplished by creating dummy variables.

$\endgroup$
  • $\begingroup$ Thank you very much for your answer. I'm new to R, so I'll have to look into the lme4 package to fully understand your examples, but what you describe sounds very much like what I'm looking for. $\endgroup$ – Simeon Aug 9 '16 at 12:35
0
$\begingroup$

I believe that conditional logistic regression will give you the desired results.

You correctly identify the need to use a repeated measures convention when analyzing this data. You have 5 respondents evaluated for the binary outcome of correct/incorrect face/object recognition over multiple conditions. The numerous responses from one person generate the need for a repeated measures approach.

If your intent is actually better stated as whether the respondent chooses face or object, you could use the same analytic approach, but note that you would be interpreting the respondents choice not correct/incorrect classification. For a third category of "both" you would need multinomial logistic regression. I will assume you are interested in correct/incorrect classification in what follows.

You state "Since every participant is measured in all 3 background conditions, it is a dependent design. Since, for one individual picture, I keep the noise constant over the 3 background conditions, it is somehow paired or matched." The conditions under evaluation, while limited in their value or quality, are not "conditioning" your analysis. The use of grey background, face picture with 45% noise is just one vector of covariates that present when a response is recorded. Grey background, object, 45% noise is another vector, while white, face, 10% noise is another. The regression will suggest to you whether background (dummy coded), noise or additional variables are associated with the correct response. The association between correct identification and change in any one value, holding all other values constant, is the interpretation of multivariable regression. Thus, you will obtain a sense of the association between background OR a one unit difference in noise OR whether a face/object was shown by using conditional logistic regression.

Your model in R would be something like:

install.packages("survival")
require("survival")
clogit(correct ~ background + noise + pic_type + strata(person), data)

A more complicated model for each specific face or object among pictures could be considered, but you will dilute your ability to detect the desired effect of background.

$\endgroup$
  • 2
    $\begingroup$ The question is not about whether the participants choose objects or faces correctly, it is about (quoting from the OP) "whether the perception of either faces, objects or both significantly differs between the 3 different background conditions" $\endgroup$ – Joe King Aug 6 '16 at 11:03
  • 1
    $\begingroup$ The analysis of "perception" can only be understood as one of two things: (1) choosing a face or an object or (2) choosing correctly what is presented. The event space only seems to containe these two possibilities. Thus, I don't see how "both" can be analyzed. The original poster should weigh in on which is desired. My answer reflects what I perceive to be some ambiguity around the outcome. I don't have a commenting ability yet, so I covered the ambiguity in my answer. $\endgroup$ – Todd D Aug 6 '16 at 16:51
  • $\begingroup$ I apologize for not being specific about what I meant with perception and for choosing a bad example for the different conditions. I have to walk a narrow path between not giving away any details of an unpublished study and (obviously) being close enough to the real thing that there wouldn't be any differences in the analysis. It seems I didn't do a good job and I apologize if this wasted anyone's time. I added one more block of information that will hopefully clarify all open questions. If anyone feels cheated by this change, let me know and I see what I can do. $\endgroup$ – Simeon Aug 8 '16 at 10:52
  • $\begingroup$ @Simeon I'm still not sure I understand your research problem completely. However, if you want to test both (1) whether background affects face perception and (2) whether background affects object perception, you could divide the data set by whether face or object was shown. You could then assess the role of background on object recognition for various backgrounds when shown an object in one analysis. The other analysis would assess background effects on face recognition when shown a face. Does this approach your intent? $\endgroup$ – Todd D Aug 8 '16 at 22:14
  • $\begingroup$ Thank you for your answer. Though I do not intend to split the data set because the effect to bias the participants response towards faces could also occur in trials presenting a noisy object on a face-background. $\endgroup$ – Simeon Aug 9 '16 at 12:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.