There are several aspects to the problem, some of which you addressed explicitly in your question, some of which you didn't.
What type of scale are ratings?
Following Stevens (1946), ratings are ordinal and thus not well suited to linear regression (or its mixed-effects extensions). However, as a rule of thumb, the regression methods that can work on ordinal scales (ordinal logistic, arguably some forms of quantile regression) are more "exotic" and thus less well supported by the various software packages. When you combine "exotic" regression with "mixed-effects", the support is even worse. If you're willing to go the Bayesian route, then the wonderful
MCMCglmm package does support mixed-effects ordinal logistic regression and will even output something like to p-values based on its MCMC estimates.
That said, many people in your position would just treat ratings as an interval scale and thus suitable for linear regression. It is up to you to determine which interpretation is appropriate and to make a convincing case for your interpretation to potential reviewers and other consumers of your scientific output. In the following, I will more or less assume that you went the linear regression route, although much of what I say is still applicable for ordinal logistic and other regression techniques.
lme4: Where are my p-values?
This is an R FAQ. A short summary would be
degrees of freedom aren't as trivial as you were taught when dealing with models where the different parts can partially or fully overlap (i.e. hierarchical/multilevel models with partially or fully crossed levels).
In traditional repeated-measures ANOVA (which is actually a special case of the linear mixed-effects model with a particular symmetry assumption (sphericity)) and other simpler models, it is possible to calculate the degrees of freedom analytically (i.e. there is a formula for them). However, for more general designs, this is not possible and we must approximate the degrees of freedom. There are several different ways to do this (Satterthwaite, Kenward-Roger, etc.) but all are approximations and can at times be quite computationally intensive. Without degrees of freedom, it is not possible to convert t-values to p-values, nor F-values to p-values for the ANOVA case. (After all, the denominator degrees of freedom for the F-test are the square of the degrees of freedom for the t-test.) Given all the issues with p-values and their interpretation (finding the relevant literature is left as an exercise for the reader, but the psychological science literature has taken to self-flagellation trying to come to terms with it, so it's a good place to start), why would you bother with a computationally expensive approximation?
If you absolutely must have p-values, there are several ways to compute them beyond these approximations, including likelihood-ratio tests, although those may be less interesting for your particular model.
You could also recall that the t-distribution approaches the normal distribution as its degrees of freedom approach infinity. This means that for sufficiently large degrees of freedom (which you often have for (psycho)linguistic data), you can treat t-values as z-values. The traditional 5% significance level occurs for (two-sided, which you have/want) at $z = 1.96$. To be a little bit more conservative, you can just say that you reach significance at $t = 2$ (cf. Baayen, Davidson and Bates, 2008, amongst others for a similar train of thought).
But again, I would discourage you from getting too attached to statistical significance. Think about the size of differences and how well we can see that size amongst the noise/error, and not about whether there is a difference!
How do I interpret my regression coefficients?
There are lots of good explanations of this online and in various textbooks, so I won't go into much detail here, but I think your fundamental problem is in your choice of contrasts. A linguistically motivated discussion of contrast coding (with links to other good discussions) can be found here, but you might want to look at a textbook discussion of contrast coding first (e.g. in one of Andy Field's Discovering Statistics books, which rely on slightly less mathematical intuition than other explanations I've seen). If you're looking at coefficient directly, then you should probably be using sum coding, and I would recommend using
contr.Sum from the
car package as it gives better names to the coefficients than the built-in
contr.sum (note the capitalization).
Finally, it looks like you may actually be more interested in certain pairwise contrasts than anything else. In that case, you may want to consider the
lsmeans package, as this is the most convenient way to do post-hoc pairwise comparison. For your case, this has advantages beyond being easier to interpret than regression contrasts/coefficients:
- It's not sensitive to the choice of contrast coding (i.e. treatment vs. sum coding, obviously if you structure your predictors completely differently that will make a difference)
- It will give you p-values for the pairwise estimates along with a multiple comparisons correction for the doing so many tests.
On the structure of you model
Your model is a good start, but based on my understanding of your experimental question, it still needs some improvement:
- You have no interaction term: Surely you care about how word order and phrase type interact with each other?
- Depending on what type of contrasts you're interested in, it may make sense to have several categorical predictors for word-order and phrase-type, e.g. three predictors
third.arg (with possibilities
IO for each of them) or maybe
indirect (with possibilities
third) for each of them. Depending on the exact contrasts you set up, some things will be easier to read from the coefficients and some things will be harder. You may also get a warning about dropping columns due to a rank-deficient matrix -- this is not actually a real problem, but just means the model doesn't compute coefficients for combinations which don't occur (e.g. both
direct simultaneously have the value
- You need a random-effects term for
Item: this isn't like classic ANOVA-by-subjects vs. ANOVA-by-items ($F_1$, $F_2$, cf. Clark 1973), you can and should model both sources of variation (subjects/participants and items) simultaneously, otherwise you may come to the wrong conclusions (cf. Judd, Westfall and Kenny 2012).
- You may want a random-effects structure that includes slopes and intercepts, but there is a lot of debate on this. See e.g. Barr, Levy and Sheepers 2013; Barr 2013; Bates, Kliegel, Vasishth and Baayen 2015/arXiv as well as the GLMM FAQ linked above.
One last thing to get started ...
A final bit of advice: it looks like you're completely new to mixed models, maybe even completely new to statistics. That's fine, we all had to start somewhere. But you might want to do some more background reading. :) Although slightly out-of-date on the software front, there is a 2008 special issue of the Journal of Memory and Language on mixed models. Other popular (book) references for this are Gelman & Hill (2006), Fox (2016 -- newest edition has much more on mixed models, but there are also several good older editions) and Pinheiro & Bates (2000). Finally, Bodo Winter has an online tutorial that many of my students have also found quite good.