Nesting either one of the variables inside the other implies there is a hirerchy between the two. Consider the following example, a factory receive shipments each week, each shipment consists of several batches. We want to measure some quality of the materials receive and we sample several shipments, several batches in each and take some boxes from each batch to inspection. Obviously, batch is nested inside shipment and formula would look something like:
Quality ~ (1|shipment/batch)
You can also note that if batch names are unique between shipments the following will be equivalent in lme4:
Quality ~ (1|shipment) + (1|batch)
Now going back to your example, a hirerchy would be on something like each year you samples different sites, then obviously you would want something like 1|year/site or the other case where each site was sampled in different years (e.g. site1 was sampled 2001-2004, site2 was sampled 2005-2007, etc). In your case (I'm just guessing) you sampled more or less the sites each year which probably makes them crossed effects and crossed effects are supported only in lmer which sadly does not support AR(1).
But, since year as you mentioned is a temporal effect, usually it will not be a random effect, it makes more sense to do one of two things: assume there's a progression in measurements and then year is probably a fixed effect and if you think the effect of year is changing between different sites you could also add it as a random covariate. Formulas in lmer would look like:
mod2 <- lmer(resp~Year+(1|Site),data=dat)
mod2 <- lmer(resp~Year+(1+Year|Site),data=dat)
mod2 <- lmer(resp~Year+(1|Site)+(0+Year|Site),data=dat)
(If you need help understanding the difference let me know, also all these have an equivalent with AR(1) in nlme..)
Alternatively, if you feel you're not expecting any progression with passing years I think best option if just to neglect year as a variable all together. i.e.
mod2 <- lmer(resp~(1|Site),data=dat)
This is very common in many cases when you think time has not effect..
