"you shouldn't be doing statistical testing like this: if p value of wiggly bit < 0.05 assume wiggly otherwise assume linear." - ok that's fair, but if p value of wiggly bit > 0.05 and linear <0.05 and it looks like mostly a flat line (or an uncurved plane for tensors), it's fairly safe to assume it's a linear relationship, no? and re. "you'll bias your results to keeping the wiggly bit in extreme data sets and drop it otherwise": I think this was a misunderstanding - I never intended to drop anything. I just want to be able to say "more wiggly, more linear, nothing at all"

It appears that in mgcv, in order to test/visualize such two-way interaction between continuous and categorical variables, I have to make this an extra term in the model (as you and I have done above). Put differently: if this were a linear model, I could inspect the two-way interaction of Sepal.Length and Sepal.Width in the model summary without having to specify it explicitly (I could just do Sepal.Length * Sepal.Width * Species), and I could pull out the marginal effects for "Sepal.Length * Sepal.Width" across all species. (2/2)

regarding question ii): "I'm not sure what the problem is because plot(mod1) will plot all the smooths allowing you to see the average effect of the smooth for sepal length separately from the species-specific smooth of that same covariate. " - Yes, but for that average effect this'll only work if I include "ti(Sepal.Length, Sepal.Width)" without 'by="species"'. (1/2)

regarding question i): "Making statistical decisions in the way you plan using p values isn't advisable ... " - I wonder which statistical decisions you refer to. I simply wish to first test for non-linear within-species interactive effects (tensor product), and if there are none, for non-linear across-species interactive effects (tensor product), and if there are none, linear within-species interactive effects (linear interaction), and so on for the univariate terms. This should be fine if accompanied by adequate visualization and effect sizes in a table, no?

following up on the comment: "testing for wiggliness beyond a linear effect" is what I want, because otherwise I'd simply use linear models - so I guess I'll update the models accordingly. can I remove the linear function from the basis of a tensor smooth in the same fashion as for a spline (e.g. "ti(x,y, bs="tp", m=c(2,0)"?

still processing your answer, but already have a first question: I thought to understand linear vs non-linear patterns I would have to include linear terms in additions to the smooths? asked differently, when does it makes sense to include a variable as a linear term?

@GavinSimpson thanks for your comments (and for pointing me to gratia - good stuff!), but I am still confused as to what include in the first place - i.e., do I include the across-group term and then exclude all "by" terms? to move away from the comments I have posted it as a new question: stats.stackexchange.com/questions/615924

@GavinSimpson thanks - but if I passed on "group" through the "by" argument then I guess I can't exclude that - or would I have to include terms with and without "by"? E.g.: ti(v1, v2) + ti(v1, v2, by="group") and then exclude the latter when predicting?

I'd be interested in doing this with my model that has two continuous variables and one categorical (separate smooths and 3-way interaction, both implemented with the "by" argument), but I currently can't imagine how I might slip an additional dummy variable in there when the "by" argument is already used - does this call for a new question or can you elaborate here?

re. GCV: I have used it in the past, and simply constrained the knots to lowest number possible - doesn't that partially take care of overly wiggly smooths?

yupp that "*" was a problem - thanks for fixing. also, you are missing a ")"

ad 1) I standardized the data because they are on vastly different scales (how does mgcv::gam deal with that?), and because I use the same standardized data in analyses where I have to standardize. ad 2) you mean always or just in this context?