I have two processes that output n-square matrices. I am interest in modelling these matrix norms as a dependent variable, according to the process used, and the matrix size n as predictors.

I generate those matrices relying on some random process that probably implies its own mathematical rules, but the aim here is more about modelling what happens from the obtained data.

Here are a small version of my larger dataset, in .txt, that should be importable through read.table(): https://file.io/FAmyVa

So my aim is to have a model following something like norm ~ process * n. By design I also have "blocks" (5) in each of which I apply both processes (P1 and P2) in several "trials" (5), each outputting a matrix norm (25 norms for P1 and 25 for P2, for a given n). In other word, I feel I can model this as a nested random effect structure, such as:

norm ~ process * n + (1|Block/trial)

From data visualization norm = f(n) I feel that you have a similarly shaped increase, with one increasing more rapidly.

ggplot(data, aes(n,norm,colour=process))+geom_jitter(alpha=0.2)+

And the difference seems non significant with low n, but the two curves part from each other as n increases, and that's something I'm interested in. So I want a type of model that translates it into estimates and confidence values.

And that's the thing: it doesn't straight look like anything I tried to model before, especially with the slight S-shaped but unbounded (?) trend.

I tried fitting a LMM with glmmTMB or even MCMCglmm, with or without scaling of the predictors and/or response variable, but I end up with a very visible trend in the residuals against n, because of the S-shape of course.

I'm getting confused about what should be my approach:

  • is there a family that will "fit" for this job? What would be the data/design justification (in terms of type of response/heteroscedasticity or else)?

  • will this work better if I apply a reasonable transformation, e.g. dividing the norm in regard to n or a function of it? scaling in relation to n, the blocks? Another transformation that might "normalize" the relation between predictors and response?

  • is this a case where it'd be best to go into a non-linear regression or a GAM(M)? I never fitted anything similar, and from my reading I feel the smoothing function could help here. But that it might be an overkill for such a case...


put on hold as unclear what you're asking by mdewey, Robert Long, Jeremy Miles, Siong Thye Goh, Yves 8 hours ago

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.