Favoring one regression line over another

Question

Skip to the end to see the question, this is just background information.

In the industry I work in, it is often desirable to find any line that best models the data as a sort of worst-case scenario. This leads to some conflict in my analysis from one project to another as:

1. Project 1 has engineering defined slope, an inverse-square root slope, so we just find the intercept that minimizes RSS.
2. Project 2 has historically been a mixture of pure linear trends and logarithmic trends. - In working on Project 2, it has also been suggested to test an inverse-square root as well, even though this project does not follow the same engineering as Project 1.

The trends above were found in R using the following coding format:

Linear: lm(Y~X)

Logarithmic: lm(Y~log(X))

Inverse Square Root: lm(Y~sqrt(X)^-1) 
                     OR 
                     lm(Y~sqrt(X^-1))
                     OR 
                     lm(Y~1/sqrt(X))

Note: These are all the same, obviously

Question

My main question is focused on Project 2. Outside of engineering specified instructions like in the first, why should I favor/choose one over the other? Is it possible to lose information if a trend that was previously found with a linear trend is found to have an inverse square root that has a better AIC/BIC score than the linear?

Also, how do other factors help choose the line that best fits? Namely, one p-value is significantly stronger than another, or our ability to believe the trend is more true than the other, aka Power is higher for one than another.

I can not share example data for proprietary related reasons. I might try to generate a random example when I have a little more time.

Answer 1

When I'm looking at my models, I tend to take an ensemble approach to decide which is more trustworthy. A better AIC/BIC matters, but make sure to also cross validate. Take 10% of your data and set it aside. Train your model with the remaining 90% and predict that 10% you set aside earlier. record the RSS and then re-do the process with another random 10/90 split. After a number of iterations, find the mean and variance of your RSS for each model.

I would then make my decision based on a balance of how close the means are to 0 and how tight the variances are. closer means to 0 means better central tendency, but small variances means better precision. Depending on what your application is, they will hold different weights.

Answer 2

My statistical take on this is that in order to fit a regression model you have to make assumptions about the errors, for linear regression these are zero mean, constant variance and independent. You can if you must add in an assumption that the errors are Normally distributed (with zero mean and constant variance). It used to be semi-common to transform the Y variable to obtain a model where these assumptions were plausible. So one thing I would be doing if I were you is checking the residual diagnostics, especially the constant variance assumption (residuals on the vertical axis and fitted values (y hat) on the x axis).