regression model performs less accurate for some tasks

Question

I have a regression problem and used different suitable models for predicting 4 dependent variables. Through many experiments, I realized that it's easy for many models to learn two tasks with high precision while the other two are less accurate.

That can be due to the less predictive nature of those two tasks in my datasets. Also in the context of my problem, it's not surprising each task has a different form of functionality. I mean, for each task $i$, the estimator $f_i(x) \approx y_i$ might have a different functional form. But I was wondering what other reasons might cause a model to perform on one task and not as good on another task.

A little info about what I've done:

• standardizing the labels so that all are centered and have unit variance.
• deployed linear regression, lasso, and neural nets as a model.
• I also tried some feature selection as my problem is a very high-dimensional one.

UPDATE: After the same answer, I recalled that some of my regressors, despite being real numbers, are almost categorical, in the sense that their range only covers a few numbers (say integer numbers from 0 to 5) and one can in fact categorize the data by the value of this regressor. Now I'm wondering if that as well can explain part of the lower precision.

Answer 1

There are many possible reasons. Here's a start on a list, I'm sure others will add more:

1. The model is misspecified
2. The variables are measured with more error in one data set than another
3. The relationship, while correctly specified, is less strong with some variables
4. Some variables are measured on a coarser scale than others.
5. Some variables have been categorized while others have not.