2
$\begingroup$

We know that the neural network will perform like a linear regression if there is only one hidden unit. So, the NN method should perform at least as well as a linear regression method. I have built a tidymodel model using the following line of code:

Data_nnet_mod <-  mlp(hidden_units = tune(),  penalty = tune(),  epochs = tune()) %>% 
  set_engine("nnet") %>% 
  set_mode("regression")

and have tuned it using

Data_nnet_fit <- 
  Data_nnet_wflow %>%  
  tune_grid(val_set,
            grid = 25,
            control = control_grid(save_pred = TRUE),
            metrics = metric_set(rmse))

It turns out that the linear regression output has a smaller RMSE than the NN method.

I wonder why the best RMSE that the NN method produces is larger than that of the regression method. Theoretically speaking, should the NN method not be at least as well as the regression method?

Improve this question
$\endgroup$
2
  • $\begingroup$ I recommend you ask discussion-type questions like this in the ML category on RStudio Community. Also, I recommend creating a reproducible example with some data that demonstrates what you are seeing. $\endgroup$
    – Julia Silge
    Commented Jul 9, 2022 at 23:44
  • $\begingroup$ A neural network without a hidden layer and an identity activation function is in fact exactly a linear regression model. $\endgroup$
    – shem
    Commented Jul 16, 2022 at 18:09

1 Answer 1

Reset to default
3
$\begingroup$

A neural network with one hidden unit and linear activation is linear regression. There may be differences though, for example, neural networks are usually trained with variants of gradient descent, while linear regression with ordinary least squares, so you have no guarantees that they end up with the same results. There also may be implementation details that differ. If you use regularization, other activation, loss, etc those would be different models so again you have no guarantees of finding the same solution, or an equally good one. Unless both models are exactly the same, you don't really have guarantees of same performance. Because all of the above reasons, linear regression may outperform neural networks for regression problems, or logistic regression can for classification.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Assuming standard squared error loss, this is a convex optimisation problem hence is it not the case that the gradient descent solution will be guaranteed to be equivalent to the OLS solution? $\endgroup$
    – shem
    Commented Jul 16, 2022 at 18:49
  • $\begingroup$ @shem Regularisation changes the objective function. $\endgroup$
    – usεr11852
    Commented Jul 16, 2022 at 18:59
  • $\begingroup$ @shem yes, but the convergence would depend on batch size, learning rate, and potentially other parameters of the optimizer, also you need to train long enough but you never know how long is that. $\endgroup$
    – Tim
    Commented Jul 16, 2022 at 18:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.