I am doing research on predicting failing time of a component of a machine. Response is failing time of the component of a machine, and the input is location information (consists of integers).

I fit the neural network model with one linear layer with 100 nodes. Also, I fit the multi linear regression and performed the prediction for failing time.

MSE based on the muliple linear regression is 0.43 and MSE based on the neural network is 0.64-0.67. However, neural network has more parameters and capture more complicated information in data, it should work better than linear regression, isn't it?

the full sample size is 19319, training size is 14489, and testing size is 4830.

I standardized input before inputing it in neural network. The detail about neural network model is as below.

class Net(torch.nn.Module):
  def __init__(self, n_feature, size_hidden, n_output):
    super(Net, self).__init__()
    self.hidden = torch.nn.Linear(cols, size_hidden) 
    self.predict = torch.nn.Linear(size_hidden, n_output)  

def forward(self, x):
    output = self.predict(self.hidden(x))
    return output

net = Net(cols, 100, n_output).cuda()

optimizer = torch.optim.RAdam(net.parameters(), lr=0.001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0.9, foreach=None)
optimizer = torch.optim.Adam(net.parameters(), lr=0.001)

criterion = torch.nn.MSELoss(size_average=True) 

batch_size = 200 or 500 or 1000 or 3000 or 5000

I tried Adam, RAdam, SGD etc all sorts of optimizer with learning rate 0.001 and 0.01. if learning rate 0.01, it does not converge.

Additionally, for aLl the different batch size, the performances were similar.

  • 2
    $\begingroup$ since you haven't used any (non-linear) activation function your "neural network" is pretty much a linear model, so you shouldn't expect it to perform better than standard linear regression would here. $\endgroup$ Commented Apr 18 at 5:58
  • $\begingroup$ yes but it has more parameters than MLR, isn't it? I thought that somehow it should be at least a little better than MLR because it give the model more flexibility...no? $\endgroup$
    – wildcat
    Commented Apr 18 at 17:53

1 Answer 1


the hidden layer needs to have nonlinearities (eg relu) otherwise it is no better than multiple linear regression (but converges worse). 1 linear layer = matrix multiplication by weight matrix $W_1 x$. 2 hidden layers $W_2 (W_1 x) = W x$ where $W x = W_2 W_1$.

So a single (different) set of weights is equivalent to multiple layers of linear hidden units

  • $\begingroup$ yes but it has more parameters than MLR, isn't it? I thought that it should be at least a little better than MLR because more parameters give the model more flexibility $\endgroup$
    – wildcat
    Commented Apr 18 at 17:54
  • $\begingroup$ Even I add any type of activation function, still NN performance is same. what this can mean? $\endgroup$
    – wildcat
    Commented Apr 18 at 18:05
  • $\begingroup$ Even after adding a few more hidden and activation layer, performance does not improve... $\endgroup$
    – wildcat
    Commented Apr 18 at 18:44
  • $\begingroup$ --> I changed my optimizer, then with two hidden layer and activation function, it perform better. But still, even without activation function, shouldn't NN perform better than MLR? Is it because of optimization problem? $\endgroup$
    – wildcat
    Commented Apr 18 at 18:50
  • $\begingroup$ take simple example y=vwx (input x, output y, parameters v and w). adding the extra parameter v does not change the range of functions you can fit, any (v,w) pair can be replaced by a single value (the product) $\endgroup$
    – seanv507
    Commented Apr 19 at 9:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.