1
$\begingroup$

My model has 6 input features populated with continuous values (MinMax from -1 to 1) and 3 output.

The aim is to mutually identify one of three classes (multiclass single label).

I did tests for about a month trying out different configurations of the model using Mean Square Error as a cost function, getting some (not so exciting) results.

Then I read that for the Logistic Regression the MSE is absolutely wrong so I tried to use the (Softmax) Cross Entropy.

The problem is that using this function regardless of the model structure (layer number / number of neurons / activation functions) learning does not seem to work or at least the result is worse: the loss increases after a few epochs and accuracy is very low. What did i do wrong?

Old model (best configuration):

  • samples: 5100
  • batch size: 100
  • learning rate: 0.0001
  • loss function: MeanSquaredError
  • eval function: MeanAbsoluteError
  • 3 input
  • 1 hidden layer with 6 neurons, ativation: Tanh
  • 3 output, activation: linear
  • n. of epochs before loss increase: 8640
  • result: train loss=0,0040; eval loss=0,369

New model (best configuration):

  • samples: 5100
  • batch size: 100
  • learning rate: 0.01
  • loss function: CrossEntropyWithSoftmax
  • eval function: ClassificationError
  • 3 input
  • 1 hidden layer with 1 neurons, ativation: Tanh
  • 3 output, activation: linear
  • n. of epochs before loss increase: 2640
  • result: train loss=0,0049; eval loss=0,457
Improve this question
$\endgroup$
3
  • 6
    $\begingroup$ You can't compare loss numbers between models with different loss functions. It's apples and oranges. You need an objective metric, such as accuracy or AUC-ROC. $\endgroup$
    – olooney
    Apr 3, 2019 at 13:29
  • 2
    $\begingroup$ Work better as judged using what? $\endgroup$
    – Tim ♦
    Apr 3, 2019 at 14:16
  • $\begingroup$ thank you, I learned a new thing, I made a wrong assumption $\endgroup$
    – Rick
    Apr 3, 2019 at 15:56

1 Answer 1

Reset to default
1
$\begingroup$

The error in reasoning is that the loss values are not directly comparable; these losses measure different things, so it's not possible to compare them.

Because these measure different things, model 1 could have lower MSE than model 2 but higher cross-entropy; or model 1 could have lower MSE and lower cross-entropy. Instead, just measure the thing you care about for both models: get the model predictions and compute the loss/figure of merit in the usual way.

Instead, you could measure both models according to MSE or cross-entropy, or measure both models according to a third quantity (e.g. ROC AUC).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.