# How to apply Softmax as Activation function in multi-layer Perceptron in scikit-learn? [on hold]

I need to apply the Softmax activation function to the multi-layer Perceptron in scikit. The scikit documantation on the topic of Neural network models (supervised) says "MLPClassifier supports multi-class classification by applying Softmax as the output function." The question is how to apply the function?

In the code snip below, when I add the Softmax under the activation parameter it does not accepts.

MLPClassifier(activation='Softmax', alpha=1e-05, batch_size='auto',
beta_1=0.9, beta_2=0.999, early_stopping=False,
epsilon=1e-08, hidden_layer_sizes=(15,), learning_rate='constant',
learning_rate_init=0.001, max_iter=200, momentum=0.9,
nesterovs_momentum=True, power_t=0.5, random_state=1, shuffle=True,
solver='lbfgs', tol=0.0001, validation_fraction=0.1, verbose=False,
warm_start=False)


The error code is:

ValueError: The activation 'Softmax' is not supported. Supported activations are ('identity', 'logistic', 'tanh', 'relu').

Is there a way to apply the Softmax activation function for multi-class classification in scikit-learn?

## put on hold as off-topic by mkt, Peter Flom♦7 hours ago

This question appears to be off-topic. The users who voted to close gave this specific reason:

• "This question appears to be off-topic because EITHER it is not about statistics, machine learning, data analysis, data mining, or data visualization, OR it focuses on programming, debugging, or performing routine operations within a statistical computing platform. If the latter, you could try the support links we maintain." – mkt, Peter Flom
If this question can be reworded to fit the rules in the help center, please edit the question.

I suposse that the Softmax function is applied when you request a probability prediction by calling the method mlp.predict_proba(X).

To support my supposition I have developed this small experiment:

from sklearn.neural_network import MLPClassifier
import numpy as np

mlp = MLPClassifier()
mlp.fit(X, Y)

print mlp.predict([3.1,  2.5,  8.4,  2.2])
print mlp.predict_proba([3.1,  2.5,  8.4,  2.2])
print "sum: %f"%np.sum(mlp.predict_proba([3.1,  2.5,  8.4,  2.2]))


Notice that no matter what values are plugged into predict_proba(), the output probability vector allways sums up to 1. This can only be achieved by the Softmax activation function (Using an activation other that Softmax there is no guaranty that the sum of the activations in the final layer will be exactly one, specially for an unseen sample).

If my guess is right, looking at the documentation I can not find any method to get the output of the network before Softmax... Maybe because this class is intended solely for classification (not regression or other fancy setups).

The MLPClassifier can be used for "multiclass classification", "binary classification" and "multilabel classification". So the output layer is decided based on type of Y :

1. Multiclass: The outmost layer is the softmax layer

2. Multilabel or Binary-class: The outmost layer is the logistic/sigmoid.

3. Regression: The outmost layer is identity

Part of code from sklearn used in MLPClassifier which confirms it:

        # Output for regression
if not is_classifier(self):
self.out_activation_ = 'identity'
# Output for multi class
elif self._label_binarizer.y_type_ == 'multiclass':
self.out_activation_ = 'softmax'
# Output for binary class and multi-label
else:
self.out_activation_ = 'logistic'

1. Multiclass classification: For a Feature X, there can only be one class. eg Sentiment Analysis Given a Text(X), is the output(Y) is positive, neutral or negative. Binary is a case of Multiclass where there are only 2 possible outputs.
2. Multilabel classification: For a Feature X, there can be multiple classes.

Can't agree with the answer from Daniel Lopez. In my case answer predict_proba() doesn't return softmax results.

The answer from TrideepRath can easily solve this issue. To apply softmax define out_activation_:

your_model.out_activation_ = 'softmax'

New contributor
MykolaSharhan is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.