Conceptual diffiulty in understanding multiclass classification

Question

Given a high dimensional feature vector x in $R^D$, I want to map it to an L bit vector, $L << D$,

z = h(x) in $\{0,1\}^L $

using a function h while preserving the neighbors of x in the binary space.

My problem is closely related to the one asked here Activation Functions

and based on https://mediatum.ub.tum.de/doc/680212/680212.pdf

The activation function f() is used as a unique code generator. What learning algorithm should I use ? f() is of the form : https://en.wikipedia.org/wiki/Piecewise_linear_function

Please correct me where wrong. If I want to apply this piece-wise linear (PWL) function as a classfier, then does that mean that the function forms an activation function in neural network or can there be other mechanisms where this can be applied? Any small simple example to show what will be the input and what will be the output. What is the role of neural network in this problem? I am having difficulty in understanding what I need to do so that the PWL function can be used as the classifier function and the role of neural networks. As an example :

Let:

$x_1 = 5.1,3.5,4.9,-1.40,-0.2,3.2$ ; $y_1 = 1,1,1,0$

The feature dimension in this case is 6 and the class labels is a vector of L = 4 bits. Each example is assigned a unique binary string of length L. We can Refer to these strings as codewords. Then L binary functions are learned, one for each bit position in these binary strings. During training, the desired outputs of these L binary functions are specified by the codeword. With artificial neural net w orks, these L functions can be implemented by the L output units of a single net work. Now, I want to use the PWL function as the activation function. How can I do this? Any example to help clear the concepts will be very useful.

Answer 1

Classification :

The introduction of the paper you quoted contains your answer I believe :

This problem is split in many two-class classification problems

To classify something into L classes {1,2,..,L}, you can cut your problem in L classification sub-problems :

• {1}/{2,3,..,L}
• {2}/{1,3,..,L}
• ...
• {L}/{1,2,...,L-1}

Each of these sub problems can now be easily solved by a neural networks giving you L binary coordinates. If your neural-networks are linear (perceptron) then you can also visualise the results with a PWL (as in figure 3 of the paper you quoted).

For such classification task, the current approach is to write a single network using a sofmax output layer.

Dimensionality reduction :

using a function h while preserving the neighbors of x in the binary space.

If you want to preserve the neighbours then that looks like dimensionality reduction rather than classification.

An auto-encoder with an output layer of size L made of strong activation functions would take your data and express them in {0,1}^L (each bit being the binarised output of it's corresponding activation function).

(If you don't especially need a binary output then other methods such as PCA and t-SNE might bring better results (preserve the neighbours) depending on the exact nature of your problem.)