I have a single percepton with sigmoid activation $y = f[\sum_{i=0}^{n}(x_i \cdot w_i)]$, where $f$ is a sigmoidal activation function. Is there a way to drop some of the inputs (or set their weigths to zero), readjust remaining weights without sacrificing model's performance? L2/L1 norm regularization helps with minimizing some weights, but doesn't make them completely zero.

Edit: Goal is not regularization, but to drop the feature completely and optimize the model for the production use.

  • 3
    $\begingroup$ "Regularization helps with minimizing some weights, but doesn't make them completely zero." - $\ell_1$-regularization does exactly that. $\endgroup$ – Firebug Feb 21 '17 at 13:33

It almost sounds like you're (re)inventing DropOut or DropConnect.

These are usually applied to units embedded in a deeper network, but the idea is pretty much the same. Some units' outputs (or input connections) are randomly set to zero during each training presentation. The weights are then adjusted at test time to account for the fact that fewer units/connections were present during testing. This process does something akin to model averaging or a very extreme form of bagging, and is thought to prevent "co-adaptation" between units within a layer.

If you had highly correlated input features, I suppose applying it to a single perceptron could work as well. This paper by Baldi and Sawdoski has some pointers into the DropOut literature and, as a special case, looks at applying it to a single linear unit in section 2.1.

This approach is a little different from regularization. Regularization adds a penalty term which is some function of the model's weights. For example, $\ell_2$ regularization uses the sum of the squared weights, which encourages the model to reduce very large weights more. $\ell_1$ uses their summed absolute values, which applies an equal penalty to all weights (i.e., moving a weight from 101 to 100 reduces the penalty as much as moving from 1 to 0). In practice, this tends to drive small weights to/towards zero. However, if you have two features that are highly correlated, it tends to select one and drive the other towards zero. If you want a sparse model, this may be a good thing. However, if you want an "OR"-like operation in your model, so that learns to response to either A or B, despite having both A and B present in most of the training data, DropOut might be a better fit.

  • $\begingroup$ Thanks for the paper. Very insightful, I did not consider dropout for regularization till now. However, DropOut / DropConnection is not really what I want. What I look is to understand which inputs are useless and I don't need to extract. My goal is to make the perceptron small and faster to run in production. $\endgroup$ – Alexandru Feb 24 '17 at 10:48
  • $\begingroup$ It almost sounds like you want some form of feature selection then. Some people tend to look down on feature selection as a method for understanding the data, but I think it's particularly defensible when you're trying to find a reduced form of your model that works as well or almost as well. $\endgroup$ – Matt Krause Feb 25 '17 at 1:34

Besides $Ω(h) = ||h||_1 = \sum_i |h_i|$ there are also other forms of penalties like Student-t, KL divergence, average activation restrictions. Completely different approach is orthogonal matching pursuit which encodes an input x with the representation h that solves the constrained optimization problem:

$$\arg\limits_{h,||h||_0<k} \min ||x − Wh||^2$$

where $||h||_0$ is the number of non-zero entries of h. Basically it gives you kind of (but not exactly) dimensionality reduction (see PCA, auto-encoders), so you can try to just compress your input as well (you'll need less neurons to process it then). Anyway, you have to learn some "useful representation of the input". Sometimes, even deep-networks with ReLU (I assume your perceptron is one-layered) can help as the more layers you have the less connections between neurons you need ("exponential gain from depth"), less connections for output (sigmoids) neurons too.

You can read more in "7.10 Sparse Representations" of deep learning book.

Another approach is parameter sharing (like in CNN), which can reduce the size of your $W$ matrix.

Goal is not regularization, but to drop the feature completely and optimize the model for the production use.

If you want to drop a feature, this feature must be some noise (redundant), getting rid of noise is basically what regularization (in general sense) does anyway :). Otherwise, you could just drop $w$ values less than some $\epsilon$ - it might actually work sometimes (you can check error on your validation set to estimate the effect of such approach).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.