3
$\begingroup$

This page shows a classification problem. They have used bias as well as bias along with class weights.

  1. What is the difference between bias and weights? In some other techniques such as Random Forest, we could adjust cut off from 50% to say 90% so that if the edge node has 90% examples of high-density class then only it will predict that class. Out of bias and weights, which one is similar to cut off probability?
  2. When should I use just one of them or both? I can always run 3 models 1 with bias, 1 with class weight and 1 with both and compare the results. But is there any better reference?
keras.layers.Dense(1, activation='sigmoid',
                   bias_initializer=output_bias)

and

weighted_history = weighted_model.fit(
    train_features,
    train_labels,
    batch_size=BATCH_SIZE,
    epochs=EPOCHS,
    callbacks = [early_stopping],
    validation_data=(val_features, val_labels),
    # The class weights go here
    class_weight=class_weight) 
$\endgroup$
3
  • 1
    $\begingroup$ The article you reference represents a basic misunderstanding of proper accuracy scoring rules. See fharrell.com/post/class-damage $\endgroup$ Jan 29 '20 at 14:53
  • 1
    $\begingroup$ This sentence is confusing. "They have used bias as well as bias along with class weights." Is it possible that you've repeated a phrase? Is there a clearer way to express what you mean? $\endgroup$
    – Sycorax
    Jan 29 '20 at 16:46
  • $\begingroup$ Cross-posted at datascience.stackexchange.com/q/67001/55122 $\endgroup$ Jan 30 '20 at 15:00
1
+50
$\begingroup$

The snippet you are showing takes is a layer that takes the output of a 16-dimensional hidden layer (let's call it $\mathbf{h}$) and does the following:

$$o = \sigma\left( \mathbf{w}^\top\mathbf{h} + b \right)$$

where $\mathbf{w} \in \mathbb{R}^{16}$ is the weight vector (inside fo TF is a matrix with a single row) and $b \in \mathbb{R}$ is the bias.

The role of bias, here, is to learn a preference for the most frequent class. If $o$ is positive, the model predicts, if it's negative, it predicts the other class. A reasonable strategy for the model is to learn a strong bias towards the most frequent class and learn to look for evidence why it should be otherwise. If you remove the bias from the model, you are basically forcing the model not to use this strategy which might be often hard to learn.

Another way if interpreting the bias is by viewing it as a kind of prior probability and the other term as getting evidence against the prior.

The class weights are used when computing the loss function. Here, you have two classes and thus you would have two weights $w_0$ and $w_1$. So, the loss will be as if there were $w_0$ examples of class $0$ and $w_1$ examples of class $1$. In this way, you can the underrepresented class more prominent and make sure the model learns more about it, but it also skews the bias term to a different distribution than is in the data.

$\endgroup$
5
  • $\begingroup$ what do you mean if o is positive? o is a calculation, right? how would we know its value before calculating? I understand function of weights. But in this case they have an extra input class_weight=class_weight, what does it do? $\endgroup$ Jan 28 '20 at 15:26
  • $\begingroup$ Oh sorry, I misunderstood your question, I thought you meant different weights. By being positive I meant "gets evaluated as a positive number when it gets evaluated". $\endgroup$
    – Jindřich
    Jan 29 '20 at 13:36
  • $\begingroup$ I edited, so it should make sense now. $\endgroup$
    – Jindřich
    Jan 29 '20 at 13:49
  • $\begingroup$ could you answer my question 2? i am still not clear $\endgroup$ Jan 30 '20 at 23:28
  • $\begingroup$ There is, of course, no clear answer, you need to experiment, it very much depends on your data. Unless your data is so imbalanced that you would not be able to learn a suitable representation for the minority class, I would avoid setting class weights, so you train on a realistic distribution of target classes. $\endgroup$
    – Jindřich
    Jan 31 '20 at 14:52
1
$\begingroup$
  1. When we're talking about the estimated parameters of a neural network, a "bias" is any constant that's added to an input. Consider logistic regression, i.e. a neural network without hidden layers and a single, sigmoidal output. This network has the prediction equation

    $$ \hat{y} = \sigma(w^\top x+b) $$

    where $x$ is the input vector, $w$ is the vector of weights and $b$ is the bias. The function $\sigma$ yields probabilities as its output: $0<\sigma(z)=\frac{1}{\exp(-z)+1}<1$. For this model, it should be obvious that increasing $b$ while keeping all else equal increases the predicted probability; contrariwise, decreasing $b$ does the opposite.

    However, the situation becomes more complex when we consider a neural network with hidden layers. This is because changing the bias for a node at the "beginning" or "middle" of the network can increase or decrease the predicted probability, depending on what happens "downstream" of the unit.

    Finally, the utility of class weights in neural networks is not as simple as presented in your TensorFlow tutorial. The intuition that we have about instance weights in simple settings like logistic regression do not hold when using over-parameterized neural networks. Indeed, neural networks are flexible enough that they can effectively overcome instance weighting.

    Importance-weighted risk minimization is a key ingredient in many machine learning algorithms for causal inference, domain adaptation, class imbalance, and off-policy reinforcement learning. While the effect of importance weighting is well-characterized for low-capacity misspecified models, little is known about how it impacts over-parameterized, deep neural networks. Inspired by recent theoretical results showing that on (linearly) separable data, deep linear networks optimized by SGD learn weight-agnostic solutions, we ask, for realistic deep networks, for which many practical datasets are separable, what is the effect of importance weighting? We present the surprising finding that while importance weighting impacts deep nets early in training, so long as the nets are able to separate the training data, its effect diminishes over successive epochs. Moreover, while L2 regularization and batch normalization (but not dropout), restore some of the impact of importance weighting, they express the effect via (seemingly) the wrong abstraction: why should practitioners tweak the L2 regularization, and by how much, to produce the correct weighting effect? We experimentally confirm these findings across a range of architectures and datasets.

    If by "bias" you only mean the ultimate bias applied in the final network layer, it should be clear that you have a choice to achieve a desired trade-off between classification errors:

    • You can adjust the final bias in the neural network to move all predicted probabilities up or down. This will distort the predicted probabilities. The caveat to focusing on hard classifications is that you're discarding all of the fine-grained information conveyed by a predicted probability. Frank Harrell has written a number of posts here and elsewhere about the dangers of using hard-and-fast cutoffs.

    • You can leave the final bias at whatever value minimizes the loss, and instead choose a different threshold to transform predicted probabilities into class designations. That is, instead of having the rule "If $\hat{y} > 0.5$, predict 'This is an image of a dog,'" you can use the rule "If $\hat{y} > 0.9$, predict 'This is an image of a dog.'" This option does not distort the predicted probabilities, and instead only changes which observations are allocated to which class.

  2. In light of the foregoing, my recommendation is to not use class weights. Let the optimizer find the optimal value of your weights and biases. Ideally, the decisions you make should result from a contextual consideration of the risks involved: if you're wrong, is a healthy person taking an unnecessary course of antibiotics? Or is a healthy person having a limb amputated? Or is a sick person missing out on a treatment? Is the treatment critical to their life, or is it too risky considering their overall health, age and other infirmities?

    If you need some sort of a hard decision (perhaps because you're creating an automation system requiring minimal human intervention), then you should pick a decision threshold which achieves your desired trade-off of correct predictions against incorrect predictions. That is, any time you're making a decision on the basis of a machine learning model's prediction, you're at risk of either a false positive or false negative. You should carefully assess the risks to your organization of both types of errors, and choose a threshold which balances the costs of both types of errors. In some problems, this means you'll tolerate a higher false positive rate in order to find as many positives as possible; in others, you'll tolerate a higher false negative rate in order that the positives that you uncover are "high confidence" targets.

$\endgroup$
0
$\begingroup$

As far as I know the bias_initializer is used in order to set the initial weights/biases.

class_weight is used when you have inbalanced distribution of classes eg. 10 roses (class 0), 1 tulip (class 1) and 2 coliflowers (class 2)

The model will learn the features of roses pretty well but disregard tulips and coliflowers since they are way less represented in the training data.

I personally take a representative sample of my training data. This poses some risk as not all classes may be included or your file might not be representative of your whole training data. I then take this representative file and calculate the class distribution on it.

The following function takes an input csv file and counts all the classes. Then the weight distribution is calculated and a dictionary returned which the model.fit will consume.

def getClassDistribution(path, lines):
    print('getting class distribution')
    progress = 0
    classes = dict()
    # open sample file and count classes
    with open(path, "r",encoding="utf-8",errors='ignore') as f:
        line = f.readline()
        while line:
            if line != '':
                label = int(line[-2:-1])
                if label in classes:
                    classes[label] += 1
                else:
                    classes[label] = 1
            printProgressBar(progress,lines)
            progress += 1
            line = f.readline()
    max = 0
    # calculate weight distribution
    for x in classes:
        if classes[x] > max:
            max = classes[x]
    for x in classes:
        classes[x] = classes[x]/max
        classes[x] = 1 + (1 - classes[x])
    print()
    return classes

classDistr = getClassDistribution(trainPath+"0.data", trainSteps*trainBadgeSize)

this will yield a dictionary like this:

classDistr = {0: 1.0,
                1: 10.0,
                2: 5.0}

then in model.fit:

class_weight=classDistr,

Basically it tells the network to train ten times "harder" on tulips and 5 times harder on coli-flowers accommodating for the different amount of samples.

Another example of the usage: I want to predict upward/downward movement. I have 70% of upward classes and 30% of downward classes. The model might get stuck predicting upward, simply because it is much more likely to occur rather than looking on the features. One solution is to get 50% up 50% down in the train data, another way of accommodation is class_weights

$\endgroup$
0
$\begingroup$

If your input has mean zero and standard deviation one, and your hidden layers are properly initialized then the output of your very first forward pass should have mean zero and standard deviation one (as I understand it). If the activation function on your output is softmax then your output for three classes will be [0.333, 0.333, 0.333] while the targets will be [0, 0, 1] etc. That's quite a large loss from the get go. If the classes are unbalanced, e.g. 1x [1, 0, 0], 1x [0, 1, 0], and 8x [0, 0, 1] then you might want the first forward pass to output [0.1, 0.1, 0.8] instead. Since the network will most likely learn some values that are close to those anyway.

Looking at https://math.stackexchange.com/a/3162684/365424 we get that we can invert a softmax function like so:

$x_{i} = \ln(S_{i}) + \ln(\sum_{i} \exp(x_{i}))$

with the logarithm of the sum being constant but not determined. So if you use the log of class frequencies you should end up with something that works.
In this example a bias of [-0.22, -0.22, -2.3].

But if you have applied class weights such that each [1, 0, 0] contributes 8 times as much to the loss as each [0, 0, 1] and so on, then you would want [0.333, 0.333, 0.333] in the first forward pass and leave the bias initialization alone. Since your loss is now balanced.
If your classes are only partially balanced then I don't think I can help you.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.