How to use weights for each observation in a classifier

Question

I have a data set with a number of observations. Based on the results of some preliminary analysis, I have calculated a score for each observation that denotes how "important" each observation is. I want to then train a classifier but I want the classifier to "focus" more on the observations which are more "important", i.e. the ones with the higher scores. Is there a way I can add these scores as "weights" in my classifier somehow? The idea is that the classifier should be able to inherit a little amount of information from the insignificant observations (as these might still have something slightly useful) but the ones with a higher score should be the ones the classifier learns "more" from. I have come across the idea of "weighted classification" but this is assigning weights to observations belonging in imbalanced classes, whereas I am looking at the case of balanced classes, just with some points being more "important" than others. Any ideas?

Answer 1

As you mentioned, it sounds similar to the imbalanced dataset problem.

Resample the dataset: You can sample important observations more often which is very easy to implement without changing your classification model. Here is an example of how to implement it in PyTochs using WeightedRandomSampler, CrossEntropyLoss(weight=class_weights), or in NumPy using probability in random choice.

Weighted loss: modify your loss so that it assigns extra weight to each observation. For example, something like this:

x = rand(16, 20)
y = randint(2, (16,))
# Try ones(16) here and it will be equivalent to
# regular CrossEntropyLoss
weights = rand(16) # set random weights

net = Linear(20, 2)

def element_weighted_loss(y_hat, y, weights):
    m = LogSoftmax(dim=1)
    criterion = NLLLoss(reduction='none')
    loss = criterion(m(y_hat), y)
    loss = loss * weights
    return loss.sum() / weights.sum()

weighted_loss = element_weighted_loss(net(x), y, weights)

Source

I would argue both methods are similar.