4
$\begingroup$

Suppose a model which predicts which location/landmark a walking tourist is going to visit next, based on two geographical input features:

  • the last neighborhood this person has walked through
  • the second-to-last neighborhood this person has walked through (so, before the last one)

The distribution of the training data looks as follows:

before-last-neighborhood last-neighborood next-visited-place count
central park times square South of times square 10,000
central park times square North of times square 16
wall street times square North of times square 90
wall street times square South of times square 3

In this case, because of the imbalance of the data, the issue is that the model puts a very strong predictive power on times-square => somewhere south, regardless of the neighborhood crossed before that.

However, common sense would have anyone say that someone going through Times Square from Wall Street is probably going North, not South. And the training data actually reflects this, it just happens that there were many more people crossing Times Square from the North, than from the South.

What would be some effective ways to make a model more robust to this phenomenon, and effectively learn that e.g. wall street + times-square => going north?

I've tried different forms of feature engineering, adding more features (e.g. nationality of the tourist, gender, and other similar attributes), combining (before-last-zone + last-zone) into a single categorical feature, but all this only helps marginally. The reality is still that most people crossing Times Square are doing so Southbound, and the model will insist on predicting South for anyone crossing Times Square, regardless of their provenance.

In a neural network context, would there be for instance a way to assign somehow "more weight" to a particular combination of features, as a way to tell the model that these are the most importants features to look at?

I'd also like to mention that the real-world problem is much more complex than this and it would be intractable to try to artificially rebalance the data via bagging/bootstrapping. This would be a 50+ dimensional task with literally thousands of edge cases to consider.

$\endgroup$
5
  • $\begingroup$ What kind of model are you using now? // It sure seems like the smart guess if that someone is headed south if they come from Central Park. Why do you want to predict that someone is headed north in the case? $\endgroup$
    – Dave
    Oct 26, 2021 at 21:02
  • $\begingroup$ @Dave a neural network built with Keras, with embedded categorical variables and a few numerical variables. The output is a vector of probabilities for the different potential locations. $\endgroup$
    – Jivan
    Oct 26, 2021 at 21:08
  • $\begingroup$ @Dave I'd like the model to predict North if the tourist is crossing Times Square from Wall Street (not from Central Park). But it won't, because it learned that Times Square = going South no matter the origin. $\endgroup$
    – Jivan
    Oct 26, 2021 at 21:10
  • $\begingroup$ Actually you're spot on, I made a typo which is now fixed. Thanks $\endgroup$
    – Jivan
    Oct 26, 2021 at 21:23
  • $\begingroup$ I've just realised that the case I'm describing might be a form of under-fitting. Will post an update. $\endgroup$
    – Jivan
    Oct 26, 2021 at 23:19

2 Answers 2

2
$\begingroup$

Sometimes the simplest solutions are the one we think of last.

I eventually removed the individual features last-neighborhood and before-last-neighborhood, to replace them with a single combined feature instead: two-last-neighborhoods (previous and last, in order).

This way, the model doesn't put everything that has last-neighborhood: times-square in the same bracket anymore, and those who cross Times Square from Wall Street are now (correctly) interpreted as going somewhere North.

$\endgroup$
1
$\begingroup$

I think your model might be too complicated. A simple logistic regression handles this just fine.

# Define previous location
#
previous_location <- c(rep("Park", 10000), 
                       rep("Park", 16), 
                       rep("Wall", 90), 
                       rep("Wall", 3))

# By R's way of encoding:
# y = 0 denotes north
# y = 1 denotes south
#
direction <- as.factor(c(rep("South", 10000),
                         rep("North", 16), 
                         rep("North", 90), 
                         rep("South", 3)))

# Regress the direction on the origin previous_location (logistic regression)
#
L <- glm(direction ~ previous_location, family = binomial)

# Predict the conditional probability of heading north, given that someone
# is coming from Wall Street
#
z <- predict(L, data.frame(previous_location = "Wall"))
1 - 1/(1 + exp(-z))

I get $P(\text{North}\vert\text{Wall Street}) = 0.967741935483876$, almost exactly the $90/93$ probability that you get from analyzing the numbers you posted.

$\endgroup$
1
  • $\begingroup$ Thanks for taking the time to do this! Chopping down a model is often a good approach, however in this case the dataset is more complex and the relationship between the features is not so straightforward, even though the concept is the same. I've tried simpler approach (naive bayes, logistic regression, boosted trees...) but the overall performance is much worse than with a NN. $\endgroup$
    – Jivan
    Oct 26, 2021 at 22:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.