3
$\begingroup$

Is it possible to encode ID variables for an ML model in a way that the model can work with IDs not seen before?

For example, I want to predict the sentiment of a tweet, the account it may be originating from, and directed to, may help the classifier a lot to decide whether classify it as negative or positive.

Hence, is it possible to pass these features to the model in a way that, if a known id (shown whilst training) appears, it can help the prediction, and if there's a new id (unseen whilst training), the model can still work with it; either by no having impact whatsoever or by weighting the fact that it is an unrecognized id? If so, how would a Python + Sklearn implementation look like?

$\endgroup$
2

2 Answers 2

3
$\begingroup$

One of the general rules of building usable machine learning models is that you never include features that won't be available at prediction time, account ID is one of such features.

Technically, you could encode the ID using something like feature hashing, but this would only mean that for an unknown ID you would be using another, random ID from the training set. Another approach would be to replace the unknown ID with the average prediction from all the training set. Such solutions are not much better than not including the ID at all.

Another problem with using IDs is the high cardinality. According to this site Twitter has 300 million active users. Let's imagine a perfect scenario, where you have the data on all the users. Let's also say that you are using a very simple linear regression model where you use as features one-hot encoded IDs of the accounts where the tweet originated from, one-hot encoded IDs of the accounts where it was directed to, and their interaction. Such model would have 300,000,000 + 300,000,000 + 300,000,000² number of parameters. Are you sure you want to do this?

The third problem is that for any ID you would usually not have much data. How many datapoints do you have on average per account ID, for how many do you have only a single point? To have reasonable estimates, you would want to have multiple points per each ID.

Moreover, it sounds like a rather useless feature if you want to predict the sentiment of a tweet. With such a feature, your model would learn things like "Jack always sends depressive tweets" and "Ann sends happy tweets", so what? This doesn't tell you anything about the sentiment of a particular tweet. Even if for some reason it makes sense for you to assume that it is a constant characteristic of a person to send happy or sad tweets (is it?), you could do better by clustering people into two groups of "happy" and "sad" users and use this as a feature, this would be both available at prediction time and results in one parameter rather than a quadrillion.

Using IDs as features may make sense in case they will be available at prediction time and you have them all represented in the training set, there is not a huge number of them, and you have many datapoints per each ID. For example, you are doing medical research and use the ICD-10 codes of diagnosed diseases as features (still, some codes may be very rare), or you could use one-hot-encoded countries as features in an econometric model, etc. Notice that such features represent some general groupings that apply to many samples in your dataset, rather than individual IDs. With individual IDs you are learning to make predictions for the particular person, but then, you need to have enough data for that person.

Finally, if you have the account IDs and other data, and data on interactions of this account with other accounts, it can be used to de-anonymize the data. Using such data may pose ethical and legal concerns. Say that you train a neural network using account ID as a feature, then one of the users fills a GDPR request to be forgotten. In such a case, you should throw away your model and re-train it with the data not containing this user or face high penalties. I'm not saying that if you didn't use account ID as a feature this won't be a problem, but if you used the ID it's a much more obvious violation since your model de facto serves as a kind of black-box database that records the history of the user indexed by their account ID.

$\endgroup$
1
$\begingroup$

That's one of the important things in Data Science : you have to create your variables, good enough so your model can predict well. Here, you pointed well, that you can't directly use IDs, but with profiles information, you can create variables that will reflect what you see as a human when you analyze the tweet. A few examples : Number of people followed by the publisher, Number of people followed by the person it's directed to, Number of people followed by both the publisher and the person it's directed to, Percentage of the publisher's follows also followed by the person it's directed to, sentiment (% positive) of past tweets done by sender, sentiment (% positive) of past tweets done by receiver, do they follow each other...

You have to use your knowledge of the problem to create such variables, and find the necessary info to create them (form APIs, etc). If you have enough of such variables, you can make each user fairly unique (just like an id variable would do). Then, you can test those variables on your model, make feature selection etc.

$\endgroup$
1
  • 1
    $\begingroup$ True, that's the obvious way, related features, I was wondering if there was some kind of way of encoding the IDs directly in a way that work because the number of followers most likely won't say much about whether the tweet is positive or not, but the account directed to (id) most likely will, so that's the doubt; so far I haven't found an answer to whether there is some smart workaround around this, I'm reluctant to think it is impossible at all haha $\endgroup$
    – billdoe
    Commented Jul 26, 2021 at 17:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.