Skip to main content
deleted 1 character in body
Source Link
Tim
  • 141.2k
  • 26
  • 270
  • 512

One of the general rules of building usable machine learning models is that you never include features that won't be available at prediction time, account ID is one of such features.

Technically, you could encode the ID using something like feature hashing, but this would only mean that for an unknown ID you would be using another, random ID from the training set. Another approach would be to replace the unknown ID with the average prediction from all the training set. Such solutions are not much better than not including the ID at all.

Another problem with using IDs is the high cardinality. According to this site Twitter has 300 million active users. Let's imagine a perfect scenario, where you have the data on all the users. Let's also say that you are using a very simple linear regression model where you use as features one-hot encoded IDs of the accounts where the tweet originatingoriginated from, one-hot encoded IDs of the accounts where it was directed to, and their interaction. Such model would have 300,000,000 + 300,000,000 + 300,000,000² number of parameters. Are you sure you want to do this?

The third problem is that for any ID you would usually not have much data. How many datapoints do you have on average per account ID, for how many do you have only a single point? To have reasonable estimates, you would want to have multiple points per each ID.

Moreover, it sounds like a rather useless feature if you want to predict the sentiment of a tweet. With such a feature, your model would learn things like "Jack always sends depressive tweets" and "Ann sends happy tweets", so what? This doesn't tell you anything about the sentiment of a particular tweet. Even if for some reason it makes sense for you to assume that it is a constant characteristic of a person to send happy or sad tweets (is it?), you could do better by clustering people into two groups of "happy" and "sad" users and use this as a feature, this would be both available at prediction time and results in one parameter rather than a quadrillion.

Using IDs as features may make sense in case they will be available at prediction time and you have them all represented in the training set, there is not a huge number of them, and you have many datapoints per each ID. For example, you are doing medical research and use the ICD-10 codes of diagnosed diseases as features (still, some codes may be very rare), or you could use one-hot-encoded countries as features in an econometric model, etc. Notice that such features represent some general groupings that apply to many samples in your dataset, rather than individual IDs. With individual IDs you are learning to make predictions for the particular person, but then, you need to have enough data for that person.

Finally, if you have the account IDs and other data, and data on interactions of this account with other accounts, it can be used to de-anonymize the data. Using such data may pose ethical and legal concerns. Say that you train a neural network using account ID as a feature, then one of the users fills a GDPR request to be forgotten. In such a case, you should throw away your model and re-train it with the data not containing this user or face high penalties. I'm not saying that if you didn't use account ID as a feature this won't be a problem, but if you used the ID it's a much more obvious violation since your model de facto serves as a kind of black-box database that records the history of the user indexed by their account ID.

One of the general rules of building usable machine learning models is that you never include features that won't be available at prediction time, account ID is one of such features.

Technically, you could encode the ID using something like feature hashing, but this would only mean that for an unknown ID you would be using another, random ID from the training set. Another approach would be to replace the unknown ID with the average prediction from all the training set. Such solutions are not much better than not including the ID at all.

Another problem with using IDs is the high cardinality. According to this site Twitter has 300 million active users. Let's imagine a perfect scenario, where you have the data on all the users. Let's also say that you are using a very simple linear regression model where you use as features one-hot encoded IDs of the accounts where the tweet originating from, one-hot encoded IDs of the accounts where it was directed to, and their interaction. Such model would have 300,000,000 + 300,000,000 + 300,000,000² number of parameters. Are you sure you want to do this?

The third problem is that for any ID you would usually not have much data. How many datapoints do you have on average per account ID, for how many do you have only a single point? To have reasonable estimates, you would want to have multiple points per each ID.

Moreover, it sounds like a rather useless feature if you want to predict the sentiment of a tweet. With such a feature, your model would learn things like "Jack always sends depressive tweets" and "Ann sends happy tweets", so what? This doesn't tell you anything about the sentiment of a particular tweet. Even if for some reason it makes sense for you to assume that it is a constant characteristic of a person to send happy or sad tweets (is it?), you could do better by clustering people into two groups of "happy" and "sad" users and use this as a feature, this would be both available at prediction time and results in one parameter rather than a quadrillion.

Using IDs as features may make sense in case they will be available at prediction time and you have them all represented in the training set, there is not a huge number of them, and you have many datapoints per each ID. For example, you are doing medical research and use the ICD-10 codes of diagnosed diseases as features (still, some codes may be very rare), or you could use one-hot-encoded countries as features in an econometric model, etc. Notice that such features represent some general groupings that apply to many samples in your dataset, rather than individual IDs. With individual IDs you are learning to make predictions for the particular person, but then, you need to have enough data for that person.

Finally, if you have the account IDs and other data, and data on interactions of this account with other accounts, it can be used to de-anonymize the data. Using such data may pose ethical and legal concerns. Say that you train a neural network using account ID as a feature, then one of the users fills a GDPR request to be forgotten. In such a case, you should throw away your model and re-train it with the data not containing this user or face high penalties. I'm not saying that if you didn't use account ID as a feature this won't be a problem, but if you used the ID it's a much more obvious violation since your model de facto serves as a kind of black-box database that records the history of the user indexed by their account ID.

One of the general rules of building usable machine learning models is that you never include features that won't be available at prediction time, account ID is one of such features.

Technically, you could encode the ID using something like feature hashing, but this would only mean that for an unknown ID you would be using another, random ID from the training set. Another approach would be to replace the unknown ID with the average prediction from all the training set. Such solutions are not much better than not including the ID at all.

Another problem with using IDs is the high cardinality. According to this site Twitter has 300 million active users. Let's imagine a perfect scenario, where you have the data on all the users. Let's also say that you are using a very simple linear regression model where you use as features one-hot encoded IDs of the accounts where the tweet originated from, one-hot encoded IDs of the accounts where it was directed to, and their interaction. Such model would have 300,000,000 + 300,000,000 + 300,000,000² number of parameters. Are you sure you want to do this?

The third problem is that for any ID you would usually not have much data. How many datapoints do you have on average per account ID, for how many do you have only a single point? To have reasonable estimates, you would want to have multiple points per each ID.

Moreover, it sounds like a rather useless feature if you want to predict the sentiment of a tweet. With such a feature, your model would learn things like "Jack always sends depressive tweets" and "Ann sends happy tweets", so what? This doesn't tell you anything about the sentiment of a particular tweet. Even if for some reason it makes sense for you to assume that it is a constant characteristic of a person to send happy or sad tweets (is it?), you could do better by clustering people into two groups of "happy" and "sad" users and use this as a feature, this would be both available at prediction time and results in one parameter rather than a quadrillion.

Using IDs as features may make sense in case they will be available at prediction time and you have them all represented in the training set, there is not a huge number of them, and you have many datapoints per each ID. For example, you are doing medical research and use the ICD-10 codes of diagnosed diseases as features (still, some codes may be very rare), or you could use one-hot-encoded countries as features in an econometric model, etc. Notice that such features represent some general groupings that apply to many samples in your dataset, rather than individual IDs. With individual IDs you are learning to make predictions for the particular person, but then, you need to have enough data for that person.

Finally, if you have the account IDs and other data, and data on interactions of this account with other accounts, it can be used to de-anonymize the data. Using such data may pose ethical and legal concerns. Say that you train a neural network using account ID as a feature, then one of the users fills a GDPR request to be forgotten. In such a case, you should throw away your model and re-train it with the data not containing this user or face high penalties. I'm not saying that if you didn't use account ID as a feature this won't be a problem, but if you used the ID it's a much more obvious violation since your model de facto serves as a kind of black-box database that records the history of the user indexed by their account ID.

Source Link
Tim
  • 141.2k
  • 26
  • 270
  • 512

One of the general rules of building usable machine learning models is that you never include features that won't be available at prediction time, account ID is one of such features.

Technically, you could encode the ID using something like feature hashing, but this would only mean that for an unknown ID you would be using another, random ID from the training set. Another approach would be to replace the unknown ID with the average prediction from all the training set. Such solutions are not much better than not including the ID at all.

Another problem with using IDs is the high cardinality. According to this site Twitter has 300 million active users. Let's imagine a perfect scenario, where you have the data on all the users. Let's also say that you are using a very simple linear regression model where you use as features one-hot encoded IDs of the accounts where the tweet originating from, one-hot encoded IDs of the accounts where it was directed to, and their interaction. Such model would have 300,000,000 + 300,000,000 + 300,000,000² number of parameters. Are you sure you want to do this?

The third problem is that for any ID you would usually not have much data. How many datapoints do you have on average per account ID, for how many do you have only a single point? To have reasonable estimates, you would want to have multiple points per each ID.

Moreover, it sounds like a rather useless feature if you want to predict the sentiment of a tweet. With such a feature, your model would learn things like "Jack always sends depressive tweets" and "Ann sends happy tweets", so what? This doesn't tell you anything about the sentiment of a particular tweet. Even if for some reason it makes sense for you to assume that it is a constant characteristic of a person to send happy or sad tweets (is it?), you could do better by clustering people into two groups of "happy" and "sad" users and use this as a feature, this would be both available at prediction time and results in one parameter rather than a quadrillion.

Using IDs as features may make sense in case they will be available at prediction time and you have them all represented in the training set, there is not a huge number of them, and you have many datapoints per each ID. For example, you are doing medical research and use the ICD-10 codes of diagnosed diseases as features (still, some codes may be very rare), or you could use one-hot-encoded countries as features in an econometric model, etc. Notice that such features represent some general groupings that apply to many samples in your dataset, rather than individual IDs. With individual IDs you are learning to make predictions for the particular person, but then, you need to have enough data for that person.

Finally, if you have the account IDs and other data, and data on interactions of this account with other accounts, it can be used to de-anonymize the data. Using such data may pose ethical and legal concerns. Say that you train a neural network using account ID as a feature, then one of the users fills a GDPR request to be forgotten. In such a case, you should throw away your model and re-train it with the data not containing this user or face high penalties. I'm not saying that if you didn't use account ID as a feature this won't be a problem, but if you used the ID it's a much more obvious violation since your model de facto serves as a kind of black-box database that records the history of the user indexed by their account ID.