0
$\begingroup$

I am preparing my dataset for a logistic regression and need to check how best to handle a column with categorical values. As the dataset is for sales transactions, the column in question is the unique product identifier - of which there are over 1000 unique values. To complicate matters, all but 5 of these values are 7-8 digit integers (the remainder are strings). Examples below

*77789876*
*2213_usd_99*

Should I assign a unique integer 1 to 1374 to each of the original product codes, and then normalise the new int in order to get this to work?

I did consider putting the product code in the index, however I am not sure that will work as ultimately, I am trying to predict the probability that it will be sold on a given date, so will need to pass it in as a variable later on.

Any help or advice would be welcomed.

UPDATE I tried a min max scaler an that ended up assigning the same values to multiple items i.e. 0.998

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ See stats.stackexchange.com/questions/146907/… How many observations do you have? What is the goal of the modeling? Tell us some more context ... $\endgroup$
    – kjetil b halvorsen ♦
    Commented Mar 9, 2020 at 12:54
  • $\begingroup$ The ultimate goal is to be able to provide a probability score for the event that each of the products will be sold on a specific date (after reporting period). Therefore I need a unique identifier per product (and date of interest) to put into the LR equation to yield the final value. $\endgroup$
    – tristar8
    Commented Mar 9, 2020 at 13:08
  • $\begingroup$ You need to represent the product categories with dummy variables, your integer coding scheme will not work. But, somewhat depending on $n$, having a thousand parameters might be to much, also, the thousand products probably do not all act differently! So some methods (see link in my first comment) such as the fused lasso will automatically shrink coefficient towards each other, making better predictions. $\endgroup$
    – kjetil b halvorsen ♦
    Commented Mar 9, 2020 at 13:12

1 Answer 1

Reset to default
1
$\begingroup$

I can't quite tell what you are doing in enough detail. What is needed is to convert your column of categorical variables to a contrast matrix of some sort. However, that is usually handled "magically" by the software (although not so magically you couldn't change the style of contrasts if you wished).

I think all you need is a column of strings (factors / categorical variables). As I say, I don't know what package you are using. But statsmodels (and packages built on that) will create a contrast matrix for you from the factor labels (the text) https://www.statsmodels.org/dev/examples/notebooks/generated/contrasts.html. You shouldn't yourself need to worry about creating a column with unique integer variables, nor should you have to worry about creating the contrast matrix.

However, one little gotcha, where it is worth knowing a little about the "magic" is that with k = a number of 1000 different values, you will create a contrast matrix with k - 1 (a number one less than the number of different values). If you have multiple rows of data per column, that is going to be a big matrix, which in turn might raise other problems.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.