I want to predict to which product category a product belongs. A total of 400k products need to be translated from the old (less refined) to the new product category tree. (E.g. alarm clock used to fall under 'Electronics' and will now belong to 'Alarm clocks'.) So far 36k products have already been partly allocated to ~400 (out of 800) new product categories. The filling rate ranges from 1% to 95%.
Product data (among others) contains variables: name, description, price, dimensions, color and the old label . The idea was to construct features out of the unstructured variables through tokenisation -> TF-IDF.
- Train one multi-label prediction model (e.g. Ridge classification + stratified CV) on the labeled data. Then predict the category only for subset that, based on the old product tree, contains all possible products. (e.g. predict if unlabelled 'Electronics' products are 'Alarm clocks')
- Based on the predicted probability present the unlabelled product to a content manager that, if labelled, would result in the highest information gain.
- Propose to which extend the remaining 400 categories should be filled (e.g. 60%) and which products to label first.
What would your preferred approach be?