I am working with daily sales of products in a supermarket. The purpose is that of accurate forecasting. For each product sales I have many predictors, which include price,stockout,promotions of the product itself and of the other products related to this.
I am not interested in fitting a different model for each product, since in such a case I would include all regressors for each regression curve and then apply ridge regression or some sort of variable selection method.
Instead I would like to have a smaller representation of my regressors: I expect that there is an intelligent being behind the price shifts policy and inventory level and promotion choices.
Stockouts and Promotions are $0/1$ flags, while prices are real. I tend to exclude Principal Component Analysis as a technique (since I have binary variables) and I would instead use Independent Component Analysis.
Which technique do you think suits best my problem?
Let me be clearer, this is the kind of data I would like to reduce dimensionally:
priceP1 priceP2 priceP3 priceP4 promP1 promP2 promP3 promP4 0 0 0 0 1 0 1 0 -1 -2 0 0 1 0 0 1 0 0 +1 0 1 0 1 1
Where -1 denotes a minor downshift in price (less than 15%), -2 denotes a big downshift in price (over 15%), and as regards promotions I have 0/1 flags.
Right now I'm looking at products correlated (for example all product Coca-Cola in pet bottles), since I expect that there are latent factors behind them ore than considering random products (like water shifts in prices and red wine ones).
I tested in Python both PCA and Factor Analysis over 13 products, considering first just shifts in prices:
from sklearn.decomposition import PCA from sklearn.decomposition import FactorAnalysis factor = FactorAnalysis(n_components=8, , random_state=101).fit(X) pca = PCA().fit(X)
PCA results are:
print pca.explained_variance_ratio_ [ 2.76790694e-01 2.37085044e-01 1.43677025e-01 1.30717864e-01 6.70670210e-02 5.33406116e-02 4.09484124e-02 1.76802848e-02 1.52392222e-02 1.23767594e-02 5.07706129e-03 4.85649858e-34 1.87735378e-35]
I would say that 6 of my components explain almost 80% of the variance. Now let me output the score of 5 products over each component found:
COCA COLA PET 500X4 ML 2000 COCA COLA S/CAFF 1750 ML 1750 \ 0 3.745765e-02 2.716548e-01 1 -1.435577e-01 -1.940115e-01 2 -7.634235e-03 -6.941557e-01 3 -6.643485e-01 3.019650e-03 4 -9.205165e-03 -1.247160e-01 5 -4.726620e-02 -5.909807e-01 6 5.509576e-03 -1.119261e-01 7 4.843525e-01 -1.656424e-01 8 2.050570e-01 -2.948274e-02 9 5.075194e-01 3.249049e-02 10 -0.000000e+00 -0.000000e+00 11 -1.158277e-16 3.832722e-17 12 -2.253293e-16 3.507935e-17 COCA S/CAF.PET 500X4 ML 2000 COCACOLA S/CAFF.500 ML 500 \ 0 -2.132371e-02 -6.776264e-21 1 -1.031520e-01 0.000000e+00 2 -2.274758e-03 -0.000000e+00 3 -7.223773e-01 0.000000e+00 4 7.432427e-02 5.551115e-17 5 1.130822e-01 -2.775558e-16 6 3.091060e-02 5.464379e-17 7 -4.619952e-01 2.220446e-16 8 -1.486847e-01 1.166059e-16 9 -4.607135e-01 0.000000e+00 10 -0.000000e+00 -0.000000e+00 11 2.139575e-16 5.861627e-02 12 1.699868e-16 9.982806e-01 COCA SENZA CAFF.LT 1,5 ML 1500 \ 0 -0.0 1 0.0 2 -0.0 3 0.0 4 0.0 5 -0.0 6 0.0 7 0.0 8 0.0 9 0.0 10 -1.0 11 -0.0 12 -0.0
As you can see there are some components which basically describe just one product, for example component 10 for product 5 and component 12 for product 4.