2
$\begingroup$

I have a dataset from the telecom sector that had 150+ columns for 7 months. Every row was essentially a line item containing cost,revenue and profit information for a subscriber. there can be between 0 and 1000 line items per subscriber per month and summing them up gives us the monthly cost, revenue and profit for that subscriber per month. However having just sums would make me lose a lot of information like most services used by subscriber etc. So I decided to work around that problem by flatening the table and converting all the catogorical variables into numerical ones (absolute value of monthly profit that came from them). I ended up with a wide data set for 2000 subscribers having about 3000 columns. Each row has now the catagorigal data for seven moths per subscriber as well as their monthly sums and the average monthly sum across 7 months. The resulting data set is fairly sparse. I would like to predict the absolute monthly profit for the 7th month based on all the values of the previous 6 months.

I'm using the following tutorial to try out ridge,lasso and elastic net regression on my data to see which one would provide better results . However, I'm having trouble interpreting the results.

I'm new to statistical learning and come from more of a computer science background so please excuse my lack of statistical knowledge. I would really appreciate it if someone more knowledgable in stats and R than I'm will have a look at my code and point out how I could get the RMSE for the best fit and also explain the plots to me.

library (glmnet)
require(caTools)
set.seed(111) 

new_flat <- fread('RED_SAMPLED_DATA_WITH_HEADERS.csv', header=TRUE, sep = ',') 
sample = sample.split(new_flat$SUBSCRIPTION_ID, SplitRatio = .80)
train = subset(new_flat, sample == TRUE)
test = subset(new_flat, sample == FALSE)

x=model.matrix(c201512_TOTAL_MARGIN~.-SUBSCRIPTION_ID,data=train) 
y=train$c201512_TOTAL_MARGIN

x1=model.matrix(c201512_TOTAL_MARGIN~.-SUBSCRIPTION_ID,data=test) 
y1=test$c201512_TOTAL_MARGIN



# Fit models:
fit.lasso <- glmnet(x, y, family="gaussian", alpha=1)
fit.ridge <- glmnet(x, y, family="gaussian", alpha=0)
fit.elnet <- glmnet(x, y, family="gaussian", alpha=.5)


# 10-fold Cross validation for each alpha = 0, 0.1, ... , 0.9, 1.0
fit.lasso.cv <- cv.glmnet(x, y, type.measure="mse", alpha=1, 
                          family="gaussian")
fit.ridge.cv <- cv.glmnet(x, y, type.measure="mse", alpha=0,
                          family="gaussian")
fit.elnet.cv <- cv.glmnet(x, y, type.measure="mse", alpha=.5,
                          family="gaussian")

for (i in 0:10) {
  assign(paste("fit", i, sep=""), cv.glmnet(x, y, type.measure="mse", 
                                            alpha=i/10,family="gaussian"))
}


# Plot solution paths:
par(mfrow=c(3,2))
# For plotting options, type '?plot.glmnet' in R console
plot(fit.lasso, xvar="lambda")
plot(fit10, main="LASSO")

plot(fit.ridge, xvar="lambda")
plot(fit0, main="Ridge")

plot(fit.elnet, xvar="lambda")
plot(fit5, main="Elastic Net")


yhat0 <- predict(fit0, s=fit0$lambda.min, newx=x1)
yhat1 <- predict(fit1, s=fit1$lambda.min, newx=x1)
yhat2 <- predict(fit2, s=fit2$lambda.min, newx=x1)
yhat3 <- predict(fit3, s=fit3$lambda.min, newx=x1)
yhat4 <- predict(fit4, s=fit4$lambda.min, newx=x1)
yhat5 <- predict(fit5, s=fit5$lambda.min, newx=x1)
yhat6 <- predict(fit6, s=fit6$lambda.min, newx=x1)
yhat7 <- predict(fit7, s=fit7$lambda.min, newx=x1)
yhat8 <- predict(fit8, s=fit8$lambda.min, newx=x1)
yhat9 <- predict(fit9, s=fit9$lambda.min, newx=x1)
yhat10 <- predict(fit10, s=fit10$lambda.min, newx=x1)

mse0 <- mean((y1 - yhat0)^2)
mse1 <- mean((y1 - yhat1)^2)
mse2 <- mean((y1 - yhat2)^2)
mse3 <- mean((y1 - yhat3)^2)
mse4 <- mean((y1 - yhat4)^2)
mse5 <- mean((y1 - yhat5)^2)
mse6 <- mean((y1 - yhat6)^2)
mse7 <- mean((y1 - yhat7)^2)
mse8 <- mean((y1 - yhat8)^2)
mse9 <- mean((y1 - yhat9)^2)
mse10 <- mean((y1 - yhat10)^2)

The plot in the code looks like this

$\endgroup$
  • $\begingroup$ A word on programming. This kind of thing: assign(paste("fit", i, sep="")) is what a list data structure is for. Whenever you end up with variables like x0, x1, etc, you should be using a list. $\endgroup$ – Matthew Drury Nov 23 '16 at 16:11
2
$\begingroup$

The RMSE is the Root square of the Mean Square Error (MSE).

It is the metric you use to assess the quality of your model. The lowest the better. In the end, you pick the model with the lowest RMSE. (or MSE)

For a good tutorial on elastic net, the one provided with the R package is the reference.

glmnet is a package that fits a generalized linear model via penalized maximum likelihood. The regularisation (penalty) is used when there is a high level of covariance among the variables or simply too much variables. It allows to do variable selection by reducing the parameter of the less significant variables to 0.

The lambda parameter allows to control the importance of the regularisation, aka, how much variables will be "muted". A grid search is done by the function glmnet and the best lambda value is obtained with fit$lambda.min.

In elastic net, the penalty is a mix of lasso and ridge. The alpha parameter control the balance of the penalty between ridge (penalisation of norm 2) and lasso (penalisation of norm 1).

Plots on the left decribe the coefficients values depending on the log of lambda.

  • In the case of lasso you have one variable which shows a rather anormal value of parameter. You may want to normalise that variable.

Plots on the right describe the MSE depending on the log of lambda.

  • It suggests to take a value of log(lambda) of 2.9 for lasso, 10.1 for ridge and 4.1 for elastic net.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.