I've been using a neural network to make predictions. So my training data is in one .csv file which I read-in and then scale. My test data is in another file that I read-in and is also scaled. However, my test data does not contain an output value column because I am going to be submitting predictions for it to Kaggle to test if the value is correct. (It is part of this Kaggle competition: https://www.kaggle.com/c/carseatsales).

I am not really sure how to scale my prediction if my test data does not have this output column.

Here is how I scaled the data:

train10           = read.csv("Carseats_training.csv")
train10$ShelveLoc = as.numeric(train10$ShelveLoc)
train10$Urban     = as.numeric(train10$Urban)
train10$US        = as.numeric(train10$US)

maxs  <- apply(train10, 2, max) 
mins  <- apply(train10, 2, min)
index <- sample(1:nrow(train10), round(1*nrow(train10)))

scaled <- as.data.frame(scale(train10, center = mins, scale = maxs - mins))

train100 <- scaled[index,]

test10           = read.csv("Carseats_testing.xls")
test10$ShelveLoc = as.numeric(test10$ShelveLoc)
test10$Urban     = as.numeric(test10$Urban)
test10$US        = as.numeric(test10$US)

maxss  <- apply(test10, 2, max) 
minss  <- apply(test10, 2, min)
index1 <- sample(1:nrow(test10), round(1*nrow(test10)))

scaleds <- as.data.frame(scale(test10, center = minss, scale = maxss - minss))

test100 <- scaleds[index1,]

This is my neural network:

nn <- neuralnet(Sales ~ CompPrice + Income + Advertising + Population + Price + ShelveLoc  
                        + Age + Education + Urban + US
                , data = train100
                , hidden = c(5,3)
                , linear.output = T)

I am trying to make a prediction on sales.

pr.nn <- compute(nn, test100[,2:11])

But now I am not really sure how to scale my result.

I would really appreciate any help. I have been stuck on this part for while now.

  • $\begingroup$ Hello there. Couple of things. (1) Please include a hyperlink to the Kaggle competition. (2) Data is often scaled by max - min or sd, and centered by med or mean. They are actual values, which you can also use to e.g. unscale. – Reviewer $\endgroup$ – Jim Apr 22 '18 at 19:53
  • $\begingroup$ Hi @Jim this is the competition : kaggle.com/c/carseatsales I used max - min to scale the data. but now I can't seem to find how to scale the result that I am getting $\endgroup$ – Plumorane Apr 22 '18 at 19:59
  • $\begingroup$ You do not need to scale the output Sales while training; only the predictors. $\endgroup$ – Jim Apr 22 '18 at 20:05
  • $\begingroup$ @Jim I edited my post to include how I scaled the data. Do you mind taking a look at it? I am not positive as to what you mean about only scaling the predictors. $\endgroup$ – Plumorane Apr 22 '18 at 20:14
  • 1
    $\begingroup$ Sales is your output, aka outcome, aka y-variable: the one you want to predict. All other variables – the ones you want to predict the outcome with – are the predictors, aka regressors, aka X-variables: CompPrice, Income, Advertising, Population, Price, ShelveLoc, Age, Education, Urban, US in your neural net. $\endgroup$ – Jim Apr 22 '18 at 20:19
  1. We only (need to) scale the predictor variables. We do this to help our machine learning algorithm converge (faster) to a minimum of the loss function. In the case of a (feed-forward) neural network, the parameters which we want to estimate are the weights and "biases".
  2. For scaling between $0$ and $1$, we use the following transform for each predictor variable: $$ \tilde{x}_{ij} = \frac{x_{ij} - \min_i(x_{ij})}{\max_i(x_{ij}) - \min_i(x_{ij})}, $$ where the rows are indexed with $i$ and columns with $j$, as is customary.
  3. Given the first point above, one sees that one can simply use the found minima and maxima of the training set for the predictors in the test set.

When you trained on the training data, you scale the input values using a single global mean and variance that you calculate from the training set. So, you'll have one training set mean, a single scalar value, and one training set variance, also a single scale, global value.

When you run on the test set, you use the exact same pair of mean and variance values as for the training set. You hope that the distribution of the test set approximately matches that of the training set.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.