2
$\begingroup$

I've been dealing with some regularization techniques for preventing overfitting in neural networks.

I'm wondering:

Is an overfitted model without modifying anything likely to get better with more data?

If I started with a capable model that had been generated using a fixed dataset, and using optimization to adjusted the parameters so that it was fitting both the generalizable form and the noise, aka overfitted, then I increased the training set size significantly, and then iterated the same optimizer over the model to make it fit the larger set, would it recover or improve its ability to generalize?

Bonus:
To continue the thought, can overfitting be thought of as either too little data or too much complexity of the model? How should I think about this?

$\endgroup$
9
  • 3
    $\begingroup$ In what sense can the model "get better"? Are you assuming the model is somehow adjusted or modified as new data arrive? If so, by what method does that occur and how can it be applied "without modifying anything"?? $\endgroup$
    – whuber
    Apr 30, 2021 at 17:23
  • 1
    $\begingroup$ This is about model-repair. I could make a decent example that shows the effect. One of the more recent ideas in ML is to get the center off the AI topology and look at the nature of the data. It is a data-centric view, and this aligns with that. $\endgroup$ Apr 30, 2021 at 17:35
  • 1
    $\begingroup$ By "adding more data" do you mean refitting the model with more data? In that case, the details really matter. If the refitting procedure includes selecting variables, then the new model may be incomparable to the old, making it difficult even to talk about what "overfitting" or "get better" even mean. If the refitting uses the same variables, then adding more data might only demonstrate the degree to which the model does not fit properly, rather than improving. $\endgroup$
    – whuber
    Apr 30, 2021 at 21:02
  • 1
    $\begingroup$ It might be heretical of me to say the following, but I feel like the process of throwing away data points That one does not like from the data is approximately the same level of sin as throwing away variables one does not like. The removal of the point is to remove all columns on a single row while the removal of a variable is to remove all rows in a single column. The difference in the sin is only the orientation of the deletion. The discarding of a point in wide data is as good or evil as discarding of a column in tall data. I try hard not to do either. $\endgroup$ May 1, 2021 at 2:45
  • 1
    $\begingroup$ @EngrStudent There are plenty of scenarios where valid conclusions can only be drawn when certain variables are omitted. See, e.g., collider bias in causal inference $\endgroup$ May 3, 2021 at 0:13

1 Answer 1

0
$\begingroup$

I think this is a good and relevant question to ask.

There is a current drive to focus on the data element of ML to move the emphasis from "big" data toward "good" data. Andrew Ng @stanford on the topic

There is also an element of "model repair", also called "design augmentation", a fundamental in statistics that gets less exposure in the ML world. ML has transfer of learning, where a pretrained model is adjusted or "fine tuned", to a particular use case, but in general the idea is to move from generalization to specialization, not the other direction.

Intended approach:

  1. make reference encoder and show its performance on test set
  2. reduce training set to 10% of original with same epochs and train,
    show overfitting
  3. starting with over-fitted (from 2) train with segregated 90% of train, and show recovery of generalization.

So this is the Keras tutorial for autoencoders: link. Here is a subset of it, but built for use in the Rstudio version. link

The code, is shown here because links to code break:

#' Trains a simple deep NN on the MNIST dataset.
#' 
#' Gets to 98.40% test accuracy after 20 epochs
#' (there is *a lot* of margin for parameter tuning).
#' 2 seconds per epoch on a K520 GPU.
#'

library(keras)

# Data Preparation ---------------------------------------------------

batch_size <- 128
num_classes <- 10
epochs <- 30

# The data, shuffled and split between train and test sets
c(c(x_train, y_train), c(x_test, y_test)) %<-% dataset_mnist()

x_train <- array_reshape(x_train, c(nrow(x_train), 784))
x_test <- array_reshape(x_test, c(nrow(x_test), 784))

# Transform RGB values into [0,1] range
x_train <- x_train / 255
x_test <- x_test / 255

cat(nrow(x_train), 'train samples\n')
cat(nrow(x_test), 'test samples\n')

# Convert class vectors to binary class matrices
y_train <- to_categorical(y_train, num_classes)
y_test <- to_categorical(y_test, num_classes)

# Define Model --------------------------------------------------------------

model <- keras_model_sequential()
model %>% 
  layer_dense(units = 256, activation = 'relu', input_shape = c(784)) %>% 
  layer_dropout(rate = 0.4) %>% 
  layer_dense(units = 128, activation = 'relu') %>%
  layer_dropout(rate = 0.3) %>%
  layer_dense(units = 10, activation = 'softmax')

summary(model)

model %>% compile(
  loss = 'categorical_crossentropy',
  optimizer = optimizer_rmsprop(),
  metrics = c('accuracy')
)

# Training & Evaluation ----------------------------------------------------

# Fit model to data
history <- model %>% fit(
  x_train, y_train,
  batch_size = batch_size,
  epochs = epochs,
  verbose = 1,
  validation_split = 0.2
)

plot(history)
  
score <- model %>% evaluate(
  x_test, y_test,
  verbose = 0
)

# Output metrics
cat('Test loss:', score[[1]], '\n')
cat('Test accuracy:', score[[2]], '\n')

The graphic output from this is shown here. enter image description here This is a reference run, so remember that on the test set the accuracy was 98.13%.

Now we insert these lines at line 26:

#save the current ones
x_train2 <- x_train
y_train2 <- y_train

#subset them (not elegantly)
df_train <- data.frame(x_train, y=y_train)
num_2_keep <- floor(0.00005*nrow(x_train))

df_train <- df_train %>% 
                    group_by(df_train$y) %>%
                    sample_n(size=num_2_keep)
                    
y_train <- df_train[,ncol(df_train2)]
x_train <- df_train[,1:784] %>% as.matrix()

What they do is make a training frame that has about 3 examples of each class for the encoder.

Here is the plot of history:

enter image description here

It clearly shows the "runaway" of validation loss and that generalization is wrecked.

Here is the accuracy to compare with "reference":

> # Output metrics
> cat('Test loss:', score[[1]], '\n')
Test loss: 2.1622 

> cat('Test accuracy:', score[[2]], '\n')
Test accuracy: 0.514 

It is terrible.

Now, let's add the following lines at line 76:

y_train2 <- to_categorical(y_train2, num_classes)
history2 <- model %>% fit(
  x_train2, y_train2,
  batch_size = batch_size,
  epochs = epochs,
  verbose = 1,
  validation_split = 0.2
)

plot(history2)

Here is the default output:

enter image description here

If you want you can modify the "repair" to have some variation in samples, but in general this shows that by extending the training set it is possible to make a good model form that has been over-fitted to recover its generalization.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.