Is an overfitted model likely to get better with more data? I've been dealing with some regularization techniques for preventing overfitting in neural networks.
I'm wondering:
Is an overfitted model without modifying anything likely to get better with more data?
If I started with a capable model that had been generated using a fixed dataset, and using optimization to adjusted the parameters so that it was fitting both the generalizable form and the noise, aka overfitted, then I increased the training set size significantly, and then iterated the same optimizer over the model to make it fit the larger set, would it recover or improve its ability to generalize?
Bonus:
To continue the thought, can overfitting be thought of as either too little data or too much complexity of the model? How should I think about this?
 A: I think this is a good and relevant question to ask.
There is a current drive to focus on the data element of ML to move the emphasis from "big" data toward "good" data.  Andrew Ng @stanford on the topic
There is also an element of "model repair", also called "design augmentation", a fundamental in statistics that gets less exposure in the ML world.  ML has transfer of learning, where a pretrained model is adjusted or "fine tuned", to a particular use case, but in general the idea is to move from generalization to specialization, not the other direction.
Intended approach:

*

*make reference encoder and show its performance on test set

*reduce training set to 10% of original with same epochs and train,
show overfitting

*starting with over-fitted (from 2) train with segregated 90% of train, and show recovery of generalization.

So this is the Keras tutorial for autoencoders: link. Here is a subset of it, but built for use in the Rstudio version.  link
The code, is shown here because links to code break:
#' Trains a simple deep NN on the MNIST dataset.
#' 
#' Gets to 98.40% test accuracy after 20 epochs
#' (there is *a lot* of margin for parameter tuning).
#' 2 seconds per epoch on a K520 GPU.
#'

library(keras)

# Data Preparation ---------------------------------------------------

batch_size <- 128
num_classes <- 10
epochs <- 30

# The data, shuffled and split between train and test sets
c(c(x_train, y_train), c(x_test, y_test)) %<-% dataset_mnist()

x_train <- array_reshape(x_train, c(nrow(x_train), 784))
x_test <- array_reshape(x_test, c(nrow(x_test), 784))

# Transform RGB values into [0,1] range
x_train <- x_train / 255
x_test <- x_test / 255

cat(nrow(x_train), 'train samples\n')
cat(nrow(x_test), 'test samples\n')

# Convert class vectors to binary class matrices
y_train <- to_categorical(y_train, num_classes)
y_test <- to_categorical(y_test, num_classes)

# Define Model --------------------------------------------------------------

model <- keras_model_sequential()
model %>% 
  layer_dense(units = 256, activation = 'relu', input_shape = c(784)) %>% 
  layer_dropout(rate = 0.4) %>% 
  layer_dense(units = 128, activation = 'relu') %>%
  layer_dropout(rate = 0.3) %>%
  layer_dense(units = 10, activation = 'softmax')

summary(model)

model %>% compile(
  loss = 'categorical_crossentropy',
  optimizer = optimizer_rmsprop(),
  metrics = c('accuracy')
)

# Training & Evaluation ----------------------------------------------------

# Fit model to data
history <- model %>% fit(
  x_train, y_train,
  batch_size = batch_size,
  epochs = epochs,
  verbose = 1,
  validation_split = 0.2
)

plot(history)
  
score <- model %>% evaluate(
  x_test, y_test,
  verbose = 0
)

# Output metrics
cat('Test loss:', score[[1]], '\n')
cat('Test accuracy:', score[[2]], '\n')

The graphic output from this is shown here.

This is a reference run, so remember that on the test set the accuracy was 98.13%.
Now we insert these lines at line 26:
#save the current ones
x_train2 <- x_train
y_train2 <- y_train

#subset them (not elegantly)
df_train <- data.frame(x_train, y=y_train)
num_2_keep <- floor(0.00005*nrow(x_train))

df_train <- df_train %>% 
                    group_by(df_train$y) %>%
                    sample_n(size=num_2_keep)
                    
y_train <- df_train[,ncol(df_train2)]
x_train <- df_train[,1:784] %>% as.matrix()

What they do is make a training frame that has about 3 examples of each class for the encoder.
Here is the plot of history:

It clearly shows the "runaway" of validation loss and that generalization is wrecked.
Here is the accuracy to compare with "reference":
> # Output metrics
> cat('Test loss:', score[[1]], '\n')
Test loss: 2.1622 

> cat('Test accuracy:', score[[2]], '\n')
Test accuracy: 0.514 

It is terrible.
Now, let's add the following lines at line 76:
y_train2 <- to_categorical(y_train2, num_classes)
history2 <- model %>% fit(
  x_train2, y_train2,
  batch_size = batch_size,
  epochs = epochs,
  verbose = 1,
  validation_split = 0.2
)

plot(history2)

Here is the default output:

If you want you can modify the "repair" to have some variation in samples, but in general this shows that by extending the training set it is possible to make a good model form that has been over-fitted to recover its generalization.
