Faster training of a DNN with recurrent input data I want to train a deep neural net (MLP) to classify vectors and I have a huge dataset with continuous high-dimensional input. While going through my dataset, I observed some recurrent patterns.
So my question is, if I already know that vector $x$ will be seen 50000 times for example in my dataset, is there a way to use that knowledge to accelerate the training procedure ? 
 A: The loss function for an MLP is typically composed of the sum of loss functions for each individual observation (plus a regularization term sometimes),
$$
L = \sum_{i=1}^N L(x_i, y_i) + \Omega,
$$
where $L$ is the loss function over the entire data set (probably cross-entropy if you're doing classification), $L(X,y)$ is the loss function associated with input vector $X$ and target variable $y$, $x_i$ and $y_i$ are the input vector and target variable of the $i$'th observation, $N$ is the number of data points, and $\Omega$ is a regularization term (if you're using one).
If there are $k$ groups of identical input-output observations, and group $j$ has $n_j$ occurrences (for $j=1, \dots, k$) with input value $X_j$ and output value $y_j$ then it's easy to see that
$$
L = \sum_{j=1}^k n_j L(X_j, y_j) + \Omega.
$$
Expressing $L$ in this second form allows us to interpret it as a weighted data set of $k$ observations, where each observation $j$ has weight $n_j.$ 
The moral of the story is that you can turn all identical (identical in both input and target value) observations into a single observation with training weight that reflects its number of occurrences. Optimizing over this data set with the appropriate weighting scheme is equivalent to optimizing over the original data set, where each observation has equal weights. Most neural network packages allow you to use weighted observations.
EDIT: I fixed a mistake in which I said it's enough to group observations with identical inputs. They must also have identical target variable values, otherwise the loss function would be different for those observations. So two observations with the same input value but two different target values should still be placed in separate groups.
