I'm just starting to learn about CNN (convolutional neural networks). Does the test data also need to be divided into batches, similar to how it's done with the training data?
$\begingroup$
$\endgroup$
1
-
$\begingroup$ Hi Xio and Welcome to CV! Could you please provide supporting information for "the test data also needs to be divided into batches" claim? $\endgroup$– utobiSep 25 at 13:23
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
3
NO
Using batches is a “trick” for the numerical optimization of the neural network parameters. Once it comes time to predict on the test set, you are not trying to adjust the network parameters. You are just seeing what the fitted model predicts for a set of input images (or whatever your inputs are, but CNNs are classically used for images). The calculation is much simpler. You just have some function and evaluate that function at a point (such as an individual image), then move to the next point, then the next.
Perhaps this means that you are using batches of size one, but that really seems like the wrong philosophy to take: using batches instead of all of the data at once is to help the computer with the optimization.
If you can’t fit the entire test set into memory, then you will have to split it into chunks to make predictions. Perhaps these chunks are reasonably called “batches”.
-
7$\begingroup$ You’ll need to divide the test data into batches if it’s too large to fit into memory. $\endgroup$– Sycorax ♦Sep 25 at 12:11
-
$\begingroup$ Thank you for your response. I still have a few more questions. I noticed some people are building models using three types of datasets: train sets, test sets, and validation sets. What distinguishes the validation set from the test set, and do both the test set and validation set need to be divided into batches? $\endgroup$– xioXueiSep 27 at 3:52
-