Stack Exchange network consists of 183 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.
You are probably mixing the validation set - used to decide when to stop the training - and the test set - used to evaluate real world performance of the model. Usually you analyze test sets using selected accuracy measures, not error/performance lines over training rounds. Under the conditions you laid, the test set must be out-of-sample, out-of-time to do accurate predictions.
Depends a lot on your data source and goal... As an classically trained statistician, I have dealt with some weird examples. CoD often causes trivial results in efficiency analysis (search for "curse of dimensionality" + "data envelopment analysis"). Data from standard agricultural experiments lives in a world with very small samples and a very large number of mandatory features (if those features aren't taken into account, the experiment is considered a failure), and data collection often takes decades, so you can't wait for more data points.
You are testing your car counting algorithm\model, not the exact estimated parameters or templates. To have a good performance estimate, your out-of-sample guys (the holdout) must be independent of your sample. If you separate pixels instead of images, you will have some form of dependence between pixel areas - probably a complex and hard to understand dependence.
All of these algorithms need some data labelled as zero and some as one with 100% centainty about the correctness of the label (or something very close to 100%). You have all ones, but you know that a small percentage of this data is mislabelled, a different situation. Without any knowledge about the domain of application, I would attack it using Anomaly Detection, then label the anomalies as zero. Then try some classification algorithm (One Class Learning, perhaps). With knowledge about the domain of application, I would seek help from a domain expert before anything.
