The short answer is: Don't drop the KL term.
The reconstruction error plus KL term optimized by a VAE is a lower bound on the log-likelihood (also called the "evidence lower bound", or ELBO) . Log-likelihood is one way to measure how well your model explains the data. If that's what you're after, it makes sense to try to evaluate the log-likelihood. This is not straight-forward, but possible .
You can use the ELBO as a conservative estimate of the log-likelihood. It therefore makes sense to use reconstruction error plus KL term as your
Ask yourself why you are training a variational autoencoder (VAE). If you can answer this question, the right way to evaluate (and train) your model will become much clearer. Is the reconstruction error important for your application? If yes, then reconstruction error would be okay to use for validation, but then I would question why you are optimizing for log-likelihood.
 Kingma & Welling, Auto-Encoding Variational Bayes, 2014 (Equation 2)
 Wu et al., On the Quantitative Analysis of Decoder Based Generative Models, 2017