One way to ensure this is to make sure you have coded up all of the things you do to fit the model, even "tinkering". This way, when you run the process repeatedly, say via cross-validation, you are keeping things consistent between runs. This ensures that all of the potential sources of variation are captured by the cross-validation process.
The other vitally important thing is to ensure that you have a representative sample, in both data sets. If your data set is not representative of the kind of data you expect to be using to predict, then there is not much that you can do. All modelling rests on an assumption that "induction" works - the things we haven't observed behave like the things we have observed.
As a general rule, stay away from complex model fitting procedures unless (i) you know what you are doing, and (ii) you have tried the simpler methods, and found that they don't work, and how the complex method fixes the problems with the simple method. "Simple" and "complex" are meant in the sense of "simple" or "complex" to the person doing the fitting. The reason this is so important is that it allows you to apply what I like to call a "sniff test" to the results. Does the result look right? You can't "smell" the results from a procedure that you don't understand.
NOTE: the next, rather long part of my answer is based on my experience, which is in the $N>>p$ area, with $p$ possibly large. I am almost certain that what follows below would not apply to the $N\approx p$ or $N<p$ cases
When you have a large sample, the difference between using and not using a given observation is very small, provided your modelling is not too "local". This is because the influence of a given data point is generally the order of $\frac{1}{N}$. So in large data sets, the residuals you get from "holding out" the test data set are basically the same as the residuals you get from using it in the training data set. You can show this using ordinary least squares. The residual you get from excluding the $i$th observation (i.e. what the test set error would be if we put the observation in the test set) is $e_i^{test}=(1-h_{ii})^{-1}e_i^\mathrm{train}$, where $e_i^\mathrm{train}$ is the training residual, and $h_{ii}$ is the leverage of the $i$th data point. Now we have that $\sum_ih_{ii}=p$, where $p$ is the number of variables in the regression. Now if $N>>p$, then it is extremely difficult for any $h_{ii}$ to be large enough to make an appreciable difference between the test set and training set errors. We can take a simplified example, suppose $p=2$ (intercept and $1$ variable), $N\times p$ design matrix is $X$ (both training and testing sets), and the leverage is
$$h_{ii}=x_i^T(X^TX)^{-1}x_i=\frac{1}{Ns_x^2}
\begin{pmatrix}1 & x_i \end{pmatrix}
\begin{pmatrix}\overline{x^2} & -\overline{x}\\ -\overline{x} & 1\end{pmatrix}
\begin{pmatrix}1 \\ x_i\end{pmatrix}
=\frac{1+\tilde{x}_i^2}{N}$$
Where $\overline{x}=N^{-1}\sum_ix_i$, $\overline{x^2}=N^{-1}\sum_ix_i^2$, and $s_x^2=\overline{x^2}-\overline{x}^2$. Finally, $\tilde{x}_i=\frac{x_i-\overline{x}}{s_x}$ is the standardised predictor variable, and measures how many standard deviations $x_i$ is from the mean. So, we know from the beginning that the test set error will be much larger than the training set error for observations "at the edge" of the training set. But this is basically that representative issue again - observations "at the edge" are less representative than observations "in the middle". Additionally, this is to order $\frac{1}{N}$. So if you have $100$ observations, even if $\tilde{x}_i=5$ (an outlier in x-space by most definitions), this means $h_{ii}=\frac{26}{100}$, and the test error is understated by a factor of just $1-\frac{26}{100}=\frac{74}{100}$. If you have a large data set, say $10000$, it is even smaller,$1-\frac{26}{10000}$, which is less than $1\text{%}$. In fact, for $10000$ observations, you would require an observation of $\tilde{x}=50$ in order to make a $25\text{%}$ under-estimate of the test set error, using the training set error.
So for big data sets, using a test set is not only inefficient, it is also unnecessary, so long as $N>>p$. This applies for OLS and also approximately applies for GLMs (details are different for GLM, but the general conclusion is the same). In more than $2$ dimensions, the "outliers" are defined by the observations with large "principal component" scores. This can be shown by writing $h_{ii}=x_i^TEE^T(X^TX)^{-1}EE^Tx_i$ Where $E$ is the (orthogonal) eigenvector matrix for $X^TX$, with eigenvalue matrix $\Lambda$. We get $h_{ii}=z_i^T\Lambda^{-1}z_i=\sum_{j=1}^p\frac{z_{ji}^2}{\Lambda_{jj}}$ where $z_i=E^Tx_i$ is the principal component scores for $x_i$.
If your test set has $k$ observations, you get a matrix version ${\bf{e}}_{\{k\}}^\mathrm{test}=(I_k-H_{\{k\}})^{-1}{\bf{e}}_{\{k\}}^\mathrm{train}$, where $H_{\{k\}}=X_{\{k\}}(X^TX)^{-1}X_{\{k\}}^T$ and $X_{\{k\}}$ is the rows of the design matrix in the test set. So, for OLS regression, you already know what the "test set" errors would have been for all possible splits of the data into training and testing sets. In this case ($N>>p$), there is no need to split the data at all. You can report "best case" and "worst case" test set errors of almost any size without actually having to split the data. This can save a lot of PC time and resources.
Basically, this all reduces to using a penalty term, to account for the difference between training and testing errors, such as BIC or AIC. This effectively achieves the same result as what using a test set does, however you aren't forced to throw away potentially useful information. With the BIC, you are approximating the evidence for the model, which looks mathematically like:
$$p(D|M_iI)=p(y_1y_2\dots y_N|M_iI)$$
Note that in this procedure, we cannot estimate any internal parameters - each model $M_i$ must be fully specified or have its internal parameters integrated out. However, we can make this look like cross validation (using a specific loss function) by repeatedly using the product rule, and then taking the log of the result:
$$p(D|M_iI)=p(y_1|M_iI)p(y_2\dots y_N|y_1M_iI)$$
$$=p(y_1|M_iI)p(y_2|y_1M_iI)p(y_3\dots y_N|y_1y_2M_iI)$$
$$=\dots=\prod_{i=1}^{N}p(y_i|y_1\dots y_{i-1}M_iI)$$
$$\implies\log\left[p(D|M_iI)\right]=\sum_{i=1}^{N}\log\left[p(y_i|y_1\dots y_{i-1}M_iI)\right]$$
This suggests a form of cross validation, but where the training set is constantly being updated, one observation at a time from the test set - similar to the Kalman Filter. We predict the next observation from the test set using the current training set, measure the deviation from the observed value using the conditional log-likelihood, and then update the training set to include the new observation. But note that this procedure fully digests all of the available data, while at the same time making sure that every observation is tested as an "out-of-sample" case. It is also invariant, in that it does not matter what you call "observation 1" or "observation 10"; the result is the same (calculations may be easier for some permutations than others). The loss function is also "adaptive" in that if we define $L_i=\log\left[p(y_i|y_1\dots y_{i-1}M_iI)\right]$, then the sharpness of $L_i$ depends on $i$, because the loss function is constantly being updated with new data.
I would suggest that assessing predictive models this way would work quite well.
