There are various reasons why a test outcome may be unreliable. Some of these can be diagnosed from the data such as certain model assumption violations. You are also encouraged to look at a confidence interval of the difference, as this will tell you something about the precision of results (assuming the model). But it seems you are concerned here in the first place about problems with your measurements.
Now this is an issue that the data themselves cannot address, as it refers to potential deviations of what you actually measured from what you want to measure. The latter is not observed in your data, so the data themselves will not serve to find out about this. Generally measurement validity and reliability are hard to address, as very often the measurements are chosen because better alternatives don't seem to be available. What you need is a theory regarding the relation of measurements to their actual target. Test-retest reliability is useful if you can run retests under conditions under which your theory suggests that the "true" results (your measurement target) should be the same. It depends on the specific problem/study whether this makes sense. From how I read your question, this may apply to measuring twice without the "manipulation" the effect of which you're interested in, but I can't tell from the given information.
You can also use alternative measurements that are supposed to measure the same concept or to be in a specific relation to the concept in order to check validity if such measurements exist, however the basic difficulty is always that the ultimate question is how your measurements relate to an assumed underlying true value that cannot directly be observed. Apart from empirical investigation, properly thinking through the measurement, how it is related to the concept of interest, and potential issues that may invalidate the measurement, is also important, plus considering the related literature for thoughts and experiences.
