You've gotten some good information from @Macro. Let me say a few general things, and hopefully something may be of help to you.
First of all, your response variable is "the time it takes to reach a goal". This is time to event data (also called survival, duration, failure time, etc.). There are two big issues with time to event data: the first is that you almost always have censoring. One example would be that your study stopped before everyone reached the goal, and so you only know that it took some people longer than some duration $t$ (where $t$ represents, e.g., the total elapsed time up until the end of your study). Do you have censoring? If so, you will need to use survival analysis methods, not the t-test. The second is that event time data are rarely adequately normal. Are your data sufficiently normal? Even if you don't have censoring, it may be preferable to use survival analysis, or to use a non-parametric approach (such as the Mann-Whitney U-test), or to transform your response data to achieve normality.
The other issue that sticks out for me is that you say that "it is so that some proportion of the times the second sample are not delayed". How do you know this? Is it because the distributions overlap? (That would not provide evidence for that proposition.) Based on your description and the conversation in the comments, I cannot see how you could know that only some of the times are delayed, but not others. I see this issue as an example of a common misconception. For instance, when hurricane Katrina hit the US Gulf coast, there was a lot of discussion in the media about global warming causing more severe storms, and whether global warming caused Katrina, or whether it had caused other storms, but not that one. Such discussions indicate a misunderstanding of distributions and what the data tell us. Typically, if we have two distributions with similar variances but different means, e.g., $\mathcal N(4, 3)~\&~\mathcal N(6, 3)$, it's best to understand that a constant (namely, $2$) has been added to every value in the first distribution to create the shifted distribution. Of course, the world is always more complicated than your theory, but this is nonetheless the best way to think about this situation. (Note that this would not necessarily be true if the variances differ.)
I believe your question is more related to the second set of issues, and that they are what lie behind some of @Macro's comments, but I suggest you take the first set of issues seriously as well in thinking about your project.