In every statistical test, we have two hypothesis as you state. But we only assume the Null Hypothesis and perform our statistical test under this assumption.
The p-value from the test doesn't tell us the probability of our data set or more extreme. It tell us the frequency of our data set (or more extreme) given many hypothetical replications of the test under our Null Hypothesis assumption.
Putting these two ideas together: We assume the means are equal (Null Hypothesis). Statistics tells how often our data set would appear (or more extreme) under this assumption if we ran many hypothetical experiments. If this data doesn't happen often, then we have evidence to reject the null hypothesis. Otherwise, we say that we don't have enough evidence to reject the null hypothesis.
A two-tailed test is used if we want to know one group is different (larger or smaller) than the other. A one-tailed test is used if we only want to know it is larger or if we only want to know it is smaller.
The t-test is a common and easy test. But it has some assumptions before the test can be applied. If the assumptions are not met, there are many alternatives like the Mann-Whitney U test that each come with their own assumptions.
When the t-test is done, we will get the t-statistics itself and corresponding p-value. We can also calculate a 95% confidence interval of the true difference of means. This interval does not say the true mean lies within this interval with 95% probability. The true mean is in the interval or it is not. There is no probability. Instead we say that we have a procedure that will contain the true difference 95% of the time if we hypothetically ran this experiment many times.
There are many links that describe this in many ways. Also, this explanation is the Frequentist approach instead of the Bayesian approach.