I will also add to Jeremy Miles' answer that makes a number of valid points. (I wrote initially that "I disagree with the claim that the reason that statistical significance testing doesn't seem to make sense is that, essentially, it doesn't", but Jeremy has made this more precise in the meantime.)
The p-value has a well defined mathematical meaning, which is the probability that given the null hypothesis is true, the test statistics is as far or farther away than what was observed from what is expected under the null hypothesis.
Now in language we are bound to be categorical, so we lack the words for saying how a p-value of 0.064 is different from 0.059, however we can talk for example about strong, weak, no evidence against the $H_0$ meaning here that something has happened that under the $H_0$ would happen very rarely/rarely/be very common.
If we want to make decisions (like going on to work as if the $H_0$ were true, or not true, if these two possibilities were the only ones to consider, i.e., we decide between only two courses of action) of course we need to decide how small is too small to stick with the $H_0$.
Now there are conventions like the ubiquitous $\alpha=0.05$. This is arbitrary to some extent, but you need to realise that specifying such a cutoff is necessary when making a binary decision - otherwise not (this by the way holds as well for any alternative to significance testing).
The interpretation in language of a p-value doesn't have a mathematical basis and will always be to some extent arbitrary, but once more, language is categorical, so if we use a finite number of different wordings, we are more transparent when having well defined cutoff values for them. With all the problems with which cutoff values come - for example if you put the cutoff between "very weak evidence" and "no evidence" at 0.1, you will in language distinguish between 0.99 and 0.101, but not between 0.101 and 0.103. That may not look particularly appropriate but somehow lies in the nature of the problem.
Overall you have some freedom as at least mathematics doesn't determine how you should say things, however you act more professionally when sticking to at least fairly generally accepted and transparent standards.
So using "reject/not reject" with a cutoff at 0.05 (or 0.01 in some fields) is a strong reduction of information and as such problematic, however people cannot accuse you of bending the evidence as you stick to a well established rule. That's something. It is important here though that you only really need to use this kind of binary distinction if a binary action decision is to be made. (Deciding to "believe" the $H_0$ or the alternative is in my view not a direct action, and models should not ever be "believed" in my view anyway.) Still then for those who like precision it can only be good to state the precise p-value.
It is also generally accepted to use "evidence language", although borderlines are not handled consistently all over the place. I don't think you can go very wrong by using 0.01/0.05/0.1 as cutoffs for saying there's "strong"/"some"/"weak"/"no" evidence against the $H_0$, but I have seen others. Also let's always keep in mind that categorisation is to some extent arbitrary, but also a necessity when people communicate in language.
It is also OK to differentiate even more by saying something like "There's some evidence as $p<0.05$ but it's really rather weak as in fact $p=0.046$." Or even "at $p=0.07$ we only have weak evidence against the $H_0$ but some doubt about it is justified and the effect estimator (...) is in fact quite large" (in case it is, in a subject matter sense).
Some wordings are clearly misleading such as "tends toward significance" (as (a) there's no "tending" and (b) the writer reveals implicitly that they would've wanted significance and are willing to sacrifice objectivity to make a certain impression) or "accepting" the $H_0$ (as accepting is routinely misunderstood as thinking it is true, which no p-value can ever tell you, and "all models are wrong" anyway).
PS: "Accept" can be appropriate wording in quality control applications where a batch of products may be "accepted" if a sample does not provide evidence against certain quality standards ("acceptance sampling"). The $\alpha$ in such (and some other) applications is best chosen taking into account considerations regarding costs of consequences, rather than using widespread defaults. Note also that the major benefit of the Neyman-Pearson setup is to characterise tests by error probabilities and to enable optimality theory (finding tests that have optimal power given the level). This does not mean that such tests in practice have to be interpreted exclusively in a binary reject/not reject manner; it does not "overwrite" the more precise information in the p-value.