Yes and no.
First the "yes"
What you've observed is that when a test and a confidence interval is based on the same statistic, there is an equivalence between them: we can interpret the $p$-value as the smallest value of $\alpha$ for which the null value of the parameter would be included in the $1-\alpha$ confidence interval.
Let $\theta$ be an unknown parameter in the parameter space $\Theta\subseteq\mathbb{R}$, and let the sample $\mathbf{x}=(x_1,\ldots,x_n)\in\mathcal{X}^ n\subseteq\mathbb{R}^n$ be a realization of the random variable $\mathbf{X}=(X_1,\ldots,X_n)$. For simplicity, define a confidence interval $I_\alpha(\mathbf{X})$ as a random interval such that its coverage probability
$$
P_\theta(\theta\in I_\alpha(\mathbf{X}))= 1-\alpha\qquad\mbox{for all }\alpha\in(0,1).
$$
(You could similarly consider more general intervals, where the coverage probability either is bounded by or approximately equal to $1-\alpha$. The reasoning is analogous.)
Consider a two-sided test of the point-null hypothesis $H_0(\theta_0): \theta=\theta_0$ against the alternative $H_1(\theta_0): \theta\neq \theta_0$. Let $\lambda(\theta_0,\mathbf{x})$ denote the p-value of the test. For any $\alpha\in(0,1)$, $H_0(\theta_0)$ is rejected at the level $\alpha$ if $\lambda(\theta_0,x)\leq\alpha$. The level $\alpha$ rejection region is the set of $\mathbf{x}$ which lead to the rejection of $H_0(\theta_0)$:
$$ R_\alpha(\theta_0)=\{\mathbf{x}\in\mathbb{R}^n: \lambda(\theta_0,\mathbf{x})\leq\alpha\}.$$
Now, consider a family of two-sided tests with p-values $\lambda(\theta,\mathbf{x})$, for $\theta\in\Theta$. For such a family we can define an inverted rejection region
$$ Q_\alpha(\mathbf{x})=\{\theta\in\Theta: \lambda(\theta,\mathbf{x})\leq\alpha\}.$$
For any fixed $\theta_0$, $H_0(\theta_0)$ is rejected if $\mathbf{x}\in R_\alpha(\theta_0)$, which happens if and only if $\theta_0\in Q_\alpha(\mathbf{x})$, that is,
$$
\mathbf{x}\in R_\alpha(\theta_0) \Leftrightarrow \theta_0\in Q_\alpha(\mathbf{x}).
$$
If the test is based on a test statistic with a completely specified absolutely continuous null distribution, then $\lambda(\theta_0,\mathbf{X})\sim \mbox{U}(0,1)$ under $H_0(\theta_0)$. Then
$$
P_{\theta_0}(\mathbf{X}\in R_\alpha(\theta_0))=P_{\theta_0}(\lambda(\theta_0,\mathbf{X})\leq\alpha)=\alpha.
$$
Since this equation holds for any $\theta_0\in\Theta$ and since the equation above it implies that $$P_{\theta_0}(\mathbf{X}\in R_\alpha(\theta_0))=P_{\theta_0}(\theta_0\in Q_\alpha(\mathbf{X})),$$ it follows that the random set $Q_\alpha(\mathbf{x})$ always covers the true parameter $\theta_0$ with probability $\alpha$. Consequently, letting $Q_\alpha^C(\mathbf{x})$ denote the complement of $Q_\alpha(\mathbf{x})$, for all $\theta_0\in\Theta$ we have
$$P_{\theta_0}(\theta_0\in Q_\alpha^C(\mathbf{X}))=1-\alpha,$$
meaning that the complement of the inverted rejection region is a $1-\alpha$ confidence interval for $\theta$.
An illustration is given below, showing rejection regions and confidence intervals corresponding to the the $z$-test for a normal mean, for different null means $\theta$ and different sample means $\bar{x}$, with $\sigma=1$. $H_0(\theta)$ is rejected if $(\bar{x},\theta)$ is in the shaded light grey region. Shown in dark grey is the rejection region $R_{0.05}(-0.9)=(-\infty,-1.52)\cup(-0.281,\infty)$ and the confidence interval $I_{0.05}(1/2)=Q_{0.05}^C(1/2)=(-0.120,1.120)$.

(Much of this is taken from my PhD thesis.)
Now for the "no"
Above I described the standard way of constructing confidence intervals. In this approach, we use some statistic related to the unknown parameter $\theta$ to construct the interval. There are also intervals based on minimization algorithms, which seek to minimize the length of the interval condition on the value of $X$. Usually, such intervals do not correspond to a test.
This phenomenon has to do with problems related to such intervals not being nested, meaning that the 94 % interval can be shorter than the 95 % interval. For more on this, see Section 2.5 of this recent paper of mine (to appear in Bernoulli).
And a second "no"
In some problems, the standard confidence interval is not based on the same statistic as the standard test (as discussed by Michael Fay in this paper). In those cases, confidence intervals and tests may not give the same results. For instance, $\theta_0=0$ may be rejected by the test even though 0 is included in the confidence interval. This does not contradict the "yes" above, as different statistics are used.
And sometimes "yes" is not a good thing
As pointed out by f coppens in a comment, sometimes intervals and tests have somewhat conflicting goals. We want short intervals and tests with high power, but the shortest interval does not always correspond to the test with the highest power. For some examples of this, see this paper (multivariate normal distribution), or this (exponential distribution), or Section 4 of my thesis.
Bayesians can also say both yes and no
Some years ago, I posted a question here about whether a test-interval-equivalence exists also in Bayesian statistics. The short answer is that using standard Bayesian hypothesis testing, the answer is "no". By reformulating the testing problem a little bit, the answer can however be "yes". (My attempts at answering my own question eventually turned into a paper!)