Yes.
We have a dummy variable regression in which (without loss of generality) the last category serves as the base case,
$$
y_i=\alpha+\sum_{j=1}^{p-1}\beta_jD_{ij}+u_i
$$
Let there be $n$ observations in total, with $n_j$ observations such that $D_{ij}=1$. We need to look at what happens to the formula for the variance of the regression coefficients, $\sigma^2(X'X)^{-1}$. Here, the regressor matrix $X$ has unit entries in the first column and another unit entry in column $j+1$ if observation $i$ belongs to group $j$ (unless it belongs to group $p$).
Consider $X'X$, which can be written as a block matrix
$$
X'X=\begin{pmatrix}
A&B\\
B'&D
\end{pmatrix}
$$
where $A=n$, $B=(n_1,\cdots,n_{p-1})$ and $D$ a diagonal matrix with main diagonal $B$. This follows by direct multiplication, exploiting that no row of $X$ has more than one entry equal to one (except for the constant column)
To obtain $(X'X)^{-1}$, we use the formula for block inverses,
\begin{align}
(X'X)^{-1} &= \begin{pmatrix}
\hspace{2cm}A &\hspace{4.3cm}B\\
\hspace{2cm}B' &\hspace{7cm}D
\hspace{2.8cm}\end{pmatrix}^{-1} \\[5pt]
&=\begin{pmatrix}
(A-BD^{-1}B')^{-1}&-(A-BD^{-1}B')^{-1}BD^{-1}\\
-D^{-1}B'(A-BD^{-1}B')^{-1}&D^{-1}+D^{-1}B'(A-BD^{-1}B')^{-1}BD^{-1}
\end{pmatrix}
\end{align}
The inverse of the diagonal matrix $D$ simply is a diagonal matrix with entries $1/n_j$. Direct multiplication then yields
$$
(A-BD^{-1}B')^{-1}=\left(n-\sum_{j=1}^{p-1}n_j\right)^{-1}=\frac{1}{n_p}
$$
Further, $BD^{-1}=\iota'$, a unit row vector, and hence $D^{-1}B'=\iota$. Putting things together gives
\begin{align}
(X'X)^{-1} &=\begin{pmatrix}
\hspace{.5cm}A &\hspace{.7cm}B\\
\hspace{.5cm}B' &\hspace{1.5cm}D
\hspace{.8cm}\end{pmatrix}^{-1} \\[5pt]
&=\begin{pmatrix}
\frac{1}{n_p}&-\frac{1}{n_p}\iota'\\
-\frac{1}{n_p}\iota'&D^{-1}+\frac{1}{n_p}\iota\iota'
\end{pmatrix}
\end{align}
This means that all off-diagonal elements of the variance-covariance matrix only depend on $1/n_p$, which you know from the first squared standard error.
In fact, the derivation hence shows a little more than what you asked: to get the off-diagonal elements, you do not even need to all variances, but only that of the base category.
Here is a little numerical illustration.
n <- 100
y <- rnorm(n) # this is the dependent variable
p <- 5
X <- matrix(0, nrow=n, ncol=p)
X[cbind(1:n, sample(1:p, n, replace=T))] <- 1 # insert an 1 into one of the columns for each row
reg3 <- summary(lm(y~X[,1:(p-1)])) # regression omitting the pth category
vcov(reg3)
(Intercept) X[, 1:(p - 1)]1 X[, 1:(p - 1)]2 X[, 1:(p - 1)]3 X[, 1:(p - 1)]4
(Intercept) 0.05463828 -0.05463828 -0.05463828 -0.05463828 -0.05463828
X[, 1:(p - 1)]1 -0.05463828 0.14440116 0.05463828 0.05463828 0.05463828
X[, 1:(p - 1)]2 -0.05463828 0.05463828 0.13318080 0.05463828 0.05463828
X[, 1:(p - 1)]3 -0.05463828 0.05463828 0.05463828 0.10927656 0.05463828
X[, 1:(p - 1)]4 -0.05463828 0.05463828 0.05463828 0.05463828 0.10699996
