You have to be very specific about what you mean. We can show mathematically that a certain neural network architecture trained with a certain loss coincides exactly with logistic regression at the optimal parameters. Other neural networks will not.

A binary logistic regression makes predictions $\hat{y}$ using this equation:
$$
\hat{y}=\sigma(X \beta + \beta_0)
$$
where $X$ is a $n \times p$ matrix of features (predictors, independent variables) and vector $\beta$ is the vector of $p$ coefficients and $\beta_0$ is the intercept and $\sigma(z)=\frac{1}{\exp(-z)+1}$. Conventionally in a logistic regression, we would roll the $\beta_0$ scalar into the vector $\beta$ and append a column of 1s to $X$, but I've moved it out of $\beta$ for clarity of exposition.

A neural network with one output neuron with a sigmoid activation makes predictions using the equation
$$
\hat{y}=\sigma(X \beta + \beta_0)
$$
with $\hat{y},\sigma,X, \beta, \beta_0$ as before. Clearly, the equation is exactly the same. In the [tag:neural-networks] literature, $\beta_0$ is usually called a "bias," even though it has nothing to do with the statistical concept of [tag:bias]. Otherwise, the terminology is identical.

A logistic regression has the Bernoulli likelihood as its objective function, or, equivalently, the Bernoulli log-likelihood function. This objective function is **maximized**:
$$
\arg\max_{\beta,\beta_0} \sum_i y_i \log(\hat{y_i}) + (1-y_i)\log(1-\hat{y_i})
$$ where $y \in \{0,1\}$.
We can motivate this objective function from a Bernoulli probability model where the probability of success depends on $X$.

A neural network can, in principle, use any loss function we like. It might use the so-called "cross-entropy" function (even though the "cross-entropy" can motivate any number of loss functions; see https://stats.stackexchange.com/questions/378274/how-to-construct-a-cross-entropy-loss-for-general-regression-targets), in which case the model **minimizes** this loss function:
$$
\arg\min_{\beta,\beta_0} - \sum_i y_i \log(\hat{y_i}) + (1-y_i)\log(1-\hat{y_i})
$$

In both cases, these objective functions are strictly convex (concave) when certain conditions are met. Strict convexity implies that there is a single minimum and that this minimum is a global. Moreover, the objective functions are identical, since *minimizing* a strictly convex function $f$ is equivalent to *maximizing* $-f$. Therefore, these two models recover the same parameter estimates $\beta, \beta_0$. As long as the model attains the single optimum, it doesn't matter what optimizer is used, because there is only one optimum for these specific models.

However, a neural network is not required to optimize this specific loss function; for instance, a [tag:triplet-loss] for this same model would likely recover different estimates $\beta,\beta_0$. And the MSE/least squares loss is not convex in this problem, so that neural network would differ from logistic regression as well (see: https://stats.stackexchange.com/questions/326350/what-is-happening-here-when-i-use-squared-loss-in-logistic-regression-setting).