Best confidence interval for sample mean of any variable in L2

I was thinking about a very basic matter, and arrived at the conclusion I know a lot less about it than I thought.

When people have data $$X_1^N$$ sampled i.i.d. from a distribution: $$X_1^N \sim X$$, it is customary to estimate the mean by $$\bar{X} = \frac{1}{N} \sum_{i=1}^N X_i$$ and to estimate the confidence interval for the true mean $$\mu$$ in the 95% confidence interval:

$$\mu \in \left[\bar{X} - 1.96 \cdot \frac{\hat{std}(X)}{\sqrt{N}}, \bar{X} + 1.96 \cdot \frac{\hat{std}(X)}{\sqrt{N}}\right]$$

Where $$\hat{std}(X)$$ is some estimate for the standard deviation of X.

Now, this comes from an assumption that X follows a normal distribution. Otherwise, this confidence interval is only valid asymptotically.

But, if the variable $$X$$ belongs to $$\mathcal{L}^2$$, that is, if the standard deviation exists and is finite, could we derive a precise, non-asymptotic interval?

The answer is yes. By Markov inequality,

$$\mathbf{P} \left[ \left| \frac{1}{N} \sum_{i=1}^N X_i - \mu \right| \gt \epsilon \right] \leq \frac{ \mathbf{P} \left[ \left( \frac{1}{N} \sum_{i=1}^N X_i - \mu \right)^2 \right] }{\epsilon^2}$$ $$= \frac{Var(X)}{N \epsilon^2}$$

So if we want a 95% confidence interval, (that is, $$\mathbf{P} \left[ \left| \frac{1}{N} \sum_{i=1}^N X_i - \mu \right| \gt \epsilon \right] \leq 0.05$$), we might take:

$$\epsilon = \sqrt{ \frac{Var(X)}{0.05 N}} \sim 4.47 \frac{std(X)}{\sqrt{N}}$$

So, if we want an exact (except by the error in $$\hat{std(X)}$$), non-asymptotic interval for $$\mu$$, instead of the one above, we should do:

$$\mu \in \left[\bar{X} -4.47 \cdot \frac{\hat{std}(X)}{\sqrt{N}}, \bar{X} + 4.47 \cdot \frac{\hat{std}(X)}{\sqrt{N}}\right]$$

Is it a "bad" non-asymptotic confidence interval? That is, is it possible to find a better one, valid for any distribution in $$\mathcal{L}^2$$?

I imagine there could be a multiplicative factor that is a function of N, such that the number 4.47 progressively turns into 1.96 while N grows. But, for a variable in $$\mathcal{L}^2$$, I am not aware of any better bound than the one coming from Markov's inequality.

Thanks!