You are pretty close to your goal already, but you should replace $x$s with $\epsilon$s in the conditional variance equation. Note also that VAR is a multivariate model, i.e. the dependent variable is a vector rather than a scalar. Then the GARCH model needs to be multivariate as well because it needs to characterize the conditional covariance matrix of a vector of errors.
So for VAR($m$)-GARCH($p,q$) you have
$$
\begin{aligned}
x_t &= \phi_0 + \sum_{i=1}^m \phi_i x_{t-i} + \epsilon_t \\
\epsilon_t &= \Sigma^{1/2}_t u_t \\
\Sigma_t &= \alpha_0 + \sum_{l=1}^{p} \alpha_l \Sigma_{t-l} + \sum_{m=1}^{q} \beta_m \epsilon_{t-m}\epsilon'_{t-m} \\
u_t &\sim i.i.d(0,I)
\end{aligned}
$$
where $x_t$ and $\epsilon_t$ are vectors; $\phi_0$ and $\alpha_0$ are parameter vectors; $\Sigma_t$ is the conditional variance matrix of $\epsilon_t$; $I$ is an identity matrix; and $\phi_1$ through $\phi_m$, $\alpha_1$ through $\alpha_p$, and $\beta_1$ through $\beta_q$ are parameter matrices. Some restrictions in the equation of $\Sigma_t$ are needed to ensure $\Sigma_t$ is a valid covariance matrix (positive definite). Also, the equation for $\Sigma_t$ could be replaced by an equation for vech-torized $\Sigma_t$ (stacking the columns below and including the diagonal) to avoid having a matrix as a dependent variable; that would change the representation but not the basic idea.
I was trying to stick to your notation as closely as possible; you would encounter somewhat different notation in time series textbooks (e.g. using capital letters for matrices, vech-torizing $\Sigma_t$, etc.).
