Return to Question

1. The last sentence of Christopher M. Bishop, Pattern Recognition and Machine Learning Section 10.1.1 Factorized distributions on p.466, states, referring to Equation $(10.9)$, that "Convergence is guaranteed because bound is convex with respect to each of the factors $q_i(Z_i)$ (Stephen Boyd and Lleven Vandenberghe (2004), Convex Optimization. Cambridge University Press)."
2. The same claim is made in the Wikipedia entry, Variational Bayesian Methods Section 2. Mean field Approximation with the same reference to the Boyd & Vandenberghe book above: "An algorithm of this sort is guaranteed to converge."

I see that $q^*_j(Z_j)$ minimizes $\mathcal L(q)$ and thus $-\mathcal L(q^*)$ is nonincreasing with respect to the iteration on $q^*$. However, I do not see how the convexity of $-\mathcal L$ with respect to $q$ leads to the convergence of $q^*$. On the contrary, I can imagine counterexamples satisfying the conditions of decreasing functional value and convexity of the function. Neither do I know where in the referenced Boyd book to find the proof. Is there a fixed-point theorem to use?

1. The last sentence of Christopher M. Bishop, Pattern Recognition and Machine Learning Section 10.1.1 Factorized distributions on p.466, states, referring to Equation $(10.9)$, that "Convergence is guaranteed because bound is convex with respect to each of the factors $q_i(Z_i)$ (Stephen Boyd and Lleven Vandenberghe (2004), Convex Optimization. Cambridge University Press)."
2. The same claim is made in the Wikipedia entry, Variational Bayesian Methods Section 2. Mean field Approximation with the same reference to the Boyd & Vandenberghe book above: "An algorithm of this sort is guaranteed to converge."

I see that $q^*_j(Z_j)$ minimizes $-\mathcal L(q)$ and thus $-\mathcal L(q^*)$ is nonincreasing with respect to the iteration on $q^*$. However, I do not see how the convexity of $-\mathcal L$ with respect to $q$ leads to the convergence of $q^*$. On the contrary, I can imagine counterexamples satisfying the conditions of decreasing functional value and convexity of the function. Neither do I know where in the referenced Boyd book to find the proof. Is there a fixed-point theorem to use?

1. The last sentence of Christopher M. Bishop, Pattern Recognition and Machine Learning Section 10.1.1 Factorized distributions on p.466, states, referring to Equation $(10.9)$, that "Convergence is guaranteed because bound is convex with respect to each of the factors $q_i(Z_i)$ (Stephen Boyd and Lleven Vandenberghe (2004), Convex Optimization. Cambridge University Press)."
2. The same claim is made in the Wikipedia entry, Variational Bayesian Methods Section 2. Mean field Approximation with the same reference to the Boyd & Vandenberghe book above: "An algorithm of this sort is guaranteed to converge."

I see that $q^*_j(Z_j)$ minimizes $\mathcal L(q)$ and thus $-\mathcal L(q^*)$ is nonincreasing with respect to the iteration on $q^*$. However, I do not see how the convexity of $-\mathcal L$ with respect to $q$ leads to the convergence of $q^*$. On the contrary, I can imagine counterexamples satisfying the conditions of decreasing functional value and convexity of the function. Neither do I know where in the referenced Boyd book to find the proof. Is there a fixed-point theorem to use?

1. The last sentence of Christopher M. Bishop, Pattern Recognition and Machine Learning Section 10.1.1 Factorized distributions on p.466, states, referring to Equation $(10.9)$, that "Convergence is guaranteed because bound is convex with respect to each of the factors $q_i(Z_i)$ (Stephen Boyd and Lleven Vandenberghe (2004), Convex Optimization. Cambridge University Press)."
2. The same claim is made in the Wikipedia entry, Variational Bayesian Methods Section 2. Mean field Approximation with the same reference to the Boyd & Vandenberghe book above: "An algorithm of this sort is guaranteed to converge."

I see that $q^*_j(Z_j)$ minimizes $\mathcal L(q)$ and thus $-\mathcal L(q^*)$ is nonincreasing with respect to the iteration on $q^*$. However, I do not see how the convexity of $-\mathcal L$ with respect to $q$ leads to the convergence of $q^*$. On the contrary, I can imagine counterexamples satisfying the conditions of decreasing functional value and convexity of the function. Neither do I know where in the referenced Boyd book to find the proof. Is there a fixed-point theorem to use?

1. The last sentence of Christopher M. Bishop, Pattern Recognition and Machine Learning Section 10.1.1 Factorized distributions, states referring to Equation $(10.9)$, that "Convergence is guaranteed because bound is convex with respect to each of the factors $q_i(Z_i)$ (Stephen Boyd and Lleven Vandenberghe (2004), Convex Optimization. Cambridge University Press)."
2. The same claim is made in the Wikipedia entry, Variational Bayesian Methods Section 2. Mean field Approximation with the same reference to the Boyd & Vandenberghe book above: "An algorithm of this sort is guaranteed to converge."

I see that $q^*_j(Z_j)$ minimizes $\mathcal L(q)$ and thus $-\mathcal L(q^*)$ is nonincreasing with respect to the iteration on $q^*$. However, I do not see how the convexity of $-\mathcal L$ with respect to $q$ leads to the convergence of $q^*$. On the contrary, I can imagine counterexamples satisfying the conditions of decreasing functional value and convexity of the function. Neither do I know where in the referenced Boyd book to find the proof. Is there a fixed-point theorem to use?

1. The last sentence of Christopher M. Bishop, Pattern Recognition and Machine Learning Section 10.1.1 Factorized distributions on p.466, states, referring to Equation $(10.9)$, that "Convergence is guaranteed because bound is convex with respect to each of the factors $q_i(Z_i)$ (Stephen Boyd and Lleven Vandenberghe (2004), Convex Optimization. Cambridge University Press)."
2. The same claim is made in the Wikipedia entry, Variational Bayesian Methods Section 2. Mean field Approximation with the same reference to the Boyd & Vandenberghe book above: "An algorithm of this sort is guaranteed to converge."

I see that $q^*_j(Z_j)$ minimizes $\mathcal L(q)$ and thus $-\mathcal L(q^*)$ is nonincreasing with respect to the iteration on $q^*$. However, I do not see how the convexity of $-\mathcal L$ with respect to $q$ leads to the convergence of $q^*$. On the contrary, I can imagine counterexamples satisfying the conditions of decreasing functional value and convexity of the function. Neither do I know where in the referenced Boyd book to find the proof. Is there a fixed-point theorem to use?