# Questions tagged [lagrange-multipliers]

The method of Lagrange multipliers finds critical points (including maxima and minima) of a differentiable function subject to differentiable constraints.

69 questions
Filter by
Sorted by
Tagged with
67 views

### Estimating the parameter of a Bernoulli distribution using probabilistic modeling and the MAP estimation

Suppose you tossed a coin multiple times. Sometimes you got heads and other times you got tails. You recorded your experiment in a dataset $X$. Now you want to estimate the parameter θ (which ...
1 vote
29 views

### Why is there only a box constraint on alpha and not on mu when solving the dual problem of soft linear SVM?

I am currently learning about the linear SVM in the non-separable case. In the dual representation, we introduce the Lagrange multipliers μk and αk (see also this source: https://...
1 vote
19 views

### Are solutions to the Lagrangian multipliers ($\alpha_i$) in a hard-margin SVM unique?

An intermediate step in the derivation of the hard-margin SVM's dual form is as follows: I also know that $a_i$ for all points not on the margin boundary is 0, which makes sense; they must be zeroed ...
15 views

### The distribution of a score test statistic in a special case

Suppose that we want to test a restriction $\alpha=0$. Unfortunately, we cannot directly estimate the parameter of interest $\alpha$. Instead, we can calculate the log-likelihood functions with and ...
1 vote
16 views

### Min max formulation conversion to max min formulation. Reason?

Question is based on the screenshot attached. Based on paper here. I am not being able to understand why min max formulation (eq 4) is first converted to max min formulation (eq 5). Is it something ...
21 views

### Subspace test for multivariate normal distribution [duplicate]

Subspace test for multivariate normal distribution Suppose $X_1, X_2,\ldots, X_n$ are i.i.d. observations from a multivariate normal distribution $N(\mu,\Sigma)$ where $\Sigma$ is known. Furthermore, ...
1 vote
53 views

### Can we convert the optimization of a loss function with regularization to the Lagrangian, constrained optimization *before* solving the optimization?

It is shown here that the optimization of a loss function with regularization, $$\text{argmin}_b L(X,b) + c ||b||_p \phantom{aaaaaaaaaaaaaaaaaaaaaaaa} (*)$$ is equivalent to the constrained ...
28 views

### Cannot understand the bound of Lagrangian parameter in SMO

I'm trying to understand SMO, but stuck to the part of bound for Lagrangian parameters. In the SMO paper(https://www.microsoft.com/en-us/research/uploads/prod/1998/04/sequential-minimal-optimization....
1 vote
173 views

### How to maximize the ELBO in coordinate ascent variational inference

In the lecture by D.Blei: https://www.cs.princeton.edu/courses/archive/fall11/cos597C/lectures/variational-inference-i.pdf Variational inference is explained and he shows how to derive the optimal ...
48 views

### deriving the optimal distribution

Let the input variable $X \in \mathcal{X}$ and the target variable $Y \in \mathcal{Y}$. For a fixed hypothesis $h \in \mathcal{H}$ I want to solve \begin{equation} \min_{p(X,Y)} \int_{\mathcal{X}}\...
1 vote
38 views

### incorporating a distribution constraint in a minimisation objective

For a given (convex) hypothesis $h \in \mathcal{H}$, and the variables $X \in \mathcal{X}$ and $Y \in \mathcal{Y}$ I have the following optimisation problem: \begin{equation} \min_{p(X,Y)} \int_{\...
1 vote
102 views

### Derivation of the SVM regression problem

EDIT: The terms at the end will not get cancelled as they involve different variables: $\alpha$ and $\alpha^*$, etc. Now the post is corrected. I will leave this question here in case someone is as ...
212 views

### Why $\gamma$ in regularization term of XGBoost is defined as minimum loss reduction (not minimum squared loss reduction) and not substracted?

From the source https://xgboost.readthedocs.io/en/stable/tutorials/model.html I guess that the mean-squared error is optimized subjected to a constraint of minimum loss reduction. It appears like ...
88 views

### KKT Conditions for thresholds?

My main question is that when I use Lagrange Multipliers/KKT conditions to perform optimization with threshold constraints, I seem to get contradictory FOC. Here is a characteristic example: take an ...
1 vote
229 views

2k views

### How to derive the $\ nR^2$ form of the LM test statistic?

Several lagrange multiplier (LM) test statistics =$\ nR^2$~$\chi^2$, where the LM test statistic is generated from regressing the square of residuals on some function in an auxiliary equation. The ...
69 views

1 vote
24 views

### Score test using restricted mle

I am doing a study on Rao score test and still confused about the score vector. How am I supposed to calculate the score vector? Can I just use (ˆθn - θn) for the score vector? As I understood that ...
1 vote
35 views

### Rewrite $\frac{1}{2}||x-u||_2^2$ subject to $||x||_1\le c$ to lagrangian form with multiplier $\lambda \ge 0$

Rewrite $\frac{1}{2}||x-u||_2^2$ subject to $||x||_1\le c$ to lagrangian form with multiplier $\lambda \ge 0$ So I'm pretty new to converting constraint functions to Lagrangian form, but I read that ...
94 views

### Non-negative Constraints in Soft-Margin SVM Lagrange Equation

I was reading the A Tutorial on Support Vector Machines for Pattern Recognition as a supplemental for my Intro to ML class and I wasn't sure why $a_i \geq 0$ and $\lambda_i \geq 0$ cannot be ...
216 views

320 views

### Maximum Entropy Discrete Distribution

In Pattern Recognition and Machine Learning the author uses Lagrange multipliers to find the discrete distribution with maximum entropy. Entropy is defined by; $$H=-\sum_i p(x_i)\ln(p(x_i))$$ and the ...
1 vote
77 views

### What is $\alpha$ in SVM Lagrangian fucntions

Is it a column matrix or a row matrix? I saw on a CMU slide that $\alpha$ is defined as $\alpha=(\alpha_1, \alpha_2, ..., \alpha_n)^T>=0$? I dont see much of a formal definition for the exact form ...
1 vote
46 views

### Convert the following expression w.r.t to the whole dataset instead of element of the dataset?

I am in the process of expressing the w in LSSVM with data points and constants. After I resolve the KKT conditions for the LSSVM I got $$w = \sum ^N _{i=1} \alpha_i x_i$$ Is it possible to convert ...
176 views

### Minimize $f(A,B)$ s.t. $\text{exp}(A)^T \text{exp}(B)=J_K$

I have a function $f(A,B)$ that maps a pair of two (tall) matrices, $A$ and $B$, to a scalar cost that I want to minimize. $A$ and $B$ both have $K$ columns. I also want to impose a set of equality ...
1 vote
56 views

6k views

### Calculating the value of $b^{*}$ in an SVM

In Andrew Ng's notes on SVMs, he claims that once we solve the dual problem and get $\alpha^*$ we can calculate $w^*$ and consequently calculate $b^*$ from the primal to get equation (11) (see notes) ...
Consider the following constrained optimization problem. $$\begin{array}{ll} \text{minimize} & f(w)\\ \text{subject to} & g_{i}(w) \leq 0\\ & h_{j}(w) = 0\end{array}$$ The equivalent ...