The Stack Overflow podcast is back! Listen to an interview with our new CEO.

Questions tagged [lagrange-multipliers]

The method of Lagrange multipliers finds critical points (including maxima and minima) of a differentiable function subject to differentiable constraints.

Filter by
Sorted by
Tagged with
0
votes
0answers
15 views

Spatial Lag or spatial Error Model? Deciding by using the Lagrange multiplier diagnostics

Honestly, my knowledge of geostatistics is limited. My assumptions are as follows: If I want to choose between a Spatial Lag Model (SLM) and a Spatial Error Model (SEM), I can use the Lagrange ...
1
vote
0answers
39 views

Optimal proposal for the Metropolis-Hastings algorithm using Tierney's theorem

Let $(E,\mathcal E,\lambda)$ be a measure space $p:E\to[0,\infty)$ be $\mathcal E$-measurable with $$c:=\lambda p\in(0,\infty)$$ and $$\mu:=\underbrace{\frac1cp}_{=:\:\tilde p}\lambda$$ $q:E^2\to[0,\...
0
votes
0answers
20 views

Solving constrained optimization problems with Adam

The adam algorithm has been very successful for solving non-convex optimization problems that appear in deep learning. Are there ways to extend adam to solve constrained optimization problems? Among ...
0
votes
0answers
7 views

Lagrange multiplier test in Mixed Level Model

I want to estimate a mixed level model with AR(1) errors and then conduct a Lagrange Multiplier test. The mixed model allows for rich covariance structures but it does not allow for AR(1) errors. Can ...
1
vote
1answer
31 views

why in SVM we have different indices for dot product?

I am confused by Lagrangian method in SVM, I can not understand why we use different indices in dot product. Suppose with using Lagrangian W is : $ W_{i}=\sum_{i}L_{i}y_{i}x_{i} $ In SVM ...
1
vote
0answers
36 views

why we use Lagrangian method in SVM?

I am wondering why we are using Lagrangian method in SVM? if we have just 3 features and 1000 rows, then objective function would be $w_{1}^{2}+w_{2}^{2}+w_{3}^{2}$ but if we use Lagrangian then we ...
1
vote
0answers
16 views

how we can constraints in dual of lagrangian?

I am confused about the general rule of Lagrangian multiplier. (which usully use for SVM). I could not find a good book or paper that explain it completely and clearly. Suppose we have $$minimize: ...
0
votes
0answers
19 views

Equivalence of two optimization problems [duplicate]

Ridge regression problem: $\sum_i^n(y_i-\beta_0-\beta_{1i}-\beta_{2i})^2 \longrightarrow min_{\beta}$ $s.t. \sum_i^p\beta_i^2 \leq c$ $\sum_i^n(y_i-\beta_0-\beta_{1i}-\beta_{2i})^2 + \lambda(\sum_i^...
0
votes
0answers
53 views

How to calculate the Langrangian of the hard margin SVM primal problem?

I need to compute the Lagrangian of the primal problem for hard margin SVMs by hand. This is an assignment for university! I have vectors $$x_0 = (0, 0), x_1=(1, 2), x_2 = (-1, 2)$$ and $$y_1 = -1, ...
2
votes
0answers
31 views

Maximum entropy probability distribution over non-negative support and finite mean?

I'm trying to derive which univariate probability distribution maximizes entropy, assuming finite mean $\mu$ and non-negative support $[0, \infty)$. I know that the answer is the exponential ...
0
votes
0answers
34 views

Score test for multiple parameter model

Suppose I have a multivariate linear model: $y = Z\bar{\alpha} + x\beta + \epsilon$ Where $Z$ is a matrix of covariates and $\bar{\alpha}$ are their effects. I refer to them as covariates as I am ...
0
votes
0answers
198 views

How to choose between dual gradient descent and the method of Lagrangian multipliers?

For an optimization problem $$ \max f(x)\\\ s.t. g(x)\le 0 $$ The Lagrangian is $$ \mathcal L(x, \lambda)=f(x)-\lambda g(x) $$ Dual gradient descent solves it by (according to Page 43 of this lecture,...
1
vote
0answers
11 views

Wide variety of results in Breusch-Pagan Tests on simulated data?

I wanted to see how the results of the BP-test would come out. This stemmed from a debate with someone who claimed that the BP-test would start rejecting the null hypothesis of homoskedasticity even ...
0
votes
0answers
41 views

Lagrange Multiplier Statistic When Testing for Heteroskedasticity

I had a debate with another student and wanted to get some more perspective. She brought up a fascinating point for when we test for heteroskedasticity through the Breusch-Pagan Test. Recall, $$LM = ...
0
votes
0answers
47 views

How to apply Lagrange Multiplier to a matrix?

I am learning about the Lagrange Multiplier and I see how to apply it to a set of equations but I don't know how to apply it to matrices. Suppose I have: ...
1
vote
0answers
176 views

How do we know we're maximizing the Lagrangian objective function in PCA?

In Principal Component Analysis, we start with $m$ observations $x_1,\dots,x_m$, each of which is an $n$-dimensional vector. Assume we have centered the data; that is, we have subtracted the variable ...
0
votes
1answer
51 views
0
votes
0answers
81 views

Unable to solve using lagrangian multipliers

Suppose $$K(x,z) = \theta(x)^T \theta(z) = \left\{ \begin{array}{ll} 1 & \text{if } x = z \\ 0 & \text{otherwise} \end{array} \right. $$ and $y_1=+1$ or $-1$. I ...
1
vote
0answers
23 views

Finding three coefficients (weight/ratio) by minimising variance

I am finding the value of a,b and c for minimising the variance of the following equation (four variables are correlated): $$ Var(\Delta V)=V ar[\Delta S-a\Delta F_1 - b\Delta F_2 -c\Delta F_3] $$ ...
3
votes
1answer
449 views

Gradient descent in SVM

I am studying about SVM now. Then I came across the problem. The dual optimization problem is as follows: \begin{align*} &\max_\alpha~~~~~ W(\alpha) = \sum_{i=1}^{n} \alpha_i -\frac{1}{2}\...
1
vote
0answers
125 views

Support Vector Machines: a beginner's question about the underlying math

I'm new to Support Vector Machines and I've been trying to get into the underlying math (instead of just using Scikit Learn or something like that). I understand the math behind it up to the point ...
0
votes
2answers
967 views

SVM: Why alpha for non support vector is zero?

In the optimization problem in SVM to compute the margin, we use Lagrange multipliers to insert the constraint: $L(w,b,\alpha)= \frac{1}{2}|w| - \sum \alpha (y_i(w*x_i+b) -1) $ Now we want to compute ...
1
vote
0answers
261 views

Lagrange multipliers and mle

Could someone give a method that works for the following question? I am new to the topic and cannot understand why there is no constant $\lambda$ in the first two equations (which would arise from the ...
14
votes
4answers
2k views

The proof of equivalent formulas of ridge regression

I have read the most popular books in statistical learning 1- The elements of statistical learning. 2- An introduction to statistical learning. Both mention that ridge regression has two formulas ...
0
votes
0answers
117 views

How do we know the value of the regularization parameter satisfies the gradient equations required by Lagrange Multipliers?

I've take multiple machine learning classes and I am always told been told say when we do regularization on the training error $\mathcal E(W) = \frac{1}{n} \sum^n_{i=1}Loss(f(x_i),y_i)$: $$ \text{ (1)...
1
vote
0answers
178 views

Determine lagrange multiplier LASSO

I used the function l1ce from the lasso2 package in R, since I need to solve a minimization problem with a lasso constraint. min f(x) s.t. ||beta|| where beta are the parameters. The reason it ...
10
votes
1answer
536 views

LASSO relationship between $\lambda$ and $t$

My understanding of LASSO regression is that the regression coefficients are selected to solve the minimisation problem: $$\min_\beta \|y - X \beta\|_2^2 \ \\s.t. \|\beta\|_1 \leq t$$ In practice ...
3
votes
1answer
126 views

Using technique of Lagrange Multiplier

How can we use the technique of Lagrange multipliers to find a new vector of parameters $w$ which solves the optimization problem: minimize J(w) = $\frac{1}{2} || w -u ||^2$ such that: $w^T (x − y)...
0
votes
1answer
406 views

R built-in Breusch-Pagan Test getting different results from manual Calculation

I was using the following code to calculate the bp-test (eaef.csv can be found here). ...
0
votes
0answers
616 views

ARCH effect - Conflicting Results from Lagrange Multiplier Test and Ljung Box Test?

I want to test for presence of conditional heteroskedasticity in a vector of ARIMA(0,0,0) residuals. For this purpose, I would like to see if both ARCH Lagrange Multiplier Test (on levels of ...
2
votes
0answers
19 views

Scaling prediction from VAR model subject to a equality constraint

I have a forecasting problem and already built a decently working VAR model which provides forecasts as $\hat{Y}_{iT}$, for $i = 1,..n$ and $T$ is forecast time period. But now I have an additional ...
0
votes
0answers
381 views

Number of lags to use in Engle's ARCH-LM test [duplicate]

How does one decide on number of lags to use in this test? Can one just justify the number based on previous papers? To clarify, I am forecasting conditional variance, and have generated my log ...
1
vote
1answer
403 views

Can I use a spatially lagged Dependent Variable while using Spatial Error Model?

I have following issue: I run spatial diagnostics on dependencies for my Log-Log Transformed regression model. LM Tests (including Robust) are highly significant. Since I am using GeoDa, I cannot ...
1
vote
0answers
300 views

SVM - Reason for using Lagrangian Dual [duplicate]

Objective To confirm if the understanding is correct regarding the reason why Lagrangian Dual is used in SVM. Background While Machine Learning, gradient descent is used at regression, logistic ...
13
votes
2answers
740 views

KKT in a nutshell graphically

Objective Confirm if the understanding of KKT is correct or not. Seek for further explanation and confirmations on KKT. Background Trying to understand KKT conditions, especially the complementary ...
4
votes
1answer
2k views

Lagrangian multiplier: role of the constraint sign

I am beginner learning Lagrange multipliers with wiki article. Consider: maximize $f(x,y)$ subject to $g(x,y) = 0$ I understand that to maximize I must follow the gradient $\nabla {_{x, y}}^{}f$. I ...
8
votes
1answer
2k views

Why are the Lagrange multipliers sparse for SVMs?

I've read that for the Maximal Margin Classifier SVM, after solving the dual problem, most of the lagrange multipliers turn out to be zeros. Only the ones corresponding to the support vectors turn out ...