Skip to main content

Questions tagged [constrained-optimization]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
20 views

Constrained Cholesky Decomposition

Suppose that I have an $(n\times 1)$ vector of random variables, $\varepsilon$. However, I know that $k$ linear combinations of $\varepsilon$ are 0. Specifically, I know that for a $(k\times n)$ ...
Leland's user avatar
  • 1
0 votes
0 answers
42 views

How is the SVM optimization objective derived from the hinge loss function?

The hinge loss function, in the context of SVMs, is given as: $$ \mathcal{L}(\mathbf{\vec w}, b\,; \mathbf{\vec x}^{(i)}, y ^{(i)}) = \max(0, 1-y ^{(i)}(\mathbf{\vec w}\cdot \mathbf{\vec x}^{(i)} + b))...
Sagnik Taraphdar's user avatar
1 vote
0 answers
42 views

Closed Form Solution for MLE parameter defining Linear Combination of two multivariate normal distributions

I have one set of $n$ observations which can be described as a single vector sampled from a multivariate normal distribution of the following form: $$ (1-\lambda)\mathbb{I}_n + \lambda \Sigma_{n} $$ ...
A Friendly Fish's user avatar
2 votes
0 answers
30 views

Transforming discrete optimisation problem into continuous optimisation problem

In Sparse Hilbert-Schmidt Independence Criterion Regression (Poignard and Yamada, AISTATS 2020), the authors consider a way to perform feature selection by taking the subset of features that maximises ...
LoveRKHS's user avatar
3 votes
0 answers
63 views

How do I change my Scipy.Optimize.Minimize configuration to better find solution for constrained optimization? [closed]

I'm trying to do constrained optimization with Scipy.Optimize.Minimize and I keep on getting no solution, when I know there is a solution, because I can find one ...
confused's user avatar
  • 3,273
0 votes
0 answers
28 views

Rationale for box-constrained optimization in adversarial example search

In Section 4.1 of "Intriguing properties of neural networks" by Szegedy et al., the authors define the optimization problem they solve to find adversarial examples in a deep neural network. ...
synack's user avatar
  • 371
0 votes
0 answers
6 views

Methods for delaying the "break" in non-linear least squares optimisation when the step size gets too small?

I am using the Levenberg-Marquardt method for calibration purposes. Typically, the RMSE of my calibration looks like: I want to break the algorithm when the algorithm step-updates start to slow down, ...
THATS MY QUANT MY QUANTITATIVE's user avatar
0 votes
0 answers
5 views

How Can I Train a Real-World-Ready Classifier with Limited Real Data and Abundant Open-Source Data?

I am trying to train a text classifier with open-source data to generalize on the real user traffic (henceforth "real data"). However, even though I have many annotated open-source data, I ...
Mr.Robot's user avatar
  • 247
0 votes
1 answer
77 views

What exactly is the KKT check and what is the point of it?

In the paper for strong screening rules for the lasso (link), the following screening algorithm is proposed (start of chapter 7): Let $S(\lambda)$ be the strong rule set. Then the following strategy ...
Sparsity's user avatar
  • 584
0 votes
0 answers
17 views

log-log regression as reward function in optimization problem

Consider the model $\hat{y}_t = e^{\text{trend} + \text{seasonality}} \prod_k^K x_{k, t}^{b_k}$ where $K$ denotes different investment alternatives. You can think that trend and seasonality are ...
pete lewis's user avatar
0 votes
0 answers
30 views

Learning over non-independent joint distributions

For integer $n\geq 1$, I have a "goodness" function $f_n(F)$ that takes as input a given joint CDF $F$ of $n$ variables, and spits out a number in $[0,1]$ on how "good" $F$ is. The ...
AspiringMat's user avatar
4 votes
0 answers
175 views

Trying to understand the theory behind my similar / better results than XGBoost using a calibrated linear model (GAM)

I just opened a discussion on reddit asking about why/how the calibrated linear models I've been training have been getting similar / better results than XGBoost in my experiments. I was told to cross ...
William's user avatar
  • 41
1 vote
0 answers
30 views

Tree-reweighted belief propagation: optimizing edge appearances $\mu$

I am currently implementing Tree-Reweighted Belief Propagation (TRBP) to optimize edge appearances. The authors in the main manuscript of this work keep the edge appearances, represented by 𝜇, fixed [...
c.uent's user avatar
  • 115
2 votes
1 answer
214 views

How to solve alphas (or the dual equation) after getting Lagrangian dual of SVM

I'm trying to learn SVM by myself, and I'm stuck after getting the dual of SVM. I understand getting the dual after the primal. But, I am stuck here. Please help. We assume that the hard margin case ...
dvdy's user avatar
  • 21
1 vote
1 answer
135 views

Full-rank approximation to a square matrix

Let $\bf A$ be an $n \times n$ matrix with rank $r$ where $r<n$. How can I get a full-rank approximation for $\bf A$? In other words, I want to find the rank-$n$ $\bf X$ that minimizes the ...
MMM's user avatar
  • 850
1 vote
0 answers
80 views

How does quadratic programming solve the Support Vector Machine problem?

I have just been reading that Quadratic Programming can be used to solve the Support Vector Machine optimization. My solver can minimize this typ of problem $$\text{J}_{min} = \frac{1}{2}x^TQx + c^Tx$$...
euraad's user avatar
  • 425
2 votes
0 answers
19 views

Constrained optimization between two bayesian variables

I have 2 separate Bayesain networks and I was hoping to maximize Value within the constraint of the Cost. What are is a good way ...
stat_math's user avatar
0 votes
0 answers
20 views

Convex optimization problem with nuclear norm constraint

We have the following convex optimization problem:$$ \text{minimize} \quad f({\bf X}) \quad \text{with constraint} \quad \|{\bf X}\|_{\rm tr} \leq t $$ Where $\|{\bf X}\|_{\rm tr}$ is the Schatten 1-...
The Limit Does Not Exist's user avatar
1 vote
0 answers
77 views

Can we convert the optimization of a loss function with regularization to the Lagrangian, constrained optimization *before* solving the optimization?

It is shown here that the optimization of a loss function with regularization, $$\text{argmin}_b L(X,b) + c ||b||_p \phantom{aaaaaaaaaaaaaaaaaaaaaaaa} (*)$$ is equivalent to the constrained ...
travelingbones's user avatar
1 vote
0 answers
42 views

Constrained imputation in Python

I actually have two original datasets (each one for a departure that are related to each one in a specific way , but it's not important to know how exactly) , but these 2 datasets contain some ...
natsuhadder's user avatar
1 vote
0 answers
35 views

Case study question on profit optimization

A famous restaurant selling burgers has closely studied the demand for burgers for past months including the times the customer requested a burger and it was already sold out. Further analysis of data ...
the_why_guy's user avatar
2 votes
0 answers
65 views

If L2-Regularization includes no bias, why do many images show a circle as the constraint region?

I got a little bit (massively, to be honest), confused by the following apparent misconceptions I have learned recently. Looking for information about L2-Regularization, the following image is one of ...
kklaw's user avatar
  • 535
1 vote
1 answer
43 views

Incorporate known class likelihood (proportions, ratios, etc..) in the classification output

I'm working on the multi-class prediction problem, with 6 output classes. These represent different types of land cover. The classification model is pixel-based and I have extracted different ...
kap.provalija's user avatar
3 votes
0 answers
27 views

How to combine ML + Expert knowledge? (constrained machine learning)

I am working the sector of computer science for agriculture research. I deal here with algorithm for crop yield prediction. However, data in agriculture is very limited. To overcome the issues of ...
MvB's user avatar
  • 31
3 votes
0 answers
152 views

Does there always exist a support vector such that $0 < a_n < C$ in SVM?

I have a question about SVM training in overlapping class distributions, where we are trying to minimize $$ \dfrac{1}{2} \Vert \mathbf w \Vert_2^2 + C \sum_{n = 1}^N \xi_n \, , $$ subject to $y_n(\...
entechnic's user avatar
  • 131
2 votes
1 answer
55 views

Algorithm for Creating 2x2 Tables to Demonstrate Simpson's Paradox

Suppose I have a 2x2 table: $T = \lbrace a,b,c,d \rbrace$, where $a=T(1,1), b=T(2,1), c=T(1,2), d=T(2,2)$, where all entries of $T$ are positive integers. Let us assume that $\frac{ad}{bc} > 1$. ...
user67724's user avatar
  • 333
0 votes
1 answer
83 views

Bayesian optimization with constraints

I want to perform Bayesian optimization for a certain physical task but with additional requirements. We have access to a set of variables and want to maximize (multiple) signal outputs from an ...
arod's user avatar
  • 23
13 votes
3 answers
2k views

Why l2 norm squared but l1 norm not squared?

In the Lasso, and ElasticNet, we use, as penalty, the l1 norm without squaring. But in the ElasticNet and Ridge, we use the l2 norm squared. Why is that, is there a particular reason (computational, ...
William de Vazelhes's user avatar
2 votes
1 answer
95 views

Interpretation of the entropic relaxation of the optimal transport problem

For two probabilities vector $r,c\in \mathbb{R}^k$, the optimal transport is to find the joint distribution of $r,c$ such that it $$\min_{T\in P} \langle T,C \rangle\\P\in\mathbb{R}^{k\times x},\text{...
rando's user avatar
  • 308
0 votes
0 answers
24 views

Implementations for non-negative regression in generalized linear models [duplicate]

What are some package for generalized linear models that give non-negative regression coefficients without regularization? I checked the thread (Nonnegative generalized linear model), but many ...
zhli12's user avatar
  • 21
1 vote
0 answers
59 views

Relaxed non-negative least squares

I am reconstructing a probability vector from data using non-negative least squares: $$ \sum_\alpha \left(\pi_\alpha - \sum_i W_{\alpha i}p_i\right)^2\rightarrow \min,\\ p_i\geq 0,\sum_i p_i=1 $$ ...
Roger V.'s user avatar
  • 4,439
4 votes
1 answer
64 views

Einstein notation $-$ or another $-$ to denote constraints in high dimensional ILP problems

When discussing marginal sums of arrays in 3 dimensions or more, is it customary in the statistical and/or data science communities to use the Einstein summation convention? Is some other form ...
Peter Leopold's user avatar
4 votes
1 answer
357 views

Handling multicollinearity with Restricted Least Squares

The dummy variable trap - including a dummy variable for every category and including a constant term in the regression together guarantees perfect multicollinearity - is most commonly resolved by ...
vpy's user avatar
  • 73
2 votes
0 answers
233 views

CVXPY PSD constraint not working

I am using CVXPY to solve for a PSD matrix, example as follows: ...
regression_practitioner's user avatar
2 votes
0 answers
214 views

Train a model subject to max error

I would like to train a neural network by minimizing a loss over samples (as usual), but doing so in a way that the maximum error is bounded. What options do I have? Some that come to mind are: ...
Franco Marchesoni's user avatar
0 votes
0 answers
45 views

How to ensure that the solution to an optimization problem is a probability distribution? [duplicate]

How to ensure that the solution to an optimisation problem is a probability distribution? For example, assume we minimise over a distribution $p$. We must ensure that $\int p(x) \,dx=1 $. But why don'...
appa's user avatar
  • 127
3 votes
1 answer
201 views

Is standard gradient descent possible when we have constrained parameters?

Is it true that standard gradient descent algorithm (be it batch or mini batch or stochastic) cannot be used when we have certain constraint on parameters? If yes, why is it so? Is it because gradient ...
Curious's user avatar
  • 421
0 votes
1 answer
49 views

deriving the optimal distribution

Let the input variable $X \in \mathcal{X}$ and the target variable $Y \in \mathcal{Y}$. For a fixed hypothesis $h \in \mathcal{H}$ I want to solve \begin{equation} \min_{p(X,Y)} \int_{\mathcal{X}}\...
appa's user avatar
  • 127
4 votes
1 answer
115 views

strange result from fused lasso estimator

Let us consider the following estimator: $$ \hat{\beta}^{F} = \underset{\beta \in \mathbb{R}^{n}}{\arg \min} (y_{i} - \beta_{i})^{2} + \lambda_{1} \sum_{i=1}^{n-1}|\beta_{i} - \beta_{i+1}|, $$ which ...
AnTlr's user avatar
  • 43
3 votes
1 answer
165 views

How does removal of symmetry (e.g. via constraints) in a Bayesian optimization search space affect search efficiency?

There are many examples of search space symmetry in real-world optimization problems in the physical sciences. To motivate this, here are some that come to mind: When optimizing a formulation such as ...
Sterling's user avatar
2 votes
0 answers
77 views

Covariance matrix of beta coefficients for constrained multiple regression

I have a linear least-squares problem with constraints that two of the coefficients must be non-negative. For a typical (unconstrained) least squares estimation, I know that the variance-covariance ...
cozisco's user avatar
  • 21
1 vote
0 answers
1k views

`Error: L-BFGS-B needs finite values of 'fn'` of Complex Objective Function

I'm trying to run R's maximum likelihood estimation function (stats4::mle), over a likelihood function in Free Shipping Is Not ...
nmck160's user avatar
  • 21
0 votes
0 answers
149 views

Why SCM weights *should* sum to 1?

When we use synthetic controls, we consider $j=2, \dots, J+1$ units across $t \in \{T_{-} \ldots T_0 \ldots T_{+}\} \in \mathbb{Z}$ pre/post event-time periods. Abadie et. al. make the point that to ...
Jared Greathouse's user avatar
3 votes
1 answer
97 views

KKT Conditions for thresholds?

My main question is that when I use Lagrange Multipliers/KKT conditions to perform optimization with threshold constraints, I seem to get contradictory FOC. Here is a characteristic example: take an ...
naveace's user avatar
  • 115
1 vote
0 answers
865 views

How do you use pytorch to solve strictly constrained optimization problems? [closed]

I am trying to solve the following problem using pytorch: given a six sided die whose average roll is known to be 4.5, what is the maximum entropy distribution for the faces? (Note: I know a bunch of ...
Paul Siegel's user avatar
0 votes
0 answers
136 views

Speeding up an optimization involving matrix products in CVXR

I have an optimization problem where I need to minimize $$-\log \det(U^T \text{diag}(p) U + V^T\text{diag}(1 - p)V)$$ where $p$ is a vector of probabilities, i.e. $0 \leq p_i \leq 1$, and $U$ and $V$ ...
user avatar
1 vote
0 answers
84 views

Discrete Bayes Net learning under parameter constraints

What is some relevant research available on estimating the parameters of a Bayes Net (with known structure) when there are known constraints on conditional and marginal probabilities? For example, ...
Innuo's user avatar
  • 1,158
1 vote
1 answer
187 views

How do linear constraints affect the convexity of my OLS-like optimisation problem?

I would like to augment a linear regression (so a convex OLS problem) with some additional constraints on the coefficients to match the subject I'm working on. Having $x\in \mathbb{R}^n$, the solution ...
quentin's user avatar
  • 11
0 votes
0 answers
220 views

Applying constraints within a neural network?

I need to solve a multi-objective problem. I would understand if there is any kind of possibility to cope this issue through a neural network. Hypothetically, I need to put some constraints within ...
Giacomo Segala's user avatar
2 votes
1 answer
22 views

Optimal Feature Engeneering creation: best optimization method?

basically I would like to solve this problem: (1) say I have N features that I want to transform with a generic f(x, theta) ...
Asher11's user avatar
  • 229