Questions tagged [constrained-optimization]
The constrained-optimization tag has no usage guidance.
41
questions with no upvoted or accepted answers
4
votes
0
answers
166
views
Trying to understand the theory behind my similar / better results than XGBoost using a calibrated linear model (GAM)
I just opened a discussion on reddit asking about why/how the calibrated linear models I've been training have been getting similar / better results than XGBoost in my experiments. I was told to cross ...
- 41
4
votes
2
answers
124
views
MSE of correlations
These might be dumb questions but I am having trouble to wrap my head around of a particular problem. I have a sparse count matrix $G $ that I want to optimize which is $N \times p$. Also, I have ...
- 93
3
votes
0
answers
27
views
How to combine ML + Expert knowledge? (constrained machine learning)
I am working the sector of computer science for agriculture research. I deal here with algorithm for crop yield prediction. However, data in agriculture is very limited.
To overcome the issues of ...
- 31
3
votes
0
answers
149
views
Does there always exist a support vector such that $0 < a_n < C$ in SVM?
I have a question about SVM training in overlapping class distributions, where we are trying to minimize
$$
\dfrac{1}{2} \Vert \mathbf w \Vert_2^2 + C \sum_{n = 1}^N \xi_n \, ,
$$
subject to $y_n(\...
- 131
3
votes
1
answer
162
views
How does removal of symmetry (e.g. via constraints) in a Bayesian optimization search space affect search efficiency?
There are many examples of search space symmetry in real-world optimization problems in the physical sciences. To motivate this, here are some that come to mind:
When optimizing a formulation such as ...
- 69
2
votes
0
answers
16
views
Transforming discrete optimisation problem into continuous optimisation problem
In Sparse Hilbert-Schmidt Independence Criterion Regression (Poignard and Yamada, AISTATS 2020), the authors consider a way to perform feature selection by taking the subset of features that maximises ...
- 51
2
votes
0
answers
19
views
Constrained optimization between two bayesian variables
I have 2 separate Bayesain networks and I was hoping to maximize Value within the constraint of the Cost. What are is a good way ...
- 21
2
votes
0
answers
65
views
If L2-Regularization includes no bias, why do many images show a circle as the constraint region?
I got a little bit (massively, to be honest), confused by the following apparent misconceptions I have learned recently.
Looking for information about L2-Regularization, the following image is one of ...
- 515
2
votes
0
answers
203
views
Train a model subject to max error
I would like to train a neural network by minimizing a loss over samples (as usual), but doing so in a way that the maximum error is bounded.
What options do I have? Some that come to mind are:
...
2
votes
0
answers
143
views
LASSO and duality theorem
I am confused with Lagrange duality theorem. Let us consider the problem
$$
\hat{\beta} = \underset{\beta \in \mathbb{R}^{n}}{\arg \min} \left[\sum_{i=1}^{n}(y_{i} - \beta_{i})^{2} + \lambda \sum_{i=...
- 73
2
votes
0
answers
76
views
Covariance matrix of beta coefficients for constrained multiple regression
I have a linear least-squares problem with constraints that two of the coefficients must be non-negative. For a typical (unconstrained) least squares estimation, I know that the variance-covariance ...
- 21
2
votes
0
answers
89
views
Can I use xboost as objective function in an optimization problem?
I am working on a marketing optimization problem, where the goal is maximize profit by optimally allocating spend to different products. Constraint is getting at least 1 Million revenue.
As a first ...
- 827
2
votes
0
answers
71
views
Gradient descent finds local minima for a problem that can be formulated as a convex problem
I am trying to find
$$ \min_W \|Y-XW \|_F^2$$ $$s.t. \exists ij, W_{ij}\geq0 $$
where X is input data and Y is the output data we try to fit to. This is a convex optimization problem that can be ...
- 281
1
vote
0
answers
31
views
Closed Form Solution for MLE parameter defining Linear Combination of two multivariate normal distributions
I have one set of $n$ observations which can be described as a single vector sampled from a multivariate normal distribution of the following form:
$$
(1-\lambda)\mathbb{I}_n + \lambda \Sigma_{n}
$$
...
- 131
1
vote
0
answers
75
views
How does quadratic programming solve the Support Vector Machine problem?
I have just been reading that Quadratic Programming can be used to solve the Support Vector Machine optimization.
My solver can minimize this typ of problem
$$\text{J}_{min} = \frac{1}{2}x^TQx + c^Tx$$...
- 415
1
vote
0
answers
73
views
Can we convert the optimization of a loss function with regularization to the Lagrangian, constrained optimization *before* solving the optimization?
It is shown here that the optimization of a loss function with regularization,
$$\text{argmin}_b L(X,b) + c ||b||_p \phantom{aaaaaaaaaaaaaaaaaaaaaaaa} (*)$$
is equivalent to the constrained ...
- 441
1
vote
0
answers
39
views
Constrained imputation in Python
I actually have two original datasets (each one for a departure that are related to each one in a specific way , but it's not important to know how exactly) , but these 2 datasets contain some ...
- 11
1
vote
0
answers
35
views
Case study question on profit optimization
A famous restaurant selling burgers has closely studied the demand for burgers for past months including the times the customer requested a burger and it was already sold out. Further analysis
of data ...
- 21
1
vote
1
answer
43
views
Incorporate known class likelihood (proportions, ratios, etc..) in the classification output
I'm working on the multi-class prediction problem, with 6 output classes. These represent different types of land cover. The classification model is pixel-based and I have extracted different ...
1
vote
0
answers
55
views
Relaxed non-negative least squares
I am reconstructing a probability vector from data using non-negative least squares:
$$
\sum_\alpha \left(\pi_\alpha - \sum_i W_{\alpha i}p_i\right)^2\rightarrow \min,\\
p_i\geq 0,\sum_i p_i=1
$$
...
- 3,933
1
vote
0
answers
225
views
CVXPY PSD constraint not working
I am using CVXPY to solve for a PSD matrix, example as follows:
...
1
vote
0
answers
41
views
alternative solution to fussed lasso
The question is related to strange result from fused lasso estimator
Let us consider fussed lasso estimator:
$$
\hat{\beta}^{FL} = \underset{\beta \in \mathbb{R}^{n}}{\arg \min} [(y_{i} - \beta_{i})^{...
- 73
1
vote
0
answers
1k
views
`Error: L-BFGS-B needs finite values of 'fn'` of Complex Objective Function
I'm trying to run R's maximum likelihood estimation function (stats4::mle), over a likelihood function in Free Shipping Is Not ...
- 21
1
vote
0
answers
84
views
Discrete Bayes Net learning under parameter constraints
What is some relevant research available on estimating the parameters of a Bayes Net (with known structure) when there are known constraints on conditional and marginal probabilities?
For example, ...
- 1,158
1
vote
0
answers
114
views
Solving coefficient sum constrained elastic net with quadratic objective term
I am looking for an algorithm to solve an equality constrained elastic net.
There are two adaptations I need to make to the standard elastic net. First the objective function includes a quadratic ...
- 11
1
vote
0
answers
50
views
Fitting sums of Gaussian-like density functions
I acknowledge that similar question has been asked a couple years ago, yet it still seems unresolved. Ignoring the domain knowledge, the statistical problem behind is to fit a non-linear regression ...
- 11
0
votes
0
answers
39
views
How is the SVM optimization objective derived from the hinge loss function?
The hinge loss function, in the context of SVMs, is given as:
$$
\mathcal{L}(\mathbf{\vec w}, b\,; \mathbf{\vec x}^{(i)}, y ^{(i)}) = \max(0, 1-y ^{(i)}(\mathbf{\vec w}\cdot \mathbf{\vec x}^{(i)} + b))...
- 115
0
votes
0
answers
28
views
Rationale for box-constrained optimization in adversarial example search
In Section 4.1 of "Intriguing properties of neural networks" by Szegedy et al., the authors define the optimization problem they solve to find adversarial examples in a deep neural network. ...
- 371
0
votes
0
answers
6
views
Methods for delaying the "break" in non-linear least squares optimisation when the step size gets too small?
I am using the Levenberg-Marquardt method for calibration purposes. Typically, the RMSE of my calibration looks like:
I want to break the algorithm when the algorithm step-updates start to slow down, ...
0
votes
0
answers
5
views
How Can I Train a Real-World-Ready Classifier with Limited Real Data and Abundant Open-Source Data?
I am trying to train a text classifier with open-source data to generalize on the real user traffic (henceforth "real data"). However, even though I have many annotated open-source data, I ...
- 217
0
votes
0
answers
17
views
log-log regression as reward function in optimization problem
Consider the model $\hat{y}_t = e^{\text{trend} + \text{seasonality}} \prod_k^K x_{k, t}^{b_k}$
where $K$ denotes different investment alternatives. You can think that trend and seasonality are ...
0
votes
0
answers
30
views
Learning over non-independent joint distributions
For integer $n\geq 1$, I have a "goodness" function $f_n(F)$ that takes as input a given joint CDF $F$ of $n$ variables, and spits out a number in $[0,1]$ on how "good" $F$ is. The ...
- 253
0
votes
0
answers
22
views
Tree-reweighted belief propagation: optimizing edge appearances $\mu$
I am currently implementing Tree-Reweighted Belief Propagation (TRBP) to optimize edge appearances. The authors in the main manuscript of this work keep the edge appearances, represented by 𝜇, fixed [...
- 105
0
votes
0
answers
20
views
Convex optimization problem with nuclear norm constraint
We have the following convex optimization problem:$$
\text{minimize} \quad f({\bf X}) \quad \text{with constraint} \quad \|{\bf X}\|_{\rm tr} \leq t
$$
Where $\|{\bf X}\|_{\rm tr}$ is the Schatten 1-...
0
votes
1
answer
67
views
Bayesian optimization with constraints
I want to perform Bayesian optimization for a certain physical task but with additional requirements. We have access to a set of variables and want to maximize (multiple) signal outputs from an ...
- 23
0
votes
0
answers
141
views
Why SCM weights *should* sum to 1?
When we use synthetic controls, we consider $j=2, \dots, J+1$ units across $t \in \{T_{-} \ldots T_0 \ldots T_{+}\} \in \mathbb{Z}$ pre/post event-time periods. Abadie et. al. make the point that to ...
- 171
0
votes
0
answers
133
views
Speeding up an optimization involving matrix products in CVXR
I have an optimization problem where I need to minimize
$$-\log \det(U^T \text{diag}(p) U + V^T\text{diag}(1 - p)V)$$
where $p$ is a vector of probabilities, i.e. $0 \leq p_i \leq 1$, and $U$ and $V$ ...
user272429
0
votes
0
answers
214
views
Applying constraints within a neural network?
I need to solve a multi-objective problem. I would understand if there is any kind of possibility to cope this issue through a neural network. Hypothetically, I need to put some constraints within ...
0
votes
0
answers
73
views
Combining two terms into a quadratic form
I have an objective function defined by $ min_{Y_{t}} \hspace{2mm} ||X_{t} - Y_{t}D_{t}^{T}||_{F}^{2} + \lambda_{2}\sum_{i,j} w_{i,j}||\mathbf{y}_{i} - \mathbf{p}_{j}^{t}||_{2}^{2}$ where capital $T$ (...
- 1,946
0
votes
0
answers
34
views
ML algorithm to create an optimal photo portfolio
I would like to train an ML algorithm that would help create a best subset of 3 photos from a set of N photos. I'm wondering what kind of model could be used for this kind of portfolio/deck ...
- 143
-1
votes
1
answer
19
views
Match a Sequence to a bigger Sequence
I am trying to solve the following problem:
Say I have a small sequence $\sigma = (σ_{1}, σ_{2}, ..., σ_{i}, ..., σ_{N})$ and a larger $\hat{\sigma} = (\hat{σ}_{1}, \hat{σ}_{2}, ..., \hat{σ}_{j}, ..., ...
- 99