Skip to main content

Questions tagged [non-convex]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
2 votes
0 answers
86 views

Why do we need $\gamma>2$ in SCAD penalty?

The SCAD penalty $p(x | \lambda, \gamma)$ from https://myweb.uiowa.edu/pbreheny/7600/s16/notes/2-29.pdf or the paper "Variable Selection via Nonconcave Penalized Likelihood and its Oracle ...
Zifeng Zhang's user avatar
3 votes
1 answer
308 views

Is there an exponential family such that its natural parameter mapping is non-invertible or has non-convex range?

On the Wikipedia article for exponential families the density of a distribution on a measure space $(X, \xi)$ from an exponential family is written as $$f_{\theta} \colon X \to \mathbb{R}_{\ge 0}, \...
ViktorStein's user avatar
0 votes
0 answers
137 views

Do contour plots over first two principal components reveal local convexity/concavity?

When I plot a contour plot of a variable over two principal components I can see what appears to be hills and valleys. But I also know I am only looking at the contours over a projection. Here is a ...
Galen's user avatar
  • 8,804
1 vote
1 answer
280 views

The effect of over-parameterization on local minima

While reading some papers about over-parameterization in deep learning models, I also read that "over-parametrization is a simple method to introduce additional dimensionality and help make the ...
AGM's user avatar
  • 11
1 vote
0 answers
35 views

How can we find a unique (suboptimal) solution to an optimisation problem with a very large search space?

A standard linear and unconstrained optimization problem has the following form: $\max_{x} f(x)$, For example $f(x) = cos(x)+sin(2x)$, with $-1 \leq x \leq 8$. The gradient $=0$ will allow finding all ...
Tessa van der Heiden's user avatar
0 votes
1 answer
329 views

Cost function of neural networks can be non-convex, then why do we use it?

I saw a thread here (Cost function of neural network is non-convex?). After I read this, I am really confused. I am wondering that if the cost function is not convex, and we do backpropagation, then ...
JAEMTO's user avatar
  • 103
1 vote
1 answer
186 views

How do linear constraints affect the convexity of my OLS-like optimisation problem?

I would like to augment a linear regression (so a convex OLS problem) with some additional constraints on the coefficients to match the subject I'm working on. Having $x\in \mathbb{R}^n$, the solution ...
quentin's user avatar
  • 11
8 votes
1 answer
2k views

Are there any "convex neural networks"?

Are there any neural network training procedure that involves solving a convex problem? Note that I am referring more to MLPs, instead of (multi-class) logistic regression which is a neural network ...
Fraïssé's user avatar
  • 1,560
-1 votes
1 answer
114 views

Is this objective function convex? [closed]

Given that $F(x)$ is the cumulative distribution function (CDF) of continuous random variable $X$, is $$\frac{\int_0^\infty 1-F(x) dx}{\int_{-\infty}^0 F(x) dx}$$ convex? or is it non-convex/concave? ...
develarist's user avatar
  • 3,957
0 votes
1 answer
253 views

Does coordinate wise convex function can be optimized more effectively?

I'm currently working on a non-convex function. It's basically a maximum likelihood problem so I'm trying to optimize this function. I know that non-convex optimization frequently reaches local optima ...
deep_lazy's user avatar
2 votes
1 answer
409 views

Convergence under large set of learning rates

What is the interpretation of a stochastic optimization problem where a gradient descent algorithm is converging under a wide range of learning rate schedules (including ones with quite large initial ...
Dion's user avatar
  • 954
1 vote
0 answers
187 views

How to minimize the sum of Frobenius norm and Nuclear norm

I have to minimize an objective function of the the form : $||X_{s} - Y_{s}D_{s}||_{F}^{2} + ||D_{s}||_{F}^{2} + ||D_{s}||_{*}^{2}$ where $||.||_{F}$ denotes the Frobenius norm and $||.||_{*}$ ...
Upendra01's user avatar
  • 1,956
3 votes
1 answer
751 views

Why are K-means and GMM (Gaussian Mixture Models) not suitable for discovering clusters with non-convexs shapes?

I have seen that mainly here and from a lot of resources that K-means and Hello all! Gaussian mixtures are not suitable for detecting clusters with non-convex shapes. I know that because both ...
yer's user avatar
  • 125
3 votes
0 answers
53 views

If $\ell_0$ regularization can be done via the proximal operator, why are people still using LASSO?

I have just learned that a general framework in constrained optimization is called "proximal gradient optimization". It is interesting that the $\ell_0$ "norm" is also associated with a proximal ...
ArtificiallyIntelligent's user avatar
3 votes
0 answers
2k views

Solving constrained optimization problems with Adam

The adam algorithm has been very successful for solving non-convex optimization problems that appear in deep learning. Are there ways to extend adam to solve constrained optimization problems? Among ...
Hari's user avatar
  • 275
2 votes
0 answers
57 views

How to solve a non-convex with equality constraint optimization problem?

I have a non-convex optimization problem with equality constraint, I can derive the KKT conditions, but it seems just one of the KKT conditions is valid. Could you please give some advice on how to ...
Andrew's user avatar
  • 21
0 votes
1 answer
688 views

Is error function always assumed and convex?

While updating weights of the neural network, most of the algorithms use convex optimisation because of the reason that error is a convex function. My doubt is that whether the convex-ness of error ...
hanugm's user avatar
  • 205
1 vote
0 answers
428 views

How to optimize ratiometric loss function with variance term in it?

I'm training a neural network (or any ML model with non-convex gradient-based optimization) to predict a continuous outcome variable. Currently, I use the mean squared error loss function, i.e., if $y$...
adpbw's user avatar
  • 11
4 votes
0 answers
366 views

Optimization textbooks for statistics and data analytics

Any statistical analysis, machine learning or data science involves some sort of optimization at the end of the day. I'm looking for good linear and nonlinear optimization textbooks for self ...
1 vote
0 answers
43 views

Minimal requirement for reaching a feasible solution in non-convex constrained gradient descent

The problem is as follows: $\max_x f(x) \enspace , \enspace \text{s.t.} \enspace g(x) \leq \alpha $ We can not assume that either $f$ nor $g$ are convex, on the contrary - we can assume they are ...
Chen's user avatar
  • 121
0 votes
0 answers
181 views

How to solve a DC (Difference of Convex) program?

I have an objective function which is the difference of two convex functions in Tensorflow and I want to minimize it. Formally, I have the following problem: $\text{min}_{x \in \mathcal{X}} \;\;f(x)-...
KiaSh's user avatar
  • 71
20 votes
3 answers
8k views

When will gradient descent converge to a critical point or to a local/global minima) for non-convex functions?

What situations do we know of where gradient descent can be shown to converge (either to a critical point or to a local/global minima) for non-convex functions? For SGD on non-convex functions, one ...
gradstudent's user avatar
3 votes
1 answer
2k views

Non-convex optimization without using gradient descent

If we want to optimize a convex function, we could use methods like gradient descent or computing the derivative of the function and equalize to zero so we can obtain the global minimum. But I ...
dotcsv's user avatar
  • 35