Questions tagged [optimization]
Use this tag for any use of optimization within statistics.
2,574
questions
0
votes
0
answers
7
views
No gradient for one parameter on the first iteration of gradient descent
Say we have a dataset $D$ of 2-tuples $(x, y)$ where $y$ is the target variable and a function $f_\theta$:
$$
D = \{(1, 3), (2, 5), (3, 8), (4, 6), (5, 9)\},\quad f_\theta(x) = \theta_0 + \theta_1 x.
$...
- 157
0
votes
0
answers
19
views
Minecraft performance lower than before? [closed]
I recently compressed the .png files located in my minecraft resource pack. I was hoping for a little bit better performance since a smaller resource pack is usually faster.
But as i found out, while ...
- 1
0
votes
0
answers
11
views
Shrinkage / L1 regularization as a loss term versus a constraint (post-process step) with momentum optimizers
I have a complex model with very non-linear operations (divisions, exponentials, matrix inversions, square roots, Cholesky decompositions, etc...) for which I want to optimize the parameters. However, ...
0
votes
0
answers
41
views
Estimating Top n Prices points on a given day for a particular product which would maximize revenue
Problem Statement :-
On any given date of the year, get top n prices which would maximize the revenue for that day for a particular platform.
Dataset :-
Date
Price ($p_{i}$)
Platform
$X_{1,i}$, $X_{...
- 133
0
votes
0
answers
17
views
Finding feature values for regression model such that output is more than a given value?
Suppose you have an (online) shop. You have a dataset containing $p$ features (representing customer characteristics) $X = (x_1, \ldots, x_p)$ and a feature $y$, representing how much money a customer ...
- 1,560
0
votes
0
answers
17
views
Detailed comparison of two methods for obtaining the ridge regression solution
I have come across two different ways of obtaining the ridge regression solution, which are as follows:
Method1:-(obtained from here)
$RSS(\beta) = (Y-X\beta)^T\cdot(Y-X\beta)+\lambda\beta^T\Omega\...
- 187
0
votes
0
answers
14
views
Non linear knapsack optimization
Problem:
Primarily problem to solve: Allocate budget for most revenue given ROIs at a given investment.
Secondarily problem: Minimize budget to meet a certain revenue threshold.
All while having some ...
- 1,489
2
votes
1
answer
72
views
Should folds in k-fold CV actually be representative?
I have read somewhere that the k of the k-fold CV should be picked in such a manner as to have representative validation sets (folds). This seems to me to be contradictory since the leave-one-out CV ...
- 23
0
votes
0
answers
19
views
Finding an optimal value for a function [closed]
I am trying to implement a change point model for stock prices, (finding the point in time where there is an abrupt shift in the time series trend)
Essentially I want to find an optimal value for the ...
0
votes
0
answers
33
views
Why does Adam optimizer with gradient clipping perform better than simple Adam optimizer?
Since Adam optimizer uses the first and second moments of gradients to adapt the learning rate, what purpose does the gradient clipping serve when used with Adam optimizer or any adaptive learning ...
- 151
0
votes
0
answers
22
views
weighted maximum likelihood as loss function
I have built a little neural network that I use for regression.
...
- 113
0
votes
0
answers
14
views
How to conduct EM algorithm when there are some outliers in GMM Models?
I'm just confused about the problem of adding an outlier component directly to the primary form of GMM models:
Suppose that the observed data contains several outliers. The mixture model could be:
$$
...
- 1
1
vote
1
answer
27
views
Support vector machine, complementary slackness and marginal hyperplane
One of the complementary slackness conditions for a support vector machine states that
$$\alpha_i ( y_i (\langle w, x_i \rangle + b ) -1 ) = 0,$$ where $\alpha_i$ is the lagrange variable. One can ...
- 129
0
votes
0
answers
3
views
A question on the projection step in Generic Adaptive Method Setup: $x_{t+1} = \Pi_{\mathcal{F},\sqrt{V_t}} (\hat{x}_{t+1}).$
I am reading the paper "ON THE CONVERGENCE OF ADAM AND BEYOND". In this paper, they proposed the following framework of adaptive methods.
I was confused on the last step: $x_{t+1} = \Pi_{\...
- 101
2
votes
1
answer
31
views
How to handle weighted examples in stochastic gradient descent (with mini-batches)?
Suppose I have $M$ data points $x_i$ and associated weights $w_i > 0$. I want to optimize a function,
$$F(\theta) = \frac{1}{M}\sum_i w_i f(x_i;\theta)$$
in the parameters $\theta$. I will assume ...
- 3,348
0
votes
0
answers
30
views
Estimating the coefficients of a non-linear regression
I am trying to estimate the coefficients $\lambda, \alpha, \beta_1, \beta_2, \gamma, \eta$ in the below equation using Python and some financial data
$$ \lambda \times \text{(participation %)} \times \...
- 133
0
votes
0
answers
24
views
Why are Gaussian Processes used in Bayesian Optimization? [duplicate]
In Bayesian Optimization, the function (i.e. objective function) that we are trying to optimize is modelled using some surrogate function - this surrogate function usually turns out to be a Gaussian ...
- 5,750
1
vote
2
answers
33
views
Why use transpose of nabla in gradient descent
For gradient descent we have the formula:
$f(x_{k}+d_{k})\approx f(x_{k}) + \nabla f(x_{k})^T d_{k} $
What I don't understand is, why we use the transpose of nabla and not just nabla.
- 13
0
votes
0
answers
7
views
Encoding knowledge of data lying in the union of hyperplanes using differentiable optimization layers
I know that a way to possibly encode prior knowledge into neural networks training is by using differentiable optimization layers (paper).
I'm in the following situation, and I'm wondering if it could ...
- 427
1
vote
1
answer
20
views
How to perform fine balance matching in R for several covariates?
I want to use fine balance matching to balance the marginal distribution of some covariates, so that it will be less stringent than exact matching.
Below are some options I found:
The ...
- 475
2
votes
2
answers
49
views
Why do we use Acquisition Functions?
In the context of Surrogate Modelling and Bayesian Optimization - Acquisition Functions (https://tune.tidymodels.org/articles/acquisition_functions.html) are often used as a "compass" to ...
- 5,750
2
votes
1
answer
63
views
Combinations from different sets with weightings
Imagine the following scenario:
I want to create 1000 unique combinations of clothing. The combinations would include the following categories: hats, shirts, shorts, socks and shoes. Each combination ...
- 23
0
votes
0
answers
9
views
Searching for combination of feature-comparisons to optimize a metric on a subset of the data?
Suppose that I have a dataset with n features X1, ..., Xn, and label Y (consider it binary for now).
Features can be constructed with "meta"-features by comparisons (>,<, >=, <=),...
- 512
0
votes
0
answers
14
views
Why validation loss can drop a lot at a higher learning rate?
When we are choosing the optimal learning rate for a neural network, I thought the normal validation_loss & learning rate trend looks like this:
Today when I'm running this pytorch forecasting ...
- 289
0
votes
0
answers
13
views
Deriving Optimizer of Quadratic Loss for Classification
I'm currently considering a binary classification problem where we have data points $X_1,\dots,X_n\in\mathbb{R}^d$ and labels $y_i=\pm1$. I'm using a simple linear model to model $y_i$, and it has the ...
- 157
1
vote
1
answer
39
views
Is it possible to pick more than 1 sample point in each iteration of bayesian optimisation?
I want to use Bayesian optimisation for my project and I plan to build a closed-loop system, such that there is a model, robot to conduct experiments, measurement of experimental data which updates ...
- 21
0
votes
0
answers
20
views
Sequential feature selection stopping condition
When using sequential feature selection approach, say forward feature selection, I want to stop adding new features when the improvement in model scores is smaller than a certain level. Is there a ...
- 1
2
votes
0
answers
48
views
Monte Carlo and function minimization (simulated annealing)
I posted this question on math.stackexchange, but did not get an answer and a limited amount of views, so I removed the question there and post it here.
Recently I was asking myself some basic ...
- 663
0
votes
1
answer
37
views
Asymptotic normality with weighted sum of objective function $\min_{x} \; f_n(x) + g_n(x)$
Suppose
$f_n(x)$, $g_n(x)$ are convex functions w.r.t. $x$
the optimal point of the two problems $\min_x f_n(x)$
and $\min_x g_n(x)$ have asymptotic normality as $n \rightarrow \infty$
they converge ...
- 33
1
vote
3
answers
175
views
No free lunch theorem proof
Assume that learning algorithm $A$ is fixed. Let $D = \{(x_1,y_1),...,(x_N,y_N)\}$, $F$ is set of a data-generating functions(meaning $f \in F$ then $f(x_i) = y_i$ and that functions in $F$ are ...
1
vote
1
answer
26
views
Gradient of the second order term of Newton's Method
I know that Netwon's method can be pushed to the second order using the 1st Taylor expansion. However, how can I generalize Netwon's method to take x_0 as a vector and have the ability take the ...
1
vote
1
answer
23
views
How does SGD training error decrease in subsequent epochs with non-iid samples when it is recommended that samples in subsequent epochs be iid?
I have been reading the Deep Learning book by Ian Goodfellow and on pg. 277, they mention:
It is also crucial that the minibatches be selected randomly.
Computing an unbiased estimate of the expected ...
- 111
2
votes
0
answers
31
views
PCA Minimizing Reconstruction Error
In PCA, we want to fit the best $k$-dimensional subspace to given data $x_1,\dots,x_n\in\mathbb{R}^d$. Specifically, we want to approximate each $x_i$ by $\mu + V_k\alpha_i$, where $\mu \in \mathbb{R}^...
- 157
0
votes
0
answers
9
views
DoE for optimization / control approach?
I'm wondering whether a DoE approach could somehow be used as kind of an optimization algorithm?
One of my current tasks is to find a set of five parameter which max a sixth one (see here for more: ...
- 2,187
1
vote
1
answer
28
views
Is gradient descent the only way to find the weights in logistic regression?
This post: When is logistic regression solved in closed form? describes that we must use nonlinear optimization methods to find the parameter estimates for logistic regression models. Does gradient ...
- 55
0
votes
0
answers
16
views
Difference between analytic and numeric gradient in Matlab
I am using matlab to solve a optimization problem. When I check the anlaytic and numeric gradient reported by matlab, they are quite different. So I want to ask if there is a mistake in the analytic ...
- 105
13
votes
3
answers
2k
views
If I use a regularization (e.g. L2) can I not apply early stopping?
I've seen that early stopping is a form of regularization that limits the movement of the parameters $\theta$ in a similar way that L2 Regularization penalizes the movement of $\theta$ to be closer to ...
- 287
0
votes
0
answers
37
views
Difference between forward-mode and reverse-mode automatic differentiation?
I have difficulty grasping the difference between forward and reverse mode automatic differentiation. To understand this problem I have created a simple equation and broken this equation into small ...
- 2,031
4
votes
2
answers
222
views
Is there a preference in the regression performance metric for regression models with the same type of loss minimization?
I applied two regression models (ordinary least square (OLS) and linear absolute regression) to the same dataset, where this dataset is split into train and test sets.
Two performance measures are ...
- 1,487
1
vote
2
answers
187
views
Why can't we just add a penalty to make the Neural Network objective convex?
When we use Neural Networks to solve various tasks, we define an objective that the Network parameters $\theta=(\theta_i)_{1\le i\le N}\in\Theta^N$ have to minimize. So, for neural networks $f(\cdot|\...
- 718
0
votes
0
answers
19
views
Noisy Optimization by Regressing the Derivative
Say that we have a neural network with a set of weights. We train the network with SGD. Looking at a single weight w, we can plot the SGD derivative of the error with respect to the weight, against ...
- 123
0
votes
0
answers
22
views
Optimise non-additive blends
I have a set of data something like:
Parameter A
Parameter B
...
Parameter N
Sample 1
numeric data
numeric data
numeric data
numeric data
Sample 2
numeric data
numeric data
numeric data
numeric ...
- 686
0
votes
0
answers
30
views
SVM: How to derive the KKT condition with soft margin term is quadratic?
I was reading the Introdunction to Data Mining (2013) when I came across this in section 5.5:
$$
f(\textbf{w})=\frac{||\textbf{w}||^2}{2}+C\left(\sum^N_{i=1}\xi_i \right)^k
$$
I found similar ...
- 23
1
vote
1
answer
127
views
Does increasing the variance increase the value of a function?
Let $V=\sum_{i=1}^{k} a_iX_i$ where $X_i's$ are IID $\sim Bern(q)$ and $V$ with $\sum a_i=k$. Note that $a_i$'s are non-negative integers.
I have a function $f$ as given below :
$$
f= \max_{q} h\...
- 185
0
votes
0
answers
32
views
Computing gradients for Gaussian processes using locally periodic kernel
I want to use stochastic gradient descent to find hyperparameters for the locally periodic kernel. The locally periodic kernel is the product of two kernels: the periodic and squared exponential ...
- 75
0
votes
0
answers
46
views
How to find an optimal sample size (or range) based on sampling errors?
Intro to problem
Increasing the sample size can be very costly in terms of time and money. However, in experiments where we measure proportions within a sample, errors associated with the sample size ...
- 1
0
votes
0
answers
12
views
Use learning rate decay with Adam [duplicate]
As Adam (i.e., Adaptive Momentum Estimation algorithm) handles the learning rate decay, won't it be redundant to perform a learning rate decay on plateau callback in the fit function?
I found this ...
- 212
0
votes
0
answers
15
views
Seeking A Scale-Independent Alternative To Q2 For Model Selection When The Response Varies Over Multiple Orders of Magnitude
I am using constrained polynomial regression to predict y = f(x). I have prior knowledge about the relationship that allows me to add constraints to the optimization problem for the first and second ...
- 1
0
votes
0
answers
36
views
TSP (Travel Salesman problem) for multigraph
I'm trying to solve the Travel Salesman problem for multigraph.
Namely, I have a fully-connected graph with 2 oriented edges between every pair of nodes. The weight of the edge from x to y corresponds ...
- 1
0
votes
1
answer
30
views
If a regression problem is ill-conditioned, does that mean we cannot perform SGD? What happens if we do?
By ill-conditioned regression problem, I mean that the feature matrix $X$ is not full rank. For example, X contains two or more columns highly correlated. If that's the case, $X^T\cdot X$ is not ...
- 143