Questions tagged [optimization]

Use this tag for any use of optimization within statistics.

Filter by
Sorted by
Tagged with
0 votes
0 answers
7 views

No gradient for one parameter on the first iteration of gradient descent

Say we have a dataset $D$ of 2-tuples $(x, y)$ where $y$ is the target variable and a function $f_\theta$: $$ D = \{(1, 3), (2, 5), (3, 8), (4, 6), (5, 9)\},\quad f_\theta(x) = \theta_0 + \theta_1 x. $...
user avatar
0 votes
0 answers
19 views

Minecraft performance lower than before? [closed]

I recently compressed the .png files located in my minecraft resource pack. I was hoping for a little bit better performance since a smaller resource pack is usually faster. But as i found out, while ...
user avatar
  • 1
0 votes
0 answers
11 views

Shrinkage / L1 regularization as a loss term versus a constraint (post-process step) with momentum optimizers

I have a complex model with very non-linear operations (divisions, exponentials, matrix inversions, square roots, Cholesky decompositions, etc...) for which I want to optimize the parameters. However, ...
user avatar
0 votes
0 answers
41 views

Estimating Top n Prices points on a given day for a particular product which would maximize revenue

Problem Statement :- On any given date of the year, get top n prices which would maximize the revenue for that day for a particular platform. Dataset :- Date Price ($p_{i}$) Platform $X_{1,i}$, $X_{...
user avatar
  • 133
0 votes
0 answers
17 views

Finding feature values for regression model such that output is more than a given value?

Suppose you have an (online) shop. You have a dataset containing $p$ features (representing customer characteristics) $X = (x_1, \ldots, x_p)$ and a feature $y$, representing how much money a customer ...
user avatar
  • 1,560
0 votes
0 answers
17 views

Detailed comparison of two methods for obtaining the ridge regression solution

I have come across two different ways of obtaining the ridge regression solution, which are as follows: Method1:-(obtained from here) $RSS(\beta) = (Y-X\beta)^T\cdot(Y-X\beta)+\lambda\beta^T\Omega\...
user avatar
  • 187
0 votes
0 answers
14 views

Non linear knapsack optimization

Problem: Primarily problem to solve: Allocate budget for most revenue given ROIs at a given investment. Secondarily problem: Minimize budget to meet a certain revenue threshold. All while having some ...
user avatar
  • 1,489
2 votes
1 answer
72 views

Should folds in k-fold CV actually be representative?

I have read somewhere that the k of the k-fold CV should be picked in such a manner as to have representative validation sets (folds). This seems to me to be contradictory since the leave-one-out CV ...
user avatar
0 votes
0 answers
19 views

Finding an optimal value for a function [closed]

I am trying to implement a change point model for stock prices, (finding the point in time where there is an abrupt shift in the time series trend) Essentially I want to find an optimal value for the ...
user avatar
0 votes
0 answers
33 views

Why does Adam optimizer with gradient clipping perform better than simple Adam optimizer?

Since Adam optimizer uses the first and second moments of gradients to adapt the learning rate, what purpose does the gradient clipping serve when used with Adam optimizer or any adaptive learning ...
user avatar
  • 151
0 votes
0 answers
22 views

weighted maximum likelihood as loss function

I have built a little neural network that I use for regression. ...
user avatar
  • 113
0 votes
0 answers
14 views

How to conduct EM algorithm when there are some outliers in GMM Models?

I'm just confused about the problem of adding an outlier component directly to the primary form of GMM models: Suppose that the observed data contains several outliers. The mixture model could be: $$ ...
user avatar
  • 1
1 vote
1 answer
27 views

Support vector machine, complementary slackness and marginal hyperplane

One of the complementary slackness conditions for a support vector machine states that $$\alpha_i ( y_i (\langle w, x_i \rangle + b ) -1 ) = 0,$$ where $\alpha_i$ is the lagrange variable. One can ...
user avatar
0 votes
0 answers
3 views

A question on the projection step in Generic Adaptive Method Setup: $x_{t+1} = \Pi_{\mathcal{F},\sqrt{V_t}} (\hat{x}_{t+1}).$

I am reading the paper "ON THE CONVERGENCE OF ADAM AND BEYOND". In this paper, they proposed the following framework of adaptive methods. I was confused on the last step: $x_{t+1} = \Pi_{\...
user avatar
  • 101
2 votes
1 answer
31 views

How to handle weighted examples in stochastic gradient descent (with mini-batches)?

Suppose I have $M$ data points $x_i$ and associated weights $w_i > 0$. I want to optimize a function, $$F(\theta) = \frac{1}{M}\sum_i w_i f(x_i;\theta)$$ in the parameters $\theta$. I will assume ...
user avatar
  • 3,348
0 votes
0 answers
30 views

Estimating the coefficients of a non-linear regression

I am trying to estimate the coefficients $\lambda, \alpha, \beta_1, \beta_2, \gamma, \eta$ in the below equation using Python and some financial data $$ \lambda \times \text{(participation %)} \times \...
user avatar
0 votes
0 answers
24 views

Why are Gaussian Processes used in Bayesian Optimization? [duplicate]

In Bayesian Optimization, the function (i.e. objective function) that we are trying to optimize is modelled using some surrogate function - this surrogate function usually turns out to be a Gaussian ...
user avatar
  • 5,750
1 vote
2 answers
33 views

Why use transpose of nabla in gradient descent

For gradient descent we have the formula: $f(x_{k}+d_{k})\approx f(x_{k}) + \nabla f(x_{k})^T d_{k} $ What I don't understand is, why we use the transpose of nabla and not just nabla.
user avatar
  • 13
0 votes
0 answers
7 views

Encoding knowledge of data lying in the union of hyperplanes using differentiable optimization layers

I know that a way to possibly encode prior knowledge into neural networks training is by using differentiable optimization layers (paper). I'm in the following situation, and I'm wondering if it could ...
user avatar
1 vote
1 answer
20 views

How to perform fine balance matching in R for several covariates?

I want to use fine balance matching to balance the marginal distribution of some covariates, so that it will be less stringent than exact matching. Below are some options I found: The ...
user avatar
  • 475
2 votes
2 answers
49 views

Why do we use Acquisition Functions?

In the context of Surrogate Modelling and Bayesian Optimization - Acquisition Functions (https://tune.tidymodels.org/articles/acquisition_functions.html) are often used as a "compass" to ...
user avatar
  • 5,750
2 votes
1 answer
63 views

Combinations from different sets with weightings

Imagine the following scenario: I want to create 1000 unique combinations of clothing. The combinations would include the following categories: hats, shirts, shorts, socks and shoes. Each combination ...
user avatar
  • 23
0 votes
0 answers
9 views

Searching for combination of feature-comparisons to optimize a metric on a subset of the data?

Suppose that I have a dataset with n features X1, ..., Xn, and label Y (consider it binary for now). Features can be constructed with "meta"-features by comparisons (>,<, >=, <=),...
user avatar
0 votes
0 answers
14 views

Why validation loss can drop a lot at a higher learning rate?

When we are choosing the optimal learning rate for a neural network, I thought the normal validation_loss & learning rate trend looks like this: Today when I'm running this pytorch forecasting ...
user avatar
  • 289
0 votes
0 answers
13 views

Deriving Optimizer of Quadratic Loss for Classification

I'm currently considering a binary classification problem where we have data points $X_1,\dots,X_n\in\mathbb{R}^d$ and labels $y_i=\pm1$. I'm using a simple linear model to model $y_i$, and it has the ...
user avatar
  • 157
1 vote
1 answer
39 views

Is it possible to pick more than 1 sample point in each iteration of bayesian optimisation?

I want to use Bayesian optimisation for my project and I plan to build a closed-loop system, such that there is a model, robot to conduct experiments, measurement of experimental data which updates ...
user avatar
  • 21
0 votes
0 answers
20 views

Sequential feature selection stopping condition

When using sequential feature selection approach, say forward feature selection, I want to stop adding new features when the improvement in model scores is smaller than a certain level. Is there a ...
user avatar
2 votes
0 answers
48 views

Monte Carlo and function minimization (simulated annealing)

I posted this question on math.stackexchange, but did not get an answer and a limited amount of views, so I removed the question there and post it here. Recently I was asking myself some basic ...
user avatar
  • 663
0 votes
1 answer
37 views

Asymptotic normality with weighted sum of objective function $\min_{x} \; f_n(x) + g_n(x)$

Suppose $f_n(x)$, $g_n(x)$ are convex functions w.r.t. $x$ the optimal point of the two problems $\min_x f_n(x)$ and $\min_x g_n(x)$ have asymptotic normality as $n \rightarrow \infty$ they converge ...
user avatar
  • 33
1 vote
3 answers
175 views

No free lunch theorem proof

Assume that learning algorithm $A$ is fixed. Let $D = \{(x_1,y_1),...,(x_N,y_N)\}$, $F$ is set of a data-generating functions(meaning $f \in F$ then $f(x_i) = y_i$ and that functions in $F$ are ...
user avatar
1 vote
1 answer
26 views

Gradient of the second order term of Newton's Method

I know that Netwon's method can be pushed to the second order using the 1st Taylor expansion. However, how can I generalize Netwon's method to take x_0 as a vector and have the ability take the ...
user avatar
1 vote
1 answer
23 views

How does SGD training error decrease in subsequent epochs with non-iid samples when it is recommended that samples in subsequent epochs be iid?

I have been reading the Deep Learning book by Ian Goodfellow and on pg. 277, they mention: It is also crucial that the minibatches be selected randomly. Computing an unbiased estimate of the expected ...
user avatar
2 votes
0 answers
31 views

PCA Minimizing Reconstruction Error

In PCA, we want to fit the best $k$-dimensional subspace to given data $x_1,\dots,x_n\in\mathbb{R}^d$. Specifically, we want to approximate each $x_i$ by $\mu + V_k\alpha_i$, where $\mu \in \mathbb{R}^...
user avatar
  • 157
0 votes
0 answers
9 views

DoE for optimization / control approach?

I'm wondering whether a DoE approach could somehow be used as kind of an optimization algorithm? One of my current tasks is to find a set of five parameter which max a sixth one (see here for more: ...
user avatar
  • 2,187
1 vote
1 answer
28 views

Is gradient descent the only way to find the weights in logistic regression?

This post: When is logistic regression solved in closed form? describes that we must use nonlinear optimization methods to find the parameter estimates for logistic regression models. Does gradient ...
user avatar
  • 55
0 votes
0 answers
16 views

Difference between analytic and numeric gradient in Matlab

I am using matlab to solve a optimization problem. When I check the anlaytic and numeric gradient reported by matlab, they are quite different. So I want to ask if there is a mistake in the analytic ...
user avatar
  • 105
13 votes
3 answers
2k views

If I use a regularization (e.g. L2) can I not apply early stopping?

I've seen that early stopping is a form of regularization that limits the movement of the parameters $\theta$ in a similar way that L2 Regularization penalizes the movement of $\theta$ to be closer to ...
user avatar
  • 287
0 votes
0 answers
37 views

Difference between forward-mode and reverse-mode automatic differentiation?

I have difficulty grasping the difference between forward and reverse mode automatic differentiation. To understand this problem I have created a simple equation and broken this equation into small ...
user avatar
  • 2,031
4 votes
2 answers
222 views

Is there a preference in the regression performance metric for regression models with the same type of loss minimization?

I applied two regression models (ordinary least square (OLS) and linear absolute regression) to the same dataset, where this dataset is split into train and test sets. Two performance measures are ...
user avatar
  • 1,487
1 vote
2 answers
187 views

Why can't we just add a penalty to make the Neural Network objective convex?

When we use Neural Networks to solve various tasks, we define an objective that the Network parameters $\theta=(\theta_i)_{1\le i\le N}\in\Theta^N$ have to minimize. So, for neural networks $f(\cdot|\...
user avatar
0 votes
0 answers
19 views

Noisy Optimization by Regressing the Derivative

Say that we have a neural network with a set of weights. We train the network with SGD. Looking at a single weight w, we can plot the SGD derivative of the error with respect to the weight, against ...
user avatar
  • 123
0 votes
0 answers
22 views

Optimise non-additive blends

I have a set of data something like: Parameter A Parameter B ... Parameter N Sample 1 numeric data numeric data numeric data numeric data Sample 2 numeric data numeric data numeric data numeric ...
user avatar
  • 686
0 votes
0 answers
30 views

SVM: How to derive the KKT condition with soft margin term is quadratic?

I was reading the Introdunction to Data Mining (2013) when I came across this in section 5.5: $$ f(\textbf{w})=\frac{||\textbf{w}||^2}{2}+C\left(\sum^N_{i=1}\xi_i \right)^k $$ I found similar ...
user avatar
1 vote
1 answer
127 views

Does increasing the variance increase the value of a function?

Let $V=\sum_{i=1}^{k} a_iX_i$ where $X_i's$ are IID $\sim Bern(q)$ and $V$ with $\sum a_i=k$. Note that $a_i$'s are non-negative integers. I have a function $f$ as given below : $$ f= \max_{q} h\...
user avatar
  • 185
0 votes
0 answers
32 views

Computing gradients for Gaussian processes using locally periodic kernel

I want to use stochastic gradient descent to find hyperparameters for the locally periodic kernel. The locally periodic kernel is the product of two kernels: the periodic and squared exponential ...
user avatar
0 votes
0 answers
46 views

How to find an optimal sample size (or range) based on sampling errors?

Intro to problem Increasing the sample size can be very costly in terms of time and money. However, in experiments where we measure proportions within a sample, errors associated with the sample size ...
user avatar
  • 1
0 votes
0 answers
12 views

Use learning rate decay with Adam [duplicate]

As Adam (i.e., Adaptive Momentum Estimation algorithm) handles the learning rate decay, won't it be redundant to perform a learning rate decay on plateau callback in the fit function? I found this ...
user avatar
  • 212
0 votes
0 answers
15 views

Seeking A Scale-Independent Alternative To Q2 For Model Selection When The Response Varies Over Multiple Orders of Magnitude

I am using constrained polynomial regression to predict y = f(x). I have prior knowledge about the relationship that allows me to add constraints to the optimization problem for the first and second ...
user avatar
0 votes
0 answers
36 views

TSP (Travel Salesman problem) for multigraph

I'm trying to solve the Travel Salesman problem for multigraph. Namely, I have a fully-connected graph with 2 oriented edges between every pair of nodes. The weight of the edge from x to y corresponds ...
user avatar
  • 1
0 votes
1 answer
30 views

If a regression problem is ill-conditioned, does that mean we cannot perform SGD? What happens if we do?

By ill-conditioned regression problem, I mean that the feature matrix $X$ is not full rank. For example, X contains two or more columns highly correlated. If that's the case, $X^T\cdot X$ is not ...
user avatar
  • 143

1
2 3 4 5
52