In statistics this refers to selecting an estimator of a parameter by maximizing or minimizing some function of the data. One very common example is choosing an estimator which maximizes the joint density (or mass function) of the observed data referred to as Maximum Likelihood Estimation (MLE).

learn more… | top users | synonyms (2)

0
votes
0answers
5 views

Drawing tangible insights from rpart model splits

I'm working on a data set for an engineering problem which involves the use of pumps. My main objective is figuring out how to reduce the number of pumps used where the maximum number of pumps is 4 ...
0
votes
0answers
18 views

Combinatorial optimization

Suppose I have some tasks $y_1$$, ... ,$ $y_k$ and some workers $1, ... ,m$ to do them. Furthermore, every task-worker combination has some cost $c_{ij}$ with $i=1, ...,k, j=1, ...,m$. In addition, ...
0
votes
2answers
25 views

Why can multicollinearity be a problem for logistic regression?

Let's say that I want to run a logistic regression on a dataset with n observations and p variables and I have a bad model. I can't understand why running again a logistic regression but this time ...
0
votes
0answers
26 views

Why do we transform constrained optimization problems to unconstrained ones? [migrated]

In methods like Lagrange multipliers or augmented Lagrangian methods we transform a constrained optimization problem into an unconstrained one and then solve it. For example in Lagrange multipliers ...
1
vote
0answers
26 views

Using gradient information in minimizing error function, in Bishop's Pattern Recognition

In Bishop's book Pattern Recognition, there appears the following paragraph on page 239, where I included the equation he refers to In the quadratic approximation to the error function, ...
0
votes
0answers
9 views

Naming of a Function which Estimates Other

Sometimes we can’t optimize a function (or objective) itself (e.g. because of complexity), and thus we try to optimize another function (or objective) that is similar to it. I think there was a name ...
0
votes
1answer
28 views

Lagrangian multiplier: role of the constraint sign

I am beginner learning Lagrange multipliers with wiki article. Consider: maximize $f(x,y)$ subject to $g(x,y) = 0$ I understand that to maximize I must follow the gradient $\nabla {_{x, y}}^{}f$. I ...
0
votes
2answers
37 views

Why it is popular to use stochastic gradient descent in neural networks rather than the BFGS algorithm?

I have made two solvers to implement neural networks, one is based on stochastic gradient descent (SGD) while the other is based on the BFGS (Broyden-Fletcher-Goldfarb-Shanno) algorithm. I have read ...
0
votes
1answer
37 views

Why normalize input variables in NN?

I'm reading the 'Efficient Backprop' paper and it's mentioned that the reason to have a zero mean for the input variables is because otherwise the eigenvalue (for the hessian I think) will be very ...
2
votes
4answers
155 views

Provable Convergence

Hopefully this question is on topic for the statistics community (rather than the pure math) since the optimization method I am using relies on a statistical model. Well I am building an algorithm ...
0
votes
0answers
14 views

Changing optimization algorithm while optimizing

I am currently training some convolutional neural networks with cross-entropy loss. Thus, the function I am optimizing is non-convex, and at the moment I am using an optimization algorithm called Adam ...
0
votes
0answers
22 views

How to estimate parameter k in hyperbolic discounting equation using maximum likelihood?

I've been learning and trying to estimate parameters using maximum likelihood, and I'm trying to understand the hyperbolic discounting equation. Here's the equation for hyperbolic discounting: $$y = ...
0
votes
1answer
23 views

What are the precise/exact halting conditions for batch Gradient Descent (GD) when working with data with random noise?

I was trying to understand the different alternatives there are for halting batch gradient descent and concluding that one has reached some minimum (local or global). In the presence of noisy data ...
1
vote
0answers
17 views

What optimization methods work best for LSTMs?

I've been using theano to experiment with LSTMs, and was wondering what optimization methods (SGD, Adagrad, Adadelta, RMSprop, Adam, etc) work best for LSTMs? Are there any research papers on this ...
0
votes
0answers
26 views

What's the difference in effect of the two ways to normalize data?

When I first started learning applied statistics, I was always taught that normalizing data for use in models was to do the following: $ \frac{x - \bar{x}}{std(x)} $ This scales your data to 0-mean, ...
0
votes
0answers
12 views

Effcient way to solve quadratic convex optimization within R in a concrete example

I'm not at all an expert in optimization and therefore have a maybe trivial question regarding the implementation of the following problem within R. It's from a phd thesis, which can be found here. ...
0
votes
1answer
24 views

Using SVM when kernel is simple and sample size is large

Consider SVM classification: $y_i \in \{+1,-1\} $ are labels, $\mathbf{x}_i$ are covariates ($i=1\ldots N$). Let $K(\cdot,\cdot)$ be the kernel function, whose corresponding feature mapping is ...
2
votes
2answers
191 views

Why is sum of squared residuals non-increasing when adding explanatory variable?

In my econometric textbook(Introductory Econometrics) covering OLS, the author write, "SSR must fall when another explanatory variable is added." Why is it?
1
vote
2answers
27 views

Problem with prior mean in MOE (Bayesian optimization)

I am playing with MOE package (yelp.github.io/MOE) - I try to optimize some function of one variable, adding one point for sample at a time. Here is the intermediate chart I got: Blue line is the ...
1
vote
1answer
27 views

variable transformation in optimization

I have an optimization problem with two sets of parameters, $x_i \in [0,1]$ and $y_k \in [-\frac{\pi}{2},\frac{\pi}{2}]$ where $i,k \in \{1...n\}$ are indices. One way to solve this problem is using ...
0
votes
0answers
33 views

Maximise a function in R [migrated]

I am trying to maximize a function I created. ...
1
vote
0answers
17 views

No need for standardization with Adadelta? (RNN/LSTM)

Often it is best to standardize data before inputting it into a machine learning algorithm. This is also the case with deep learning algorithms such as convolutional neural network. However, when ...
3
votes
2answers
134 views

Why does adding L1 penalty to R's optim slows things down so much (relative to no penalty or L2)?

I'm running some optimizations with optim's implementation of BFGS. The objective function is actually a computational algorithm, not just math. I found when I add an L1 penalty, things slow down ...
0
votes
1answer
36 views

Stochastic Gradient Descent vs Online Gradient Descent

I was wondering what the difference between stochastic gradient descent and online gradient descent is? Or is it the same algorithm? Thanks.
3
votes
2answers
71 views

Maximization of a nasty Gaussian likelihood

I asked this question in math.SE before. One answer so far, and we were unable to reach a conclusion. It is more related to statistics, so I wanted to post this here. I have a Gaussian likelihood ...
0
votes
0answers
12 views

How to choose the ordering of weighting parameters in a optimization problem

I have the following optimization problem from a paper: $\underline{x}^* = \underset{\underline{x}}{\operatorname{argmin}} \left\{ \lambda_1 \lambda_2 \Theta_s(\underline{x},\underline{I}) + ...
0
votes
4answers
36 views

Gradient Descent: Guaranteeing Cost Function Decreases

I'm reading this and am a bit confused starting around equation (9). Suppose we have a real-valued function of many variables, $$v = v_1, v_2, ...$$ Let the gradient of our cost function, C, be: ...
1
vote
1answer
46 views

What is the best statistical technique to optimize Online Advertising Spend?

I am trying to optimize marketing spend across multiple websites i.e., Nanigans (Facebook), Google, etc, to increase customer conversion (purchasing). Each ad placement results in two things: new ...
0
votes
0answers
22 views

$\alpha$-Convex and $\alpha$-Concave functions

I see that in the definition of $\alpha$-concave functions (see, e.g., Shapiro et al. 2009 p.94), it is assumed that the function is non-negative I think it is not necessary to have a non-negative ...
1
vote
1answer
36 views

Why is coxph() so fast for survival analysis on big data?

I frequently do survival analysis on large data sets. One million samples or more is typical, and this seems to be much more than typical research usage. Many algorithms I've used are prohibitively ...
3
votes
0answers
65 views

Advantages and disadvantages of machine learning hyperparameter optimizers [closed]

What are the respective advantages/disadvantages of the following optimization algorithms for ML applications? (that is, to optimize the hyperparameters of a SVM, RForest, Boosting model, etc.). In ...
0
votes
0answers
37 views

Optimization Business Problem

We have a business problem and we are trying to understand how we should approach it. There are two firms, A and B. Firm A provides products to Firm B. For A, to make one product, it needs a total of ...
0
votes
0answers
23 views

Calculating the optimal Holt Winters parameters (not in R)?

The Holt Winters (HW) technique requires the following parameters: Alpha, Beta and Gamma. The accuracy of the forecasts depends on these parameters. Some software packages (like in R) are able to find ...
0
votes
0answers
22 views

Optimization of a non homogenous Poisson process with unknown intensity distribution

Customers arrive at a bank counter according to a non-homogenous Poisson process with rate parameter $\lambda_k$. The service time at the counter is exponentially distributed where $\lambda_k$ and ...
1
vote
1answer
20 views

Minimization of function with random error

I'm aware of techniques to find the minimum of a function $f(x)$ by taking iterative samples, such as Newton's Method. Suppose, however, that what I can actually observe is $g(x) = f(x) + E$, where ...
0
votes
0answers
12 views

How does Shuffled-Complex-Evolution-Metropolis algorithm compare to other adaptive samplers (e.g. NUTS)?

I recently heard of the Shuffled-Complex-Evolution-Metropolis algorithm and am curious how it compares to other adaptive MCMC sampling algorithms. Unfortunately I am still learning about optimizing ...
1
vote
0answers
17 views

How to find the threshold that maximizes the F1 score?

I have a probabilistic, binary classifier. Is there any principled way to select the threshold that maximizes the F1 score? Currently I simply choose many different thresholds, apply them on some ...
1
vote
2answers
83 views

How to constrain cumulative Gaussian parameters so that the function will intersect one given point?

I am analyzing data from one study where participants had to choose (between two stimuli) the one with higher intensity. One way to look at the data is to fit the proportion of correct choices as a ...
0
votes
1answer
41 views

SVD from Matrix formulation to objective function

I'm writing the question to try to complete the circle after reading the 2 other questions on Cross Validated and the link on the third bullet point: What is the objective function of PCA? What ...
0
votes
1answer
50 views

How do you turn the output of a nnet neural network model into an equation?

Assuming the output of the above nnet feedforward model (nnetModel) is such that the following summary is produced: ...
0
votes
1answer
47 views

GLMER and Model is nearly unidentifiable: very large eigenvalue

I'm working on a logistic regression analysis using the lme4 package and function glmer. I built the following model: ...
1
vote
1answer
82 views

Is it possible to use genetic algorithms as a learning algorithm for artificial neural networks?

I've been using back-propagation to optimize neural networks without problems. But I've read in some books that genetic algorithms can be used to optimize an ANN. I want to know if its possible to use ...
0
votes
1answer
35 views

Is Representer theorem valid with constraints on coefficients?

By the representer theorem, we have that in a Reproducing Kernel Hilbert Space the function being learnt in a regularizer + loss function problem under some conditions, can be represented as $\sum_i ...
1
vote
0answers
31 views

Gradient of a sum of indicators

EDITED w.r.t. whuber's comment: Say I have a function $\mathbb R^n \rightarrow \mathbb R$: $$f(w_1,\ldots,w_n) = \frac{n^-\sum_{i\in I^-}w_ix_i}{n^+\sum_{i\in I^+}w_ix_i}$$ with fixed $x_i\in\mathbb ...
1
vote
0answers
44 views

How does one extend radial basis function (RBF) networks formally from regularization but with vector valued outputs?

I was reading the following paper on hyper & radial basis function (HBFs & RBFs) networks and also this one that kind of summarizes the first one and was trying to understand how to extend ...
1
vote
0answers
202 views

What are key differences between Theano (Python) and Torch (Lua) for deep learning? [closed]

Theano and Torch both supports GPU calculations. My question is whether Theano or Torch have significant differences in: performance ease of use (assuming one knows the programming language) libaray ...
5
votes
0answers
78 views

Why do Elo rating system use wrong update rule?

Elo rating system use a gradient descent minimization algorithm of the cross-entropy loss function between the expected and observed probability of an outcome in paired comparisons. We can write the ...
0
votes
0answers
10 views

Instances of this Gaussian variance model with non-convex log-likelihood

On page 4 of Thomas Minka's Beyond Newton's Method, he mentions the following Gaussian variance model with a non-convex maximum likelihood objective. What are some examples/instances of models where ...
0
votes
0answers
14 views

Interpolation of Data Value using Optimized Weighting of Its Features

I have a question regarding "Interpolation" / "Prediction" of a value. Assuming we have a data set $ { \left\{ \left( {x}_{i}, {y}_{i} \right) \right\}}_{i = 1}^{N} $ where $ {x}_{i} \in ...
0
votes
1answer
22 views

First Derivative / Maximisation Problem

It appears to be super easy, but I just cannot reconcile how Lambert et al. (2012) on page 8 come from (3) to (4) - please see screenshot! Particularly bothering: where does the "Lambda" in the ...