Questions tagged [optimization]
Use this tag for any use of optimization within statistics.
1,965
questions
1
vote
0answers
11 views
Obtaining parameter errors with the basinhoppin method in Python
I use the basinhoppin method use in Python for global optimization. The result is named FittedParameters and is shown below:
...
0
votes
0answers
9 views
Cranking up my validation score
My way of building a model on the MNIST dataset was to first overfit it, find a good learning scheme and then try to regularize this model to reduce the overfitting.
To reduce the overfitting I'm ...
0
votes
0answers
13 views
Constrained optimization with piecewise objective function
Here's the issue. Let's say that I have a set of inputs theta, that must satisfy some set of linear constraints such that
$ui * theta - ci >= 0$
(Which is nothing more than the classical linear ...
0
votes
0answers
6 views
Likelihood surface plotting in MATLAB [closed]
I need to find $\delta_v$ and $\delta_h$ through MLE of the Likekiood function 'f(...)' of two parameters, defined as (other than $\delta_v$ and $\delta_h$ all all known variables)
where
How to get ...
1
vote
0answers
40 views
Optimal decisions based on frequentist estimators
Consider a decision problem aimed at minimizing the expected loss1 where the argument is a parameter estimate. In a Bayesian setting, given a posterior distribution of the parameter and the loss ...
2
votes
0answers
82 views
+250
Does EM algorithm require us to know the joint (predictive) distribution of the latent variables $Z$ when $Z$ is two-dimensional?
In its general form the E-step of the EM algorithm finds the expectation
$$ Q(\theta|\theta') =\int \log[ p(Y,Z | \theta)] p(Z|Y,\theta') d Z$$
where $Y$ the data, $Z$ the latent variables, $\theta'$...
1
vote
0answers
12 views
Optimisation problem with strange side conditions
I'm dealing with the following optimisation problem:
$$\min_{a \in H,\ b \in \mathbb{R}} F(a,b),$$
where $H$ is the solution space of another optimisation problem ($a$ is a real vector). I suppose as ...
0
votes
0answers
29 views
Estimate probabilites of a sum of independant binomial (with sizes fixed)
I collect observations which are sum of K (known parameter) independant binomial variables with known number of trials but unknown probability of success:
The number of trials varies for each ...
3
votes
1answer
64 views
Optimal selection with multiple parameters
I have a problem of optimal selection given many parameters. I will try to simplify my problem in this example below:
Suppose I have a basket with 100 balls in 10 different colors (not equally ...
0
votes
0answers
11 views
Is optimizing an embedding a convex or non-convex process?
Suppose we have input data with several thousand one-hot dimensions per element, representing, say, words in a passage of text. An embedding layer is a common feature at the top of machine learning ...
0
votes
0answers
10 views
Is penalized logistic regression convex? [duplicate]
Is logistic regression problem penalized by elastic net penalty convex optimization problem? More specifically, I want to find out whether it is suitable for dual formulation such that the duality gap ...
-3
votes
2answers
65 views
Why are frequentists uncomfortable with bayesian statistics when “optimization” algorithms used in frequentist statistics is bayesian?
In Step 1, we have a prior. Using bayes rule we construct the posterior.
In step 2 of some iterated bayesian procedure, the prior becomes the posterior from step one and use bayes rule to calculate ...
1
vote
0answers
17 views
How to minimize the sum of Frobenius norm and Nuclear norm
I have to minimize an objective function of the the form :
$||X_{s} - Y_{s}D_{s}||_{F}^{2} + ||D_{s}||_{F}^{2} + ||D_{s}||_{*}^{2}$
where $||.||_{F}$ denotes the Frobenius norm and $||.||_{*}$ ...
1
vote
0answers
44 views
Numerical Integration with respect to a mixture of Normals [closed]
I have a likelihood function that contains an integral of a latent parameter. I would like to numerically integrate it using Monte Carlo, as in, $L = \prod_{i=1}^N \int f(X, \tilde{\theta}; \beta) d ...
0
votes
1answer
20 views
What kind of model to use, with multiple target variables?
With normal Machine Learning we are focusing on 1 target variable. However, I have a case where I want to see the 'pattern' of the X-values/ predictors on 5 (instead of 1) target variable(s).
The ...
1
vote
1answer
48 views
Analytical solution for optimization problem
Is there an analytical solution for the following optimization problem:
$b, y \in \mathbb{R}^n$ (vectors), $b^* = \arg min_{b \in \mathbb{R}^n} f(b)$, where $f(b) = ||b-y||_{2}^{2} + \varphi ||b||_{...
1
vote
2answers
31 views
What exactly is meant by bias in this context?
I'm working through an example of survival-time analysis with censored and un-censored data.
We're given the survival times of 94 patients.
Some of these survival times are censored i.e.in this ...
0
votes
0answers
25 views
Is there a typo in this statement on the uniqueness of maximum likelihood estimators?
Working through a textbook on maximum likelihood estimation and the following is said:
"the uniqueness of the maximum likelihood estimator is not guaranteed, there may exist at least two parameter ...
1
vote
0answers
20 views
Hessian Matrix for MultiClass Softmax in Gradient Boosting (XGBoost): $2p_i(1-p_i)$
In the context of MultiClass Softmax, for a particular training instance, label and prediction $y, p \in \mathbb{R}^K$ (K categories).
The hessian matrix for Multiclass SoftMax with K categories is a $...
3
votes
1answer
32 views
Can the eigenvalues of the Hessian change sign during a Newton-Raphson optimization?
Can the eigenvalues of the Hessian change sign during a Newton-Raphson optimization?
Because from simple examples with 1D functions it seems as if one always converges to the nearest optimum, ...
2
votes
2answers
221 views
Difference between Feature engineering and hyperparameter optimizations?
Hyperparameter optimizations and feature engineering can(in my understanding) both be used to create a machine learning model. But what is the difference? And what is done to the y = wx + b formula in ...
0
votes
0answers
15 views
Optimization of a Correlation Matrix with constraints
I have 4 loss distributions (A,B,C,D) with a certain dependence structure described by a 4x4 correlation matrix.
As I'm interested in observing extreme losses behaviour, I calculate, for each ...
0
votes
2answers
36 views
With Ordinary Least Squares, how do we know that when the partial derivative of RSS is 0, that is a minima?
To minimise the residual sum of squares, we take its derivative with respect to the beta parameters and set this to 0. But when a derivative is set to 0, this means it can be one of the minima or one ...
1
vote
0answers
20 views
How valid is saying “improve x while not decreasing y” in business metric optimization?
Suppose you are an advertisement business, and you have two revenue streams x and y. Let's say the value of ...
3
votes
1answer
83 views
Iteratively Reweighted Least Squares, (Logistic Regression)
I'm trying to obtain the parameters estimates in a Logistic Regression using the IRLS (Iteratively Reweighted Least Squares) algorithm.
I'm following this great and simple reference slides: (Logistic ...
1
vote
3answers
120 views
Interview Question at Gaming Company
I came across the following interview question :
In an online gaming company, customer churn is defined in terms of the
number of days of continuous inactivity of the player. So how will you
...
0
votes
0answers
15 views
Is my train loss, test loss and dev loss normal?
I have two data sets D1 and D2. Each data set is composed of two parts: 1)sentences and 2)scores. I want to learn scores for the sentences by an LSTM network. I have examined four scenarios:
...
3
votes
2answers
437 views
How Does L2 Norm Regularization Work with Negative Weights?
L2 norm regularization penalizes large weights to avoid overfitting, basically by subtracting the magnitude of the weight vector (times a regularization parameter) from each weight during each update. ...
0
votes
0answers
54 views
Clustering products to optimize logistics
we are facing the following problem at my work. Our company is specialzed in retail and we experience enormous increase in webshop sales several times a year and have a hard time in satisfying all the ...
0
votes
0answers
28 views
Can we optimize heterogeneous parameters of RBF Network using Gradient Descent?
There're three parameters in the Radial Basis Function Networks (RBFN).
Centers of RBFs
Width of RBFs
Weights of RBFs
It's a fact that Weights can be easily updated using a simple Gradient Descent. ...
2
votes
3answers
63 views
Prove that the likelihood function L(θ|x) is equivalent to maximizing log L(θ|x) where log is the natural logarithm [closed]
In other words, why
$\text{argmax} \text{ } L(\theta) = \text{argmax} \text{ } \text{log} \text{ } L(\theta)$
?
0
votes
1answer
33 views
Difference between glm and optim for likelihood value on logistic regression in R
I still confuse my previous question on here1 and here2. About logLik of logistic regression in the case of proportion(=yes/yes+no). I try to validate it using optim() by following program. But it was ...
3
votes
4answers
1k views
Fitting SIR model with 2019-nCoV data doesn't conververge
I am trying to calculate the basic reproduction number $R_0$ of the new 2019-nCoV virus by fitting a SIR model to the current data. My code is based on https://arxiv.org/pdf/1605.01931.pdf, p. 11ff:
<...
0
votes
0answers
6 views
Finding and presenting solution for optimized supplier portfolio
I am trying to find the best solution to create a "model" and presentation of the following challenge:
I have several products which shall be bought from as few suppliers as possible. For each ...
3
votes
1answer
40 views
Generative design - algorithms and sources
I have recently encountered the term "generative design" where a computer algorithm (usually with the help of Machine Learning) comes up with new designs that conform to a certain set of requirements. ...
1
vote
0answers
20 views
off-policy evaluation in reinforcement learning
IPS estimator, which is used for off-policy evaluation in a contextual bandit problem, is well explained here: Doubly Robust Policy Evaluation andOptimization https://arxiv.org/pdf/1503.02834.pdf
The ...
3
votes
1answer
53 views
How to construct a function with given local minima?
I need to construct a function $f(x,y)$ in which there are 3 minima: 2 local and 1 global as given below.
Locals are: z = f(0.2,0.3) = 0.7 | z = f(0.6,0.8) = 0.8
Global is: z = f(0.85,0.5) = ...
1
vote
0answers
13 views
Early stopping in wilcoxon signed rank test?
I have decided to acquire N joint samples of variables X and Y (for example, N=200), and test whether their means are different using the signed-rank test. I will repeat this procedure many times, ...
-1
votes
1answer
47 views
Determining Profit for 150 different Treatments Types [closed]
My objective is to show which treatment or group of treatments was most successful at generating profit.
I have a data.frame with about 7 million observations.
Each of these observations could ...
2
votes
1answer
24 views
Some tricky details about PCA non-convexity
I am reading about PCA and I came up with a some contradictions. PCA based on this post, PCA optimization problem is given by:
$$\begin{aligned}
\max_{w} \quad & w^T\Sigma w\\
\textrm{s.t.} \...
1
vote
1answer
30 views
Regarding 0-1 loss and hinge loss functions of SVM
I would like to ask about SVM. I want to compare between 0-1 loss and hinge loss functions. My question is how to compare between them?!. Should we construct different SVM models, which each for ...
2
votes
1answer
44 views
Decomposing Gradient Decent Error in Eigenvector Space
I'm going through Why Momentum Really Works and am unable to understand the following line in the article.
"By writing the contributions of each eigenspace’s error to the loss
$$f(w^{k})-f(w^{\star})=...
5
votes
1answer
67 views
Statistical relationship between the stages of a stochastic optimization problem
What exactly do the "stages" of a stochastic program say about the statistical relationship between the problem variables?
From what I understand, the stages imply both an "ordering" and "grouping" ...
0
votes
2answers
28 views
Automatic differentiation for a function without representation
I have been studying AD for these days and I think I understand how it works, but all functions for which AD has been applied in the lectures I've studied are elementary in the mathematical sense, I ...
5
votes
1answer
115 views
Differentiable programming for general Bayesian decision theory
It is my understanding that differentiable programming and thus libraries like TensorFlow (e.g. TFP) and JAX can be used to solve Bayesian decision theory problems where e.g. we have a probabilistic ...
1
vote
0answers
14 views
Interested in optimizing the number of appointments based on the counts of two features
Key Facts:
Currently have a regression model (XGBoost) that takes 85 features and predicts the number of appointments.
There are two features that I am particularly interested in optimizing 2 ...
1
vote
0answers
11 views
Is there any proof of global convergence for 1D convex numeric optimization using cross entropy method?
Suppose we have the following 1D numeric optimization problem:
$min_{x} f(x)$ given $0< x \le x_{MAX}$ where $f(x)$ is a convex function. And I want to apply the cross-entropy method to optimize ...
0
votes
1answer
24 views
What is a reason that in Lasso Regression we can force all coefficients positive & intercept =0?
I have a regression problem where I need all coefficents to be positive and the intercept to be zero. I can do this in sklearn but i don't understand how the algoritm can force this conditions through ...
1
vote
0answers
35 views
Probability regression models
I have 2-d data points and have probabilities for those data points (which are of the order of 10e-8). Now I have some functional form of the probability distribution (which has a complicated ...
2
votes
2answers
41 views
Maximum Likelihood Estimator with exponential noise
So I need a little help with this please. I'm given N measurements of a signal $Y_{i} = A + v_{i}, i = 1,...,N$, where $v_{i}$ is measurement noise with the exponential pdf $f_{v}(v) = e^{-v}, v \geq ...
