Questions tagged [irls]
IRLS stands for Iteratively Re-weighted Least Squares. IRLS is a commonly used method to find maximum likelihood estimates when they cannot be found analytically.
33
questions
4
votes
2answers
111 views
Are GLMs just glorified WLS regressions?
When performing weighted least squares $L = \frac{1}{2} \sum_i w_i r_i^2$, Aitken showed that one ought to weight each sample by the inverse of its variance $w_i=1/\sigma_i^2$. This leads to gradients ...
4
votes
0answers
119 views
IRLS for truncated normal GLM
I have data for which responses fall in $y \in [0,\infty)$ for which, it seems, the standard GLMs based on, say, gamma or inverse-Gaussian fail since they don't allow responses with values equal to 0. ...
3
votes
1answer
55 views
Can the maximum-likelihood method be derived from something else?
I am an author of a paper, in which we show that the maximum-likelihood (ML) method can be derived a limiting case of an iterated weighted least-squares fit. https://arxiv.org/abs/1807.07911
We, the ...
5
votes
1answer
67 views
Heteroscedasticity that depends on the regression parameters
Consider a vector of observations $\mathbf{Y}$ that can be modeled as
\begin{equation}
\mathbf{Y} \sim \mathcal{N}( \mathbf{H}\boldsymbol{\beta} , \boldsymbol{\Sigma} )
\end{equation}
with $\mathbf{...
17
votes
1answer
3k views
Why using Newton's method for logistic regression optimization is called iterative re-weighted least squares?
Why using Newton's method for logistic regression optimization is called iterative re-weighted least squares?
It seems not clear to me because logistic loss and least squares loss are completely ...
5
votes
1answer
181 views
Strange variance weights for Poisson GLM for square root link
Is there a reason why all the variance weights from a Poisson GLM are equal when a square root link is used? That is (on R):
...
1
vote
0answers
633 views
Iterative reweighted least squares versus MLE for heteroscedastic errors
Iterative reweighted least squares (IRLS) is used when errors are heteroscedastic. Let us assume that error comes from a distribution where its mean is zero and the variance is a function of the ...
3
votes
1answer
926 views
Iteratively Reweighted Least squares for logistic regression when features are dependent?
I was solving logistic regression using IRLS (wiki) described in the wiki link. Now I have a doubt, if $X$ has dependent features then $X^TS_kX$ will not have full rank and thus will not be invertible ...
2
votes
1answer
484 views
Difference between robust regression and weighted regression
In stata, robust regression (rreg) uses weights proportional to the size of the residuals. Is this conceptually the same as weighted OLS (weight by 1/variance)? And both can be applied, for example, ...
2
votes
1answer
301 views
Iteratively reweighted least squares in Machine Learning Probability perspective
I am studying Machine Learning Probability perspective and I have a question with Algorithm 8.2, which is
$\mathbf{w} = 0_{D}$ $w_{0} = \log (\bar{y}/(1-\bar{y}))$
Repeat:
&...
2
votes
1answer
449 views
Derivation of the Iterative Reweighted Least Squares Solution for $ {L}_{1} $ Regularized Least Squares Problem
I'm trying to fitting a line with IRLS with L1 norm, but I'm struggling to understand why my idea is wrong.
1 - init the weights $w$
2 - fit with simple LS and obtain a starting model $\beta_0$
3 - ...
1
vote
1answer
583 views
Initialization of weights in IRLS (Iteratively reweighted least squares)
How to initialize weights in IRLS?
On the wikipedia page about IRLS it is stated that:
$W^{(t)}$ is the diagonal matrix of weights, usually with all elements set initially to: $w^{(0)}_i = 1$
but ...
12
votes
1answer
3k views
Can you give a simple intuitive explanation of IRLS method to find the MLE of a GLM?
Background:
I'm trying to follow Princeton's review of MLE estimation for GLM.
I understand the basics of MLE estimation: likelihood, ...
4
votes
2answers
9k views
glmer vs lmer, what is best for a binomial outcome?
I am trying to fit a mixed effects model with a binary outcome. I have one fixed effect (Offset) and one random effect (chamber, with muliple data points coming from each chamber).
In the text book "...
0
votes
2answers
162 views
a question about solving logistic regression
While studying the slides [Link] (https://www.dropbox.com/s/rdwzjjah9f2mb2j/Logistic%20Regression%20to%20ILRS.pdf?dl=0) on logistic regression, I faced a question.
In slide 15 and 16 it is stated ...
1
vote
0answers
92 views
Reconcile Different Statements of the IRWLS Algorithm
Different textbooks seem to have different definitions for the weight matrix W in the Iteratively Re-Weighted Least Squares (IRWLS) algorithm. They should be equivalent but I can't seem to piece it ...
1
vote
0answers
1k views
Weighted ridge regression in R
How can we do weighted ridge regression in R?
In MASS package in R, I can do weighted linear regression by passing a weight parameter to lm. It can be seen that ...
1
vote
0answers
420 views
Fitting heteroscedastic models using the `gls` function
Consider the following heteroscedastic model:
$$y_i = f(x_i, \beta) + g(x_i, \theta)\varepsilon_i, i = 1, \ldots, n, \tag{1}$$
where $f(\cdot, \beta)$ is the regression function and $g(\cdot, \theta)$
...
1
vote
0answers
287 views
Fitting a glm in practice
This question will be a little wordy - I'll try to summarize at the end.
I'm currently working on a machine learning library and I'm implementing GLMs. To fit my models I've been implementing an ...
0
votes
1answer
2k views
Choosing IRLS over gradient descent in logistic regression
I am currently reading Bishop [1] and got confusion on why should we take IRLS (Iterative Re-weighted Least Square) as it seems that using gradient descent that with one derivative at a time would ...
3
votes
1answer
806 views
Why should we use IRLS in logistic regression?
I am really confused on why should we take IRLS as it seems that using gradient that with one derivatives at a time would solve the problem, what is the meaning of introducing Hessian matrix? Or did I ...
2
votes
0answers
92 views
Intuitive explanation of failed convergence for GLM
Sometimes the software complains about some matrix not being positive definite. Is it the singularity that's causing issues (because the iterative algorithm involves inverting matrices)? Also, what ...
7
votes
1answer
183 views
If $\operatorname{Var}\left(\epsilon_i\right) = h\left(X\right) \neq \sigma^2$, what can we know about $\operatorname{Var}\left(\hat{\beta}\right)$?
This question uses the derivations found here.
The short version
Consider a regression model. If the error variance is a known function of the data (rather than a constant), under what conditions ...
7
votes
1answer
10k views
How to correctly implement iteratively reweighted least squares algorithm for multiple logistic regression?
I'm confused about the iteratively reweighted least squares algorithm used to solve for logistic regression coefficients as described on page 121 of The Elements of Statistical Learning, 2nd Edition (...
8
votes
2answers
3k views
What are some reasons iteratively reweighted least squares would not converge when used for logistic regression?
I've been using the glm.fit function in R to fit parameters to a logistic regression model. By default, glm.fit uses iteratively reweighted least squares to fit the parameters. What are some reasons ...
2
votes
0answers
608 views
Poisson Regression Model [duplicate]
Kindly explain me how to estimate poisson regression model using iterative weighted least square method. I know it can be easily estimated in any software like Stata, SAS, SPSS etc but I want to ...
1
vote
0answers
372 views
What is wrong with this implementation of logistic regression (using iterative reweighted least squares)?
I am trying to implement logistic regression using the following algorithm:
fit a simple linear model $y \sim Xb_0$
calculate $W = \frac{e^{Xb_0}}{(1+e^{Xb_0})^2}$
calculate $z = Xb_0 + y \cdot \frac{...
1
vote
0answers
231 views
Iteratively reweighted least squares : asymmetric weights
For robust m-estimation, all the convergence results I'm aware of assume symmetric weights (eg: Huber function) in their formulation of the iterative reweighted least squares algorithm. Does the ...
1
vote
1answer
4k views
Logistic regression: Fisher's scoring iterations do not match the selected iterations in glm
it happened to me that in a logistic regression in R with glm the Fisher scoring iterations in the output are less than the iterations selected with the argument <...
35
votes
2answers
35k views
Purpose of the link function in generalized linear model
What is the purpose of the link function as a component of the generalized linear model? Why do we need it?
Wikipedia states:
It can be convenient to match the domain of the link function to the ...
16
votes
1answer
2k views
Definition and Convergence of Iteratively Reweighted Least Squares
I've been using iteratively reweighted least squares (IRLS) to minimize functions of the following form,
$J(m) = \sum_{i=1}^{N} \rho \left(\left| x_i - m \right|\right)$
where $N$ is the number of ...
1
vote
0answers
73 views
Is the symmetry of u function for the robust M estimation mandatory?
I have build a $\rho$ function which has the following definition:
\begin{equation}
\rho(x)=
\left\{
\begin{array}{ll}
4- \frac{8}{x^2} \text{if } x \lt-2\\
\frac{x^2}{2} \text{if } x \in [-2,3]...
9
votes
4answers
865 views
Standard algorithms for doing hierarchical linear regression?
Are there standard algorithms (as opposed to programs) for doing hierarchical linear regression? Do people usually just do MCMC or are there more specialized, perhaps partially closed form, algorithms?...
