The Stack Overflow podcast is back! Listen to an interview with our new CEO.

Questions tagged [irls]

IRLS stands for Iteratively Re-weighted Least Squares. IRLS is a commonly used method to find maximum likelihood estimates when they cannot be found analytically.

Filter by
Sorted by
Tagged with
4
votes
2answers
111 views

Are GLMs just glorified WLS regressions?

When performing weighted least squares $L = \frac{1}{2} \sum_i w_i r_i^2$, Aitken showed that one ought to weight each sample by the inverse of its variance $w_i=1/\sigma_i^2$. This leads to gradients ...
4
votes
0answers
119 views

IRLS for truncated normal GLM

I have data for which responses fall in $y \in [0,\infty)$ for which, it seems, the standard GLMs based on, say, gamma or inverse-Gaussian fail since they don't allow responses with values equal to 0. ...
3
votes
1answer
55 views

Can the maximum-likelihood method be derived from something else?

I am an author of a paper, in which we show that the maximum-likelihood (ML) method can be derived a limiting case of an iterated weighted least-squares fit. https://arxiv.org/abs/1807.07911 We, the ...
5
votes
1answer
67 views

Heteroscedasticity that depends on the regression parameters

Consider a vector of observations $\mathbf{Y}$ that can be modeled as \begin{equation} \mathbf{Y} \sim \mathcal{N}( \mathbf{H}\boldsymbol{\beta} , \boldsymbol{\Sigma} ) \end{equation} with $\mathbf{...
17
votes
1answer
3k views

Why using Newton's method for logistic regression optimization is called iterative re-weighted least squares?

Why using Newton's method for logistic regression optimization is called iterative re-weighted least squares? It seems not clear to me because logistic loss and least squares loss are completely ...
5
votes
1answer
181 views

Strange variance weights for Poisson GLM for square root link

Is there a reason why all the variance weights from a Poisson GLM are equal when a square root link is used? That is (on R): ...
1
vote
0answers
633 views

Iterative reweighted least squares versus MLE for heteroscedastic errors

Iterative reweighted least squares (IRLS) is used when errors are heteroscedastic. Let us assume that error comes from a distribution where its mean is zero and the variance is a function of the ...
3
votes
1answer
926 views

Iteratively Reweighted Least squares for logistic regression when features are dependent?

I was solving logistic regression using IRLS (wiki) described in the wiki link. Now I have a doubt, if $X$ has dependent features then $X^TS_kX$ will not have full rank and thus will not be invertible ...
2
votes
1answer
484 views

Difference between robust regression and weighted regression

In stata, robust regression (rreg) uses weights proportional to the size of the residuals. Is this conceptually the same as weighted OLS (weight by 1/variance)? And both can be applied, for example, ...
2
votes
1answer
301 views

Iteratively reweighted least squares in Machine Learning Probability perspective

I am studying Machine Learning Probability perspective and I have a question with Algorithm 8.2, which is $\mathbf{w} = 0_{D}$ $w_{0} = \log (\bar{y}/(1-\bar{y}))$ Repeat:   &...
2
votes
1answer
449 views

Derivation of the Iterative Reweighted Least Squares Solution for $ {L}_{1} $ Regularized Least Squares Problem

I'm trying to fitting a line with IRLS with L1 norm, but I'm struggling to understand why my idea is wrong. 1 - init the weights $w$ 2 - fit with simple LS and obtain a starting model $\beta_0$ 3 - ...
1
vote
1answer
583 views

Initialization of weights in IRLS (Iteratively reweighted least squares)

How to initialize weights in IRLS? On the wikipedia page about IRLS it is stated that: $W^{(t)}$ is the diagonal matrix of weights, usually with all elements set initially to: $w^{(0)}_i = 1$ but ...
12
votes
1answer
3k views

Can you give a simple intuitive explanation of IRLS method to find the MLE of a GLM?

Background: I'm trying to follow Princeton's review of MLE estimation for GLM. I understand the basics of MLE estimation: likelihood, ...
4
votes
2answers
9k views

glmer vs lmer, what is best for a binomial outcome?

I am trying to fit a mixed effects model with a binary outcome. I have one fixed effect (Offset) and one random effect (chamber, with muliple data points coming from each chamber). In the text book "...
0
votes
2answers
162 views

a question about solving logistic regression

While studying the slides [Link] (https://www.dropbox.com/s/rdwzjjah9f2mb2j/Logistic%20Regression%20to%20ILRS.pdf?dl=0) on logistic regression, I faced a question. In slide 15 and 16 it is stated ...
1
vote
0answers
92 views

Reconcile Different Statements of the IRWLS Algorithm

Different textbooks seem to have different definitions for the weight matrix W in the Iteratively Re-Weighted Least Squares (IRWLS) algorithm. They should be equivalent but I can't seem to piece it ...
1
vote
0answers
1k views

Weighted ridge regression in R

How can we do weighted ridge regression in R? In MASS package in R, I can do weighted linear regression by passing a weight parameter to lm. It can be seen that ...
1
vote
0answers
420 views

Fitting heteroscedastic models using the `gls` function

Consider the following heteroscedastic model: $$y_i = f(x_i, \beta) + g(x_i, \theta)\varepsilon_i, i = 1, \ldots, n, \tag{1}$$ where $f(\cdot, \beta)$ is the regression function and $g(\cdot, \theta)$ ...
1
vote
0answers
287 views

Fitting a glm in practice

This question will be a little wordy - I'll try to summarize at the end. I'm currently working on a machine learning library and I'm implementing GLMs. To fit my models I've been implementing an ...
0
votes
1answer
2k views

Choosing IRLS over gradient descent in logistic regression

I am currently reading Bishop [1] and got confusion on why should we take IRLS (Iterative Re-weighted Least Square) as it seems that using gradient descent that with one derivative at a time would ...
3
votes
1answer
806 views

Why should we use IRLS in logistic regression?

I am really confused on why should we take IRLS as it seems that using gradient that with one derivatives at a time would solve the problem, what is the meaning of introducing Hessian matrix? Or did I ...
2
votes
0answers
92 views

Intuitive explanation of failed convergence for GLM

Sometimes the software complains about some matrix not being positive definite. Is it the singularity that's causing issues (because the iterative algorithm involves inverting matrices)? Also, what ...
7
votes
1answer
183 views

If $\operatorname{Var}\left(\epsilon_i\right) = h\left(X\right) \neq \sigma^2$, what can we know about $\operatorname{Var}\left(\hat{\beta}\right)$?

This question uses the derivations found here. The short version Consider a regression model. If the error variance is a known function of the data (rather than a constant), under what conditions ...
7
votes
1answer
10k views

How to correctly implement iteratively reweighted least squares algorithm for multiple logistic regression?

I'm confused about the iteratively reweighted least squares algorithm used to solve for logistic regression coefficients as described on page 121 of The Elements of Statistical Learning, 2nd Edition (...
8
votes
2answers
3k views

What are some reasons iteratively reweighted least squares would not converge when used for logistic regression?

I've been using the glm.fit function in R to fit parameters to a logistic regression model. By default, glm.fit uses iteratively reweighted least squares to fit the parameters. What are some reasons ...
2
votes
0answers
608 views

Poisson Regression Model [duplicate]

Kindly explain me how to estimate poisson regression model using iterative weighted least square method. I know it can be easily estimated in any software like Stata, SAS, SPSS etc but I want to ...
1
vote
0answers
372 views

What is wrong with this implementation of logistic regression (using iterative reweighted least squares)?

I am trying to implement logistic regression using the following algorithm: fit a simple linear model $y \sim Xb_0$ calculate $W = \frac{e^{Xb_0}}{(1+e^{Xb_0})^2}$ calculate $z = Xb_0 + y \cdot \frac{...
1
vote
0answers
231 views

Iteratively reweighted least squares : asymmetric weights

For robust m-estimation, all the convergence results I'm aware of assume symmetric weights (eg: Huber function) in their formulation of the iterative reweighted least squares algorithm. Does the ...
1
vote
1answer
4k views

Logistic regression: Fisher's scoring iterations do not match the selected iterations in glm

it happened to me that in a logistic regression in R with glm the Fisher scoring iterations in the output are less than the iterations selected with the argument <...
35
votes
2answers
35k views

Purpose of the link function in generalized linear model

What is the purpose of the link function as a component of the generalized linear model? Why do we need it? Wikipedia states: It can be convenient to match the domain of the link function to the ...
16
votes
1answer
2k views

Definition and Convergence of Iteratively Reweighted Least Squares

I've been using iteratively reweighted least squares (IRLS) to minimize functions of the following form, $J(m) = \sum_{i=1}^{N} \rho \left(\left| x_i - m \right|\right)$ where $N$ is the number of ...
1
vote
0answers
73 views

Is the symmetry of u function for the robust M estimation mandatory?

I have build a $\rho$ function which has the following definition: \begin{equation} \rho(x)= \left\{ \begin{array}{ll} 4- \frac{8}{x^2} \text{if } x \lt-2\\ \frac{x^2}{2} \text{if } x \in [-2,3]...
9
votes
4answers
865 views

Standard algorithms for doing hierarchical linear regression?

Are there standard algorithms (as opposed to programs) for doing hierarchical linear regression? Do people usually just do MCMC or are there more specialized, perhaps partially closed form, algorithms?...