Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 5 down vote favorite

Is there an easy way to obtain the covariance of the parameters from a constrained regression fit?

I am using the PCLS function in the MGCV package in R to fit the constrained regression, however I am open to other approaches. The constraint I am imposing is that the coefficients must be positive.

share|improve this question
How large is your sample? – jbowman Jun 20 '12 at 16:09
Aren't you concerned that a covariance matrix is unlikely to be a useful description of parameter uncertainty for any estimates at or close to the constraints? – whuber♦ Jun 22 '12 at 16:12
The sample is around 500. @whuber Yes that is a concern. And since I know some estimates will be close to the constraints it may not even make sense to think of a covariance matrix. However since the PCLS function does some sort of optimization procedure, you would think you could get the Hessian matrix, which would provide some info. – Glen Jul 6 '12 at 1:55

1 Answer

up vote 3 down vote accepted
+50

At first I would go with very simple bootstrap.

Basically something as follows:

  1. Create a new data-set by resampling pairs of $(x,y)$.
  2. Run your regression on this new data set and you will get some parameters $\hat \beta$.
  3. Repeat 1 and 2 as many times as possible. Now you will have a large sets of $ \hat \beta$
  4. Now just take the sample covariance of your $\hat \beta$.
  5. Done
share|improve this answer
This is what I was thinking. Can you elaborate a bit of step 1 though? Re-sample with replacement? Or...? – Macro Jun 18 '12 at 15:47
Yes. Vanilla bootstrap is sampling with replacement. so for example if your data is $(x,y) = \{(1,1);(1,2);(1,3)\}$ a new data set would be $(1,3);(1,3);(1,1)$. This is not the only kind of resampling possible of course. Another one is to reuse all the $X$ and shocking all $Y$ by a noise. But pair resampling is the simplest and most agnostic – CarrKnight Jun 18 '12 at 17:00
I read a lot on bootstrap but I never heard the term vanilla bootstrap. Maybe it is equivalent to what is called the ordinary or naive bootstrap. As far as bootstrap pairs vs bootstrapping residuals, bootstrapping pairs is simpler but what do you mean by agnostic (not heavily model-dependent?)? – Michael Chernick Jun 18 '12 at 19:41
3  
@MichaelChernick: Vanilla is a colloquial term for the "regular", "usual", or "simple" version of something. – cardinal Jun 18 '12 at 20:50
1  
@MichaelChernick: I am not sure how much of a regional colloquialism that use of vanilla might be. (I figured you'd likely have heard it.) At any rate, what I meant (but didn't say very clearly) is that I'm almost positive CarrKnight was not using the term in some bootstrap-specific way. :) – cardinal Jun 18 '12 at 21:01
show 3 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.