How to obtain covariance matrix for constrained regression fit?

Is there an easy way to obtain the covariance of the parameters from a constrained regression fit?

I am using the PCLS function in the MGCV package in R to fit the constrained regression, however I am open to other approaches. The constraint I am imposing is that the coefficients must be positive.

-
How large is your sample? –  jbowman Jun 20 '12 at 16:09
Aren't you concerned that a covariance matrix is unlikely to be a useful description of parameter uncertainty for any estimates at or close to the constraints? –  whuber Jun 22 '12 at 16:12
The sample is around 500. @whuber Yes that is a concern. And since I know some estimates will be close to the constraints it may not even make sense to think of a covariance matrix. However since the PCLS function does some sort of optimization procedure, you would think you could get the Hessian matrix, which would provide some info. –  Glen Jul 6 '12 at 1:55
add comment

1 Answer

At first I would go with very simple bootstrap.

Basically something as follows:

1. Create a new data-set by resampling pairs of $(x,y)$.
2. Run your regression on this new data set and you will get some parameters $\hat \beta$.
3. Repeat 1 and 2 as many times as possible. Now you will have a large sets of $\hat \beta$
4. Now just take the sample covariance of your $\hat \beta$.
5. Done
-
This is what I was thinking. Can you elaborate a bit of step 1 though? Re-sample with replacement? Or...? –  Macro Jun 18 '12 at 15:47
Yes. Vanilla bootstrap is sampling with replacement. so for example if your data is $(x,y) = \{(1,1);(1,2);(1,3)\}$ a new data set would be $(1,3);(1,3);(1,1)$. This is not the only kind of resampling possible of course. Another one is to reuse all the $X$ and shocking all $Y$ by a noise. But pair resampling is the simplest and most agnostic –  CarrKnight Jun 18 '12 at 17:00
I read a lot on bootstrap but I never heard the term vanilla bootstrap. Maybe it is equivalent to what is called the ordinary or naive bootstrap. As far as bootstrap pairs vs bootstrapping residuals, bootstrapping pairs is simpler but what do you mean by agnostic (not heavily model-dependent?)? –  Michael Chernick Jun 18 '12 at 19:41
@MichaelChernick: Vanilla is a colloquial term for the "regular", "usual", or "simple" version of something. –  cardinal Jun 18 '12 at 20:50
@MichaelChernick: I am not sure how much of a regional colloquialism that use of vanilla might be. (I figured you'd likely have heard it.) At any rate, what I meant (but didn't say very clearly) is that I'm almost positive CarrKnight was not using the term in some bootstrap-specific way. :) –  cardinal Jun 18 '12 at 21:01
show 3 more comments