Return to Answer

This is just how I always interpreted it, so I'd happily be corrected:

Their approach was suggested for the classic linear model $ y=\sum_j \beta_j x_j + \epsilon $ and they argued putting a spike and slap prior on the $\beta_j$.

If we have a categorical variable (class variable) $C$ with more than 2 categories $c_k, k =1, \dots, K, K>2$ we can embed them in the linear model by creating $K-1$ dummy variables that contrast $k-1$ categories with the reference category.

Let $x_j$ denote the $p$ metric variables and $d_s$ the $s=1,\dots, K-1$ dummy variables. Then the model becomes (I just use a single categorical variable):

$ y=\sum_j \beta_j x_j + \sum_s\gamma_s d_s + \epsilon $

with $\beta_j$ denoting the coefficients for the metric variables and $\gamma_s$ the coefficients for the dummy variables (note I only used $\beta$ and $\gamma$ for making their difference explicit).

If we now set up a spike and slap prior for the $\beta$ and the $\gamma$ we have for the metric variables

$ P(\beta_j=0) = h_{0j} \\ P(\beta_j<b,\beta_j\neq0)=(b+f_j)h_{1j}\\ P(|\beta_j|>f_j)=0 $

and for the dummies

$ P(\gamma_s=0) = g_{0s} \\ P(\gamma_s<b,\gamma_s\neq0)=(b+r_s)g_{1s}\\ P(|\gamma_s|>r_s)=0 $

with the definition of the $f_j$ and $r_s$ as in their paper.

The key to my interpretation is now that we have $K-1$ dummies, $K-1$ $\gamma_s$ and $K-1$ "spike and slaps" and the prior over the submodels (2.7 in their paper) is therefore a product over all $K$ categories. In other words, the selection happens on the level of the coefficients for the dummies and therefore on the level of the $K-1$ categories of $C$, not on the variable $C$ itself. $C$ is therefore always selected whenever a category of $C$ is selected. Put differently, $C$ is only excluded from the model when all $\gamma_s$ are shrunk to zero. For class variables with more than 2 categories, their approach therefore does coefficient selection rather than variable selection.

This is just how I always interpreted it, so I'd happily be corrected:

Their approach was suggested for the classic linear model $ y=\sum_j \beta_j x_j + \epsilon $ and they argued putting a spike and slap prior on the $\beta_j$.

If we have a categorical variable (class variable) $C$ with more than 2 categories $c_k, k =1, \dots, K, K>2$ we can embed them in the linear model by creating $K-1$ dummy variables that contrast $k-1$ categories with the reference category.

Let $x_j$ denote the $p$ metric variables and $d_s$ the $s=1,\dots, K-1$ dummy variables. Then the model becomes (I just use a single categorical variable):

$ y=\sum_j \beta_j x_j + \sum_s\gamma_s d_s + \epsilon $

with $\beta_j$ denoting the coefficients for the metric variables and $\gamma_s$ the coefficients for the dummy variables (note I only used $\beta$ and $\gamma$ for making their difference explicit).

If we now set up a spike and slap prior for the $\beta$ and the $\gamma$ we have for the metric variables

$ P(\beta_j=0) = h_{0j} \\ P(\beta_j<b,\beta_j\neq0)=(b+f_j)h_{1j}\\ P(|\beta_j|>f_j)=0 $

and for the dummies

$ P(\gamma_s=0) = g_{0s} \\ P(\gamma_s<b,\gamma_s\neq0)=(b+r_s)g_{1s}\\ P(|\gamma_s|>r_s)=0 $

with the definition of the $f_j$ and $r_s$ as in their paper.

The key is now that we have $K-1$ dummies, $K-1$ $\gamma_s$ and $K-1$ "spike and slaps" and the prior over the submodels (2.7 in their paper) is therefore a product over all $K-1$ coefficients. In other words, the selection happens on the level of the coefficients for the dummies and therefore on the level of the $K-1$ categories of $C$, not on the variable $C$ itself. The shrinkage therefore refers to the difference between category $c_k$ and the reference category.

This has a number of implications:

• Shrinkage depends on the coding scheme

• The choice of the reference category matters

• Selection only refers to the currently chosen reference category (but in class variables that should be arbitrary)

• The selected models are not invariant against permutations of class labels

$C$ is therefore always selected whenever a category difference of two levels of $C$ is selected. Put differently, $C$ is only excluded from the model when all $\gamma_s$ are shrunk to zero. For class variables with more than 2 categories, their approach therefore does non-invariant coefficient selection rather than variable selection.

This is just how I always interpreted it, so I'd happily be corrected:

Their approach was suggested for the classic linear model $ y=\sum_j \beta_j x_j + \epsilon $ and they argued putting a spike and slap prior on the $\beta_j$.

If we have a categorical variable (class variable) $C$ with more than 2 categories $c_k, k =1, \dots, K, K>2$ we can embed them in the linear model by creating $K-1$ dummy variables that contrast $k-1$ categories with the reference category.

Let $x_j$ denote the $p$ metric variables and $d_s$ the $s=1,\dots, K-1$ dummy variables. Then the model becomes (I just use a single categorical variable):

$ y=\sum_j \beta_j x_j + \sum_s\gamma_s d_s + \epsilon $

with $\beta_j$ denoting the coefficients for the metric variables and $\gamma_s$ the coefficients for the dummy variables (note I only used $\beta$ and $\gamma$ for making their difference explicit).

If we now set up a spike and slap prior for the $\beta$ and the $\gamma$ we have for the metric variables

$ P(\beta_j=0) = h_{0j} \\ P(\beta_j<b,\beta_j\neq0)=(b+f_j)h_{1j}\\ P(|\beta_j|>f_j)=0 $

and for the dummies

$ P(\gamma_s=0) = g_{0s} \\ P(\gamma_s<b,\gamma_s\neq0)=(b+r_s)g_{1s}\\ P(|\gamma_s|>r_s)=0 $

with the definition of the $f_j$ and $r_s$ as in their paper.

The key to my interpretation is now that we have $K-1$ dummies, $K-1$ $\gamma_s$ and $K-1$ "spike and slaps" and the prior over the submodels (2.7 in their paper) is therefore a product over all $K$ categories. In other words, the variable selection happens on the dummies and therefore on the level of the $K-1$ categories of $C$, not on the variable $C$ itself. $C$ is therefore always selected whenever a category of $C$ is selected. Put differently, $C$ is only excluded from the model when all $\gamma_s$ are shrunk to zero. For class variables with more than 2 categories, their approach therefore does category selection rather than variable selection.

This is just how I always interpreted it, so I'd happily be corrected:

Their approach was suggested for the classic linear model $ y=\sum_j \beta_j x_j + \epsilon $ and they argued putting a spike and slap prior on the $\beta_j$.

If we have a categorical variable (class variable) $C$ with more than 2 categories $c_k, k =1, \dots, K, K>2$ we can embed them in the linear model by creating $K-1$ dummy variables that contrast $k-1$ categories with the reference category.

Let $x_j$ denote the $p$ metric variables and $d_s$ the $s=1,\dots, K-1$ dummy variables. Then the model becomes (I just use a single categorical variable):

$ y=\sum_j \beta_j x_j + \sum_s\gamma_s d_s + \epsilon $

with $\beta_j$ denoting the coefficients for the metric variables and $\gamma_s$ the coefficients for the dummy variables (note I only used $\beta$ and $\gamma$ for making their difference explicit).

If we now set up a spike and slap prior for the $\beta$ and the $\gamma$ we have for the metric variables

$ P(\beta_j=0) = h_{0j} \\ P(\beta_j<b,\beta_j\neq0)=(b+f_j)h_{1j}\\ P(|\beta_j|>f_j)=0 $

and for the dummies

$ P(\gamma_s=0) = g_{0s} \\ P(\gamma_s<b,\gamma_s\neq0)=(b+r_s)g_{1s}\\ P(|\gamma_s|>r_s)=0 $

with the definition of the $f_j$ and $r_s$ as in their paper.

The key to my interpretation is now that we have $K-1$ dummies, $K-1$ $\gamma_s$ and $K-1$ "spike and slaps" and the prior over the submodels (2.7 in their paper) is therefore a product over all $K$ categories. In other words, the selection happens on the level of the coefficients for the dummies and therefore on the level of the $K-1$ categories of $C$, not on the variable $C$ itself. $C$ is therefore always selected whenever a category of $C$ is selected. Put differently, $C$ is only excluded from the model when all $\gamma_s$ are shrunk to zero. For class variables with more than 2 categories, their approach therefore does coefficient selection rather than variable selection.

This is just how I always interpreted it, so I'd happily be corrected:

Their approach was suggested for the classic linear model $ y=\sum_j \beta_j x_j + \epsilon $ and they argued putting a spike and slap prior on the $\beta_j$.

If we have a categorical variable (class variable) $C$ with more than 2 categories $c_k, k =1, \dots, K, K>2$ we can embed them in the linear model by creating $K-1$ dummy variables that contrast $k-1$ categories with the reference category.

Let $x_j$ denote the $p$ metric variables and $d_s$ the $s=1,\dots, K-1$ dummy variables. Then the model becomes (I just use a single categorical variable):

$ y=\sum_j \beta_j x_j + \sum_s\gamma_s d_s + \epsilon $

with $\beta_j$ denoting the coefficients for the metric variables and $\gamma_s$ the coefficients for the dummy variables (note I only used $\beta$ and $\gamma$ for making their difference explicit).

If we now set up a spike and slap prior for the $\beta$ and the $\gamma$ we have for the metric variables

$ P(\beta_j=0) = h_{0j} \\ P(\beta_j<b,\beta_j\neq0)=(b+f_j)h_{1j}\\ P(|\beta_j|>f_j)=0 $

and for the dummies

$ P(\gamma_s=0) = g_{0s} \\ P(\gamma_s<b,\gamma_s\neq0)=(b+r_s)g_{1s}\\ P(|\beta_s|>r_s)=0 $

with the definition of the $f_j$ and $r_s$ as in their paper.

The key to my interpretation is now that we have $K-1$ dummies, $K-1$ $\gamma_s$ and $K-1$ "spike and slaps" and the prior over the submodels (2.7 in their paper) is therefore a product over all $K$ categories. In other words, the variable selection happens on the dummies and therefore on the level of the $K-1$ categories of $C$, not on the variable $C$ itself. $C$ is therefore always selected whenever a category of $C$ is selected. Put differently, $C$ is only excluded from the model when all $\gamma_s$ are shrunk to zero. For class variables with more than 2 categories, their approach therefore does category selection rather than variable selection.

This is just how I always interpreted it, so I'd happily be corrected:

Their approach was suggested for the classic linear model $ y=\sum_j \beta_j x_j + \epsilon $ and they argued putting a spike and slap prior on the $\beta_j$.

If we have a categorical variable (class variable) $C$ with more than 2 categories $c_k, k =1, \dots, K, K>2$ we can embed them in the linear model by creating $K-1$ dummy variables that contrast $k-1$ categories with the reference category.

Let $x_j$ denote the $p$ metric variables and $d_s$ the $s=1,\dots, K-1$ dummy variables. Then the model becomes (I just use a single categorical variable):

$ y=\sum_j \beta_j x_j + \sum_s\gamma_s d_s + \epsilon $

with $\beta_j$ denoting the coefficients for the metric variables and $\gamma_s$ the coefficients for the dummy variables (note I only used $\beta$ and $\gamma$ for making their difference explicit).

If we now set up a spike and slap prior for the $\beta$ and the $\gamma$ we have for the metric variables

$ P(\beta_j=0) = h_{0j} \\ P(\beta_j<b,\beta_j\neq0)=(b+f_j)h_{1j}\\ P(|\beta_j|>f_j)=0 $

and for the dummies

$ P(\gamma_s=0) = g_{0s} \\ P(\gamma_s<b,\gamma_s\neq0)=(b+r_s)g_{1s}\\ P(|\gamma_s|>r_s)=0 $

with the definition of the $f_j$ and $r_s$ as in their paper.

The key to my interpretation is now that we have $K-1$ dummies, $K-1$ $\gamma_s$ and $K-1$ "spike and slaps" and the prior over the submodels (2.7 in their paper) is therefore a product over all $K$ categories. In other words, the variable selection happens on the dummies and therefore on the level of the $K-1$ categories of $C$, not on the variable $C$ itself. $C$ is therefore always selected whenever a category of $C$ is selected. Put differently, $C$ is only excluded from the model when all $\gamma_s$ are shrunk to zero. For class variables with more than 2 categories, their approach therefore does category selection rather than variable selection.