Model selection in the classification problem with costly information Let's assume we have a $X_T$ matrix of $N$ variables and $Y_T$ available for training a model to solve classification problem for variable $y$.
Normally, we can use all $N$ variables for training and then collect all $N$ variables to predict $y$.
But in real life, obtaining all variables to predict $y$ is expensive, even if the model uses only a subset of $X_T$ and shows good statistical performance. Suppose we predict a loan default and each piece of information on the borrower requires a separate phone call. If we make many calls, we have a lot of information, but it's not free. So, we have to balance between the quality of prediction and the cost of information.
How should I choose a model in this case, given matrix $X_T$ and the cost of each variable? Are there any formal approaches, articles on this? 
 A: I presume we are assuming each feature of $X$ to be equally expensive to collect. In that case, we are interested in a supervised learning feature selection algorithm which is sparse in nature. Sparsity refers to having a smaller overall number of features in a predictive model. Aside from cost, it also has the practical application that it is easier to understand by nontechnical experts as opposed to, say, partial least squares which has an estimated effect for each feature in the predictive model.
Sparse feature selection can be achieved with Lasso. Lasso regularizes  regression models with an L-1 penalty. This constrains the model in terms of the total L1 area of the coefficient vector. What we observe is that features which have values that are very close to zero tend to be set to zero most often. You can "tune" the penalty factor, $\lambda$ to select a desired number of features. 
To the best of my knowledge, there is no weighted analogue of this, but it would be easy to implement and could be an interesting area of research. The basic idea of a weighted penalty would be:
For a matrix of features $X_1, \ldots X_p$ having associated collection costs $w_1, \ldots w_p$ having $\sum_{i=1}^p w_i = 1$. for collecting a single sample, the objective function for the weighted Lasso would be given by:
$$ \rho(\beta) = \sum_{i=1}^n \left( Y_i - \mathbf{X}^T \vec{\beta} \right) ^2 + w^T  \left| \vec{\beta} \right|
$$
and the value of $\beta$ which minimizes this value will hopefully be "more sparse" among costly features.
