4

This is our objective function, composed of a loss function and a regularizer. $$\mathcal O(w,x,y) = \mathcal L(w,x,y)+\mathcal R(w)$$ So $\mathcal R(w)=\|w\|_2^2=\sum{w_i^2}$ in the case of $\ell_2$ regularization. Let's perform gradient-based minimization, i.e. we will update params based on the negative partial derivatives of the objective function. ...


4

Feature engineering is about data and is the process of finding/creating features that might improve your model's performance. You sometimes engineer new features from raw data you have, use the existing ones and perform univariate/multivariate transformations. So, by engineering more features your linear regression model may for example become $y=w_1x_1+...


3

It is well known that even in 1D, N-R needn't converge at all and it can hop all over the place. (Numerical Recipes has a good discussion of this behavior, as I recall.) In more than one dimension, consider a function with a global optimum that nevertheless has regions where the Hessian is non-definite. Start the algorithm with a point in one of those ...


3

Additionnally to @forgottenscience answer, let consider the following (more intuitive, to my view) point: First, $x \rightarrow log(x)$ is a strictly increasing function Let call $\theta^*$ the value maximizing $L$ and $\nu^*$ the value maximizing $\log(L)$ (i.e. i.e. $\theta^* =\arg\max_{\theta} L(\theta)$ and $\nu^* =\arg\max_{\nu} \log L(\nu)$). Then, ...


2

I'll assume that the likelihood is differentiable and has a unique maximum. If $\theta^*$ is the argmax of $L(\theta)$, $$\frac{\partial}{\partial \theta} L(\theta^*) = 0, $$ while it follows from the chain rule that $$\frac{\partial}{\partial \theta} \log L(\theta) = \frac{\frac{\partial}{\partial \theta} L(\theta)}{L(\theta)}, $$ so in particular $$ \frac{...


2

L2 regularization adds $w_i^2$ term to the loss function. In iterative approaches using gradients, we subtract the gradient of the loss function not the magnitude of the weight itself. And in the loss function, the regularization part's derivative with respect to $w_i$ is going to be ${d\over dw_i}(w_i^2)=2w_i$. Typically, this part is multiplied with a ...


2

I'd first take a look at the available data, in particular at the distribution of inactivity days before activity is resumed. Some players pause for a day or two and then become active again, some pause for a week etc. For active users, I'd expect a Poisson or an exponential distribution. Whatever the distribution, I'd try to find its parameters. The next ...


2

Because OLS belongs in a subfield of mathematical optimization called convex optimization, and that field has some nice properties, such as every local minimum is a global minimum


2

Feature engineering: working with the available data in order to create/transform good predictors (your $X$). This can usually be done by transforming, averaging, combining, etc. the available columns of your database, in order to obtain the predictors that are the most meaningful (or just work better) for the problem at hand. This can also include feature ...


2

I'm not sure I really understand your analogy, but it seems like your analogy doesn't fully capture either the objections or the philosophy of Bayesianism. If I were going to criticize Bayesianism, I'd point to the fact that there are no guarantees on the results. Frequentists enjoy frequency properties of their estimators. They can say, at least in ...


2

Using vector math re-writing your problem, also replacing b with X: $$f(X)=(X-Y)^T(X-Y)+\varphi |X|^T 1$$ First Order Conditions: $$f(X)'=0$$ $$2(X-Y)+\varphi \space\mathrm{sign}(X)=0$$ Solution: $$X=Y-\mathrm{sign}(X)\varphi/2$$ when $|Y_i|\ge\varphi/2$: $$X=Y-\mathrm{sign}(Y)\varphi /2$$ when $|Y_i|<\varphi/2$: $$X=0$$


2

There is an interesting paper proposing to maximize not the observed likelihood, but the expected likelihood Expected Maximum Log Likelihood Estimation. In many examples this gives the same results as MLE, but in some examples where it is different, it as arguably better, or at least different in an interesting way. Note that this is a pure frequentist ...


1

Let there be $k_i$ balls of color $i$ where $i$ ranges from $1$ to $m=10,$ so that the total number of balls is $k_1+\cdots+k_m=n=100,$ and let $p_i$ be the chance that any randomly chosen ball of color $i$ contains a winning ticket. You are invited to sample up to $s \le m$ balls (without replacement). How to maximize your chances of a winning ticket? To ...


1

As you mentioned we do not know what happens with the censored individuals. Let's say the event in question is death. In this case, censorship means we have not observed the patient die, therefore that information is unavailable. Censored patients may die in the near or far future, but the important thing is that we do not know when they will die (or whether ...


1

\begin{aligned} \min\limits_{\boldsymbol{b}} \boldsymbol{e}^T\boldsymbol{e} = (\boldsymbol{Y}-\boldsymbol{Xb})^T(\boldsymbol{Y}-\boldsymbol{Xb}) \\ \end{aligned} FONC: let $\boldsymbol{u} = \boldsymbol{Y}-\boldsymbol{Xb}$ \begin{aligned} \frac{\partial \boldsymbol{e}^T\boldsymbol{e}}{\partial{\boldsymbol{b}}} &= 0 \\ \frac{\partial [\boldsymbol{Y}^T-\...


1

I just found that inside the algorithm in the "matrix form" the variable it's equal to Wdiag = deriv2(eta) so in this case this variable stays always the same. So we have to change it to Wdiag = deriv2(eta1). So now both of these algorithms works fine. There's another error anyway with this algorithm (this is also present in the original question, linked ...


1

The question is very vague, so I expect they want to see your view on it, how you would solve it. Without more information there are many ways in which you could do this. One example would be using a logistic regression model, which would not only "classify" the players into churned or not churned, but would also give you the odds of a player doing so, ...


1

This is true for any strictly increasing function, and it is quite trivial to prove using the definitions of strict monotonicity and the argmax. In fact, the result does not ]require either of the functions to be differentiable. If $f$ is any strictly increasing real function then, by definition, we have: $$L > L' \quad \quad \iff \quad \quad f(L) >...


1

This is only marginally related to the detailed coding discussion, but seems highly relevant to the original question concerning modeling of the current 2019-nCoV epidemic. Please see arxiv:2002.00418v1 (paper at https://arxiv.org/pdf/2002.00418v1.pdf ) for a delayed diff equation system ~5 component model, with parameter estimation and predictions using ...


1

Lagrange multipliers are fine but you don't actually need that to get a decent intuitive picture of why eigenvectors maximize the variance (the projected lengths). So we want to find the unit length $w$ such that $\|Aw\|$ is maximal, where $A$ is the centered data matrix and $\frac{A^TA}{n} = C$ is our covariance matrix. Since squaring is monotonically ...


Only top voted, non community-wiki answers of a minimum length are eligible