3

This thread may be useful to statisticians and data scientists because it shows how to construct arbitrarily "nasty" functions (that nevertheless are easy to handle mathematically and computationally) for testing algorithms that rely on optimization. One way to construct functions with specified local properties (like local minima) is to assemble them from ...


3

$w^T\Sigma w$ is convex function, you're right. As far as I see, whuber's answer defines concave functions as what we usually know as convex functions. It's also pointed out in the comments. Take two points on the unit sphere, and connect them with a line. Is the line (all of it) inside the domain? No, then the domain is not convex. Because, you maximize ...


3

I see what you're saying, but the answer is, not necessarily. In fact the opposite might occur -- in an industrial application, you might be able to collect more data, or improve your sensors, so hyperparameter tuning is not the only way to improve the performance of your system. In academic research, since people generally compete on a set of standardized ...


2

This is often called algorithmic design or optimal design. The last name hints on some optimality criteria, and there are many to choose from! Much used is D-optimality. Some related posts here is Motivations for experiment design in statistical learning? and Is DoE applicable to collect data for machine learning model?, look at the links and references in ...


1

Genetic algorithms are best when many processors can be used in parallel. and when the object function has a high modality (many local optima). Also, for multi-objective optimization, there are multi-objective genetic algorithms, MOGA. However, I think Genetic algorithms are overrated. A lot of the popularity probably comes from the fact that they are ...


1

Without choosing a loss function, you cannot construct the SVM because you won't have an optimization objective to work on. So, you first choose your loss function and construct your model. In the end, there will be two different models to compare.


1

This is just the result of a bit of tedious algebra. From the other definitions in the article, we have the following identities: $$w^* = A^{-1}b$$ $$w^k = A^{-1}b + Qx^k$$ One identity they assume you know is that $Q^TQ = I$, which implies: $$ Q^T A Q = Q^T Q \text{diag}[\lambda_1,\dots,\lambda_n]Q^T Q = \text{diag}[\lambda_1,\dots,\lambda_n] $$ The next ...


1

The staging does explicitly state anything about the statistical relation of the variables, but rather tells us which conditional distributions we care about. For example, suppose some variable A is a first stage variable and B is a second stage variable. Then, the two probability distributions we care about are $P(A)$ and $P(B | A)$. This is because at the ...


1

You can't do AD without a functional form because you can't take the derivative of something that you don't know. You can approximate derivatives numerically by sampling the function after making small changes to the inputs, which is a technique used in curve fitting. People have managed to apply AD to some non-differentiable functions (like max pooling). ...


1

TensorFlow Probability is a standalone probabilistic programming module for TensorFlow, numpyro package uses JAX, while Pyro is a PyTorch framework. All those frameworks enable you to do Variational Inference and Markov Chain Monte Carlo sampling. The simplest way of computing the expected posterior loss, is using Monte Carlo samples from the posterior ...


1

Hints: You are implicitly using the fact that the likelihood is zero when any $v_i <0$. You should make this explicit Your calculations do make sense and suggest that the likelihood is an increasing function of $A$, i.e. $A$ should be as large as possible The key to this is as possible


1

Not every optimization problem is solved by taking derivatives. And, the PDF is actually $$f_V(v)=e^{-v}\mathbb{I}(v\geq 0)$$ So, we try to maximize $$L=\prod_{i=1}^N e^{A-Y_i}\mathbb{I}(Y_i\geq A)$$ Increasing $A$ monotonically increases the first multiplicand, regardless of $Y_i$. But, there is a limit that we can increase $A$, since the second ...


1

It is relatively simple to write the log-density of the negative binomial distribution in terms of its mean, and then use this to get a log-likelihood expression for the negative binomial GLM. For all values $y=0,1,2,...$ the log-density of the negative binomial distribution is: $$\log f(y|r,\theta) = \log {y+r-1 \choose y} + r \log (1-\theta) + y \log (\...


1

Indeed hyperparameter tuning is very important in research papers most times. The reason is that the research papers should be novel, which means that "default value" is hardly available since your method is a new one and has different parameters from the old methods, so you must tune your parameters carefully. Then the tuned parameters should be published ...


Only top voted, non community-wiki answers of a minimum length are eligible