cost function which does not have a min/max point In machine learning, is it meaningful to have a cost function which does not have a min/max point, i.e., the cost function's min/max point is negative/positive infinity? Any examples?
 A: One example is the so-called death penalty in optimization where infeasible solutions, or constraints, are handled with infinite penalty so to be easily discarded by the optimization algorithm. For example, you want to estimate positive-valued parameter using a black-box optimizer that searches for optimal value over real line. One way to tell your optimizer that it should not consider negative values is to give them infinite penalty, so that any non-negative outcome will be better then negative one. This may not be the best and most efficient way of dealing with such cases, but it is one of the possible approaches, especially if you do not have any other way of defineing constraints. Check Modern Optimization with R by Paulo Cortez, or here.
A: Imagine you have an optimization problem
\begin{equation}
 \begin{array}{*2{>{\displaystyle}r}}
 \mbox{minimize (over $x$)} & c(x) \\
 \mbox{subject to} & x \in X \end{array}
\end{equation}


*

*A cost function that goes to infinity is perfectly reasonable. One can model infeasible choices as infinite cost choices. 


*

*Example: Sure I can buy a spaceship that travels to Alpha Centurai! But the cost is $+\infty$

*Define $\hat{c}(x) = \left\{ \begin{array}{cl} c(x) &\text{ if } x \in X \\+\infty & \text{if } x \notin X \end{array}\right\}$ . Original problem is equivalent to minimize $\hat{c}(x)$


*A cost function $c$ that is unbounded below on the feasible set $X$ is problematic and bizarre. If an infimum does not exist on the feasible set, you probably have done something strange in formulating your problem.

Update: If your cost function is an expectation of some kind of sub-cost function, eg. $c(x) = \int f(x) dP$ then $c$ can still be bounded below even if $f$ is not bounded below over regions of no-support (or perhaps other $-\infty$ times 0 like technical conditions). Questions on existence of a minimum or existence of an infimum are approached far more rigorously in real analysis textbooks, optimization books etc.... 
