There are different types of cost functions like Cross-entropy, absolute error, mean squared error.
When can we expect the cost function to be non-convex? Does this depend upon the type of cost function we choose? Or does this depend upon the Model we choose like when we have higher order polynomial Model?
In general, the convexity of the cost function will depend on both the form of the cost function itself and the chosen model.
The model tells you how to make predictions. In other words, it is a function $f$ of the form $$ f: \mathcal{X} \to \mathcal{Y}, $$ where $\mathcal{X}$ is the feature space and $\mathcal{Y}$ is the output space. Given a sample $x \in \mathcal{X}$, the predicted output is $f(x)$. Usually the function $f$ is fully specified by a choice of parameters $\theta \in \Theta$. One such example is the linear model; once you specify the values of $\theta$, predictions are determined according to $f(x) = \theta^\intercal x$.
Meanwhile the cost is a function $C$ of the form $$ C:\mathcal{Y} \times \mathcal{Y} \to \mathbb{R}. $$ Given a predicted value $f(x)$ and the true label $y$ (which are both members of the set $\mathcal{Y}$), the cost $C(f(x), y)$ tells you exactly how good or bad your prediction was.
Since we are usually interested in the convexity of the cost function with respect to the parameters $\theta$, we need to know both how the parameters are used to make predictions and how these figure into the cost function itself. This is only possible once both $f$ and $C$ are specified.
