We know "if a function is a non-convex loss function without plotting the graph" by using Calculus.
To quote Wikipedia's convex function article: "If the function is twice differentiable, and the second derivative is always greater than or equal to zero for its entire domain, then the function is convex." If the second derivative is always greater than zero then it is strictly convex.
Therefore if we can prove that the second derivatives of our selected cost function are always positive the function is convex.
We care about convexity because the minimum of a convex function is also a global minimum. If the function is strictly convex function then it will have at most one global minimum which is also convenient; we prove the global optimality of particular solution. Please see the thread here as why we generally want an objective function to be a convex function and here on whether gradient descent can be applied to non-convex functions, for some more information on the matter.