What do think about switching from L2-loss to L1-loss when inner difference became less then 1? Square of number < 1.0 makes that number smaller. So the gradient became smaller and learning very slow. So for me, it looks naturally to use L2 loss when the difference between target and output is more than one and use L1 when it is < 1.0. The same is true for example for Mean Relative Squared Error there we don't need to take square when a relative error is less than one.
I have tested my thought on a real time series prediction dataset with different gradient descent optimizers(Adam, vanilla, Adadelta) and that gave me much faster and better convergence. What do you think. Is this a dataset-specific or should be general practice? Error surface became not smooth and not convex when using two losses, but the same is true for ReLu and that is not a problem. What are the potential dangers of this approach?
 A: Changing the loss, changes the problem, so you can't objectively compare using one loss with other.
As for your idea of using a hybrid between L1 and L2 loss, Huber loss, does this, but it's the other way around, preferring L2 loss when the difference is small and L1 loss otherwise.
A: There are a couple of problems with your plan.  While your goal of improving speed is laudable is it appropriate for the problem at hand.  
The first problem is that if you are minimizing $f(\theta,\hat{\theta})$ and you are trying to substitute solving it as $g(\theta,\hat{\theta})$ then the question would become "what about this substitution would make the result acceptable to whoever needs the result?"  
The second problem, and this gets brushed aside way too much, is that the loss function is the loss to be experienced if you get a bad sample.  Why are you using the loss function that you are using?  If your answer happens to be "well everyone uses it" or, "it's the standard default" then you may be solving the wrong problem anyway.  What would you lose if you got a bad sample.
Let me provide you an example.  I believe this either comes from Lindley or Jeffreys originally.
Let us imagine you had to lay a cable as a single piece.  For whatever reason, you cannot splice two together.  Until recently, that was the case for fiber optic cable and is often the case with concrete.  The true value is $\theta=1000.$  You will use an estimator to determine how much cable to purchase and deliver.
The estimation tool has a known standard deviation of one unit for this type of problem.  You decide to use quadratic loss because you know the data is normally distributed and the estimator will be unbiased.
The cost of cable is $1000 per unit of length.  There is no recovery income if too little is purchased.  It cannot be resold.  If you buy too little, then you must simply discard it and buy more and try again.
If your unbiased estimator produces a result of 999, then your loss is \$999,000.  You must then purchase another \$1,000,000 worth of cable.  If your unbiased estimator produces a result of 1001, then you cut off the last unit of length and use the remaining 1000 in the actual problem.  Your loss is $1000.
An unbiased estimator will produce a catastrophic loss fifty percent of the time.  Your expected loss would be around 498 thousand dollars (rounded).  Biased estimators fall off to a meaninglessly close to zero expected loss to the right depending on your sampling assumptions.
If time is of the essence, for example, if in robotic surgery time delays could cause death, then you could build a complexity penalty or a sample size penalty in to account for the role time plays.  Nonetheless, your solution is unlikely to be optimal in any sense if that were the case.
You should be asking the question, "what do I lose if I am wrong and how do I know that?"  That is your loss function.
Loss functions only appear to be cardinal measures.  A loss function is a negative utility function.  As a utility function, comparisons are on ordinal rankings and not cardinal.  You cannot compare a quadratic loss to an absolute linear loss or any other except by stating "loss function $f$ is better for me than loss function $g$ because..."
If $f\succ{g}$ then $f$ is better than $g$ even if the total loss, in a particular instance, for $g$ is 50 units and the total loss for $f$ is 100 units$^2$.
