You're seeing ringing, which is the result of passing a high-frequency change, ie a step-function, through a low-pass filter, ie the GAM.

When you apply the log transformation, you change the gradient of the near-vertical section of the graph, on the left hand side, so that it is slightly less steep, with fewer implicit high frequencies, and the ringing effect goes away.

Edit: some pictures of ringing here: https://electronics.stackexchange.com/questions/79717/what-can-reduce-overshoot-and-ringing-on-a-simple-square-wave-pulse-generator

Edit2: note that increasing smoothing would increase ringing, since the smoothing is essentially the low-pass filter that is causing the ringing. What would reduce ringing would be for example 1. remove the points in the rising cliff-edge on the left, and refit, or 2. reduce the smoothing, or 3. reduce the frequency/increase wavelength/increase the cutoff frequency of the smoothing.

You can see that if you remove the cliff-edge bit, the rest of the graph is more or less a straight-line, so why is the GAM fitting a sinusoidal wave through those points? Its entirely because the cliff-edge part is forcing a very high gradient, which then causes subsequent overshoot.

You're seeing ringing, which is the result of passing a high-frequency change, ie a step-function, through a low-pass filter, ie the GAM.

When you apply the log transformation, you change the gradient of the near-vertical section of the graph, on the left hand side, so that it is slightly less steep, with fewer implicit high frequencies, and the ringing effect goes away.

Edit: some pictures of ringing here: https://electronics.stackexchange.com/questions/79717/what-can-reduce-overshoot-and-ringing-on-a-simple-square-wave-pulse-generator

Edit2: note that increasing smoothing would increase ringing, since the smoothing is essentially the low-pass filter that is causing the ringing. What would reduce ringing would be for example 1. remove the points in the rising cliff-edge on the left, and refit, or 2. reduce the smoothing, or 3. reduce the frequency/increase wavelength/increase the cutoff frequency of the smoothing.

You can see that if you remove the cliff-edge bit, the rest of the graph is more or less a straight-line, so why is the GAM fitting a sinusoidal wave through those points? Its entirely because the cliff-edge part is forcing a very high gradient, which then causes subsequent overshoot.

Edit3: if it was me, I think I'd try to find a transform that will transform the graph into an approximately straight line. I'm not quite sure what that transform would be, but looks like the graph is very close to being a flat line, asymptotic to ~380 or so. This is a stronger non-linearity than eg log, which will become flat-ish, but not quite so quickly I think. Maybe something like an inverse sigmoid? Sigmoid is:

$$ y = \frac{1}{1 + \exp(-x)} $$

... and looks like (from wikipedia https://en.wikipedia.org/wiki/Sigmoid_function )

Inverse sigmoid is the logit function, https://en.wikipedia.org/wiki/Logit:

$$ f(x) = \log \left( \frac{1}{1-x} \right) $$

Maybe a transformation related to this, or a parameterized version of this, might make the graph closer to being a straight line, and thus more amenable to standard statistical techniques?

You're seeing ringing, which is the result of passing a high-frequency change, ie a step-function, through a low-pass filter, ie the GAM.

When you apply the log transformation, you change the gradient of the near-vertical section of the graph, on the left hand side, so that it is slightly less steep, with fewer implicit high frequencies, and the ringing effect goes away.

Edit: some pictures of ringing here: https://electronics.stackexchange.com/questions/79717/what-can-reduce-overshoot-and-ringing-on-a-simple-square-wave-pulse-generator

You're seeing ringing, which is the result of passing a high-frequency change, ie a step-function, through a low-pass filter, ie the GAM.

When you apply the log transformation, you change the gradient of the near-vertical section of the graph, on the left hand side, so that it is slightly less steep, with fewer implicit high frequencies, and the ringing effect goes away.

Edit: some pictures of ringing here: https://electronics.stackexchange.com/questions/79717/what-can-reduce-overshoot-and-ringing-on-a-simple-square-wave-pulse-generator

Edit2: note that increasing smoothing would increase ringing, since the smoothing is essentially the low-pass filter that is causing the ringing. What would reduce ringing would be for example 1. remove the points in the rising cliff-edge on the left, and refit, or 2. reduce the smoothing, or 3. reduce the frequency/increase wavelength/increase the cutoff frequency of the smoothing.

You can see that if you remove the cliff-edge bit, the rest of the graph is more or less a straight-line, so why is the GAM fitting a sinusoidal wave through those points? Its entirely because the cliff-edge part is forcing a very high gradient, which then causes subsequent overshoot.