What does it mean to say that a regression method is (not) "scale invariant"? I was just studying partial least squares regression, and I read that it is "not scale invariant". What does "scale invariant" mean, and why is partial least squares, and why would a regression methodology in general be, not "scale invariant"? Some examples to better illustrate exactly what this means would be greatly appreciated.
 A: Scale invariance means that rescaling any or all of the columns will not change the results - that is, multiplying or dividing all the values from any variable will not affect the model predictions (ref). As @ericperkeson mentioned, rescaling in this manner is known as dilation (ref). Scale invariance for metrics about contingency tables refers to rescaling rows as well as columns, though I don't believe it applies here (see the scaling property section here).
As to why PLSR is not scale invariant, I'm not completely certain, but I'll leave notes on what I've learned and possibly a better mathematician can clarify. Generally, regression with no regularisation (e.g. OLS) is scale invariant, and regularised regression (e.g. ridge regression) is not scale invariant, because the minimisers of the function change (ref).
Now, I can't see an explicit penalty term in PLSR, but I it's constrained in a similar way to PCA. PCA chooses the axes of maximal variance - so if you rescale a variable, the variance relative to other variables can change (ref). PLSR tries to find the ' multidimensional direction in the X space that explains the maximum multidimensional variance direction in the Y space', hence rescaling an input can change the direction of maximum variance (ref).
A: Start with the technical meanings of "location" and "scale" with respect to a one-dimensional probability distribution. The NIST handbook says:

A probability distribution is characterized by location and scale parameters ... a location parameter simply shifts the graph left or right on the horizontal axis ... The effect of the scale parameter [with a value greater than 1] is to stretch out the graph ...  The standard form of any distribution is the form that has location parameter zero and scale parameter one.

Think of a data sample as a collection of empirical probability distributions for each of the predictors and outcomes. For the example in a comment, temperatures expressed either as degrees F or degrees C, there is a transformation with respect to both location and scale. Transformation from degrees C to degrees F changes the numerical values of degrees by a factor of $\frac {9}{5}$ (along with a subsequent location change of 32 degrees F). The variance of temperature values thus also changes by a factor of $\frac{81}{25}$. By "stretching out the graph," a transformation of the scale of a predictor changes the numerical values for the predictor and for its variance. Nevertheless, the underlying physical reality is the same.
With standard multiple regression, a change in the units of a predictor can be counterbalanced by a corresponding change in the units of the regression coefficients. If temperature in degrees C is a predictor in a model and you switch from degrees C to degrees F then (along with altering the intercept appropriately) you multiply the regression coefficient for temperature by a factor of $\frac{5}{9}$ and the model is the same. In that sense, the modeling process is "scale invariant." Similarly, correlation coefficients are scale invariant as the calculation corrects for the scales of the variables.
Regression modeling processes that differentially penalize predictors, in contrast, fundamentally depend on comparisons among the numerical values of the various predictors. That includes approaches like LASSO, ridge regression, principal components regression (PCR), and partial least squares (PLS). Say that both temperature and distance are predictors in a penalized model. In building the model you need to have a way to decide whether temperature or distance is relatively more important to weight in the model, yet all you have to work with is their numerical values. Those numerical comparisons between the temperature and distance predictor values will differ depending on whether temperature is expressed in degrees F or C, and on whether distances are expressed in miles or in millimeters. Such a modeling process is not scale invariant.
With respect to PCR and PLS, you can see this in the problems that they solve at each step, as expressed on page 81 of ESL, second edition:

... partial least squares seeks directions that have high variance [of predictors] and have high correlation with the response, in contrast to principal components regression which keys only on high variance...  In particular, the $m$th principal component direction $v_m$ solves:
$$ \operatorname{max}_\alpha \operatorname{Var}(\mathbf{X} \alpha) $$
$$ \text{subject to } \lVert \alpha \rVert =1,\: \alpha^T \mathbf{S} v_{\ell} =0, \: \ell =1,\dots,m−1,$$
where $\mathbf{S}$ is the sample covariance matrix of the [vectors of predictor values, indexed by $j$ for predictors] $\mathbf{x}_j$. The conditions $ \alpha^T \mathbf{S} v_{\ell} =0$ ensures that $\mathbf{z}_m = \mathbf{X} \alpha$ is uncorrelated with all the previous linear combinations $\mathbf{z}_{\ell} = \mathbf{X} v{_\ell}$. The $m$th PLS direction $\hat{\varphi}_m$ solves:
$$\operatorname{max}_{\alpha} \operatorname{Corr}^2(\mathbf{y},\mathbf{X}\alpha)\operatorname{Var}(\mathbf{X} \alpha) $$
$$\text{subject to } \lVert \alpha \rVert =1,\: \alpha^T \mathbf{S} \hat{\varphi}_{\ell} =0,\: \ell=1,\dots,m−1.$$

Here, the unit-norm vector $\alpha$ is the relative weighting of the predictors that will be added to the model at that step. $\operatorname{Var}(\mathbf{X} \alpha)$ is the variance among the observations of that weighted sum of predictor values. If the scales of the predictor values are transformed, that variance and thus the model itself is fundamentally transformed in a way that can't be undone by a simple change of units of the regression coefficients. So these are not scale-invariant modeling procedures.
The usual procedure to maintain equivalence among continuous-valued predictors for such modeling approaches is to transform them to zero mean and unit standard deviation before anything that requires comparisons among predictors. Categorical predictors require some thought in terms of how to put them into "equivalent" scales with respect to each other or to continuous predictors, particularly if there are more than 2 categories. See this page and its links for some discussion.
A: I think the comment by user "erikperkerson" was short and highly informative:

I was under the impression that scale invariant usually means invariant with respect to a dilation (a proper linear mapping, like $f(x) = kx$ for some constant $k$), such as the unit conversion from miles to millimeters that EdM suggested. The example of converting C to F is not a dilation, because it is an affine linear mapping like $f(x) = kx + b$ instead of a proper linear mapping. Invariance under affine linear mappings would imply both scale and shift invariance.

