Data transformations to increase variance

Question

Here's a brief snippet of my data (the first two columns are measured spectral data attributes, and the latter columns are indexes calculated from the first two):

The issue I'm running into is that the variance of the two measured attributes are drastically different (first column StdDev is <.5% of mean, second column StdDev is about 9% of mean). Consequently, when I calculate any index attributes based on these values, the result is essentially completely determined by the second column--the first column is almost a constant.

There are common techniques for reducing high variance in data (log or sqrt transformations...) but I'm not sure what's a legitimate approach to increasing variance in an attribute? This must be a common issue though, right? I thought about standardizing the attributes, but that results in nonsense when I calculate indexes, because if the denominator is an "average" data point, the Z-score is near zero, resulting in huge values.

Other things I've considered, but am not sure about the mathematical legitimacy of:

1. Using a negative log transformation, or a negative power transformation. This seems mathematically reasonable, but I haven't seen it done.

2. Simply subtracting a constant from the column with low variance. For example, the attribute I'm concerned about basically has only values between 246 and 248.I could just subtract 240 from all the values, and that would dramatically increase the variance. But it also would be problematic from a mathematical perspective, right? This data indicates the strength of an electrical signal from a sensor instrument. It is quantitatively connected to a real phenomena in the real world (spectral properties of tree needles) so the values are meaningful, in the sense that a 1% increase in the value = a 1% increase in signal from the sensor. So, if I remove a constant to increase the variance, I think I'd be loosing important quantitative information?

That's all I've come up with so far. Anybody have mathematical support (or criticism) for either of these suggestions? Or another approach for dealing with this issue?

Answer 1

First, since these are physical values with real meaning then you may want the behavior you are trying to eliminate. If all the items are essentially the same on NIR then why should it play much of a role in creating an index?

I think this is why our approach of standardizing the scores yields nonsense.

Second, if you decide that you do want them both to contribute equally, you'd have to say exactly what you mean by "contribute equally". Do you mean they should have equal variance? If you square the values in the first column, the sd will increase, but not by much as a proportion of the mean:

set.seed(12345)

x <- rnorm(1000, 200, 1)
sd(x)/mean(x)  #0.004
x2 <- x*x
sd(x2)/mean(x2)  #0.009

so you'd have to make some absurd transformation to get the proportion the same as the other variable.

Answer 2

As @Flom mentioned, if a feature (dimension) is not contributing to the results, perhaps this is what you need to pay attention to instead of forcing it to have a noticeable affect on the results. Not all features play a significant role in every single analysis. This could be a clue from reducing the dimension of your dataset, or looking for other features.

But what make sense to me is applying PCA, to find a new space where you can look at your data from a better angle or in other words, to find better attributes. If the first column (NIR) has a very low variance, it may have a much larger variance (hence, information) if looked at from a different angle. I am trying to elaborate on the idea of Principal Components (PC) here.

For example, in the scatter plot below, for a 2 dimensional dataset, you see that the original X and Y axes would let us look at our data from 2 orthogonal perspectives with a certain variance for each of them.

But if we find the eigen-vectors of our data matrix, we can find new axes (a new space with up to 2 dimensions). Now $X=PC1$ and $Y = PC2$. In the new space, variance of the components are in total maximized.

This is a classic dimensionality reduction task. But what is important to note before you decide whether you want to use it, is that your space will be transformed. Meaning that, each of the new dimensions are a combination of the old dimensions and perhaps, not perfectly interpretable. Unless you find a way to transform the results back to the original space.

---

1. Plot from a useful post on StatistiXL.