4
$\begingroup$

In the paper Novel PCA-based Color-to-gray Image Conversion, the authors project the three-dimensional $(R, G, B)$ value of each pixel onto a one-dimensional grayscale space via a curious application of PCA (emphasis mine):

(......) To compute a gray image ($I_{gray} \in \mathbb{R}^n$), the proposed ELSSP is conducted, and then the output is scaled to $[0, 255]$. Note that we utilize the eigenvalues as weighting factors for projection results on corresponding eigenvectors. As a result, the color-to-gray mapping is dominated by the first subspace projection, and the second and third subspace projections contribute to preserving details of a color image in a gray image.

The motivation behind this proposal can be found in Section 2.1. Subspace Projections. Intuitively that makes sense to me, but from a theoretical point of view, this operation seems awkward.

As far as I know, the first subspace projection preserved the most information of the original data, so such a linear combination must contain less information than the first subspace projection along, yet according to the paper's abstract, "experimental results demonstrate that the proposed method is superior to the state-of-the-art methods in terms of both conversion speed and image quality."

How is this possible?

$\endgroup$
2
+50
$\begingroup$

As you indicated in your question, if you projected the PCA onto the data as it is, all you would get is an identical image so some step is required to make a meaningful difference since that is the point of image processing.

Let's walk through what the processing is doing:

  1. PCA identifies the principal sources of variation between 3 mean centred visual properties (brightness, and color channel differences) in the image of N pixels (so a 3xN matrix), which gives 3 principle components.
  2. If you were to reconstruct the image by a straight projection of the PCA onto the data you would simply get the same result as you would without the PCA since the PCA would perfectly reconstruct the image (being underdetermined). By weighting with the eigenvalue you change the output, emphasising the major components, giving a difference due to the processing.
  3. However, the process does not actually reconstruct the image, rather it sums across the weighted eigenvectors. This means it is a weighted sum of the principle sources of variation between intensity and the two colour contrasts. However to understand the implications of this it is necessary to understand the impact on plain projection of the reweighting.

So what is the difference between standard PCA projection and weighting by eigenvalues?

I propose to illustrate using the singular value decomposition (one of many algorithms used to calculate PCA).

In SVD the data is decomposed into unit vector eigenvectors on both sides of the inner product, necessitating a diagonal matrix of weights to scale the transformation to the data, the so called singular value. We will call the left unit vector $U$, the singular values $s$ and we'll call the right eigenvector $\nu$ as used in the paper.

$$Image = Us\nu^T$$

This is converted to PCA by multiplying $U$ and $S$ to give scores weighted by the singular value. This singular value is the square root of the eigenvalue ( see Wiki Page or Math SE Question), i.e. $\lambda=s^2$.

So this means the projection is not symmetric, but instead is

$$Image = Us^2\nu^T$$

So the weighting is raised to the power of 2 compared to a standard projection. This means that although all the information is used in the reconstruction (your original concern), the information is very heavily weighted towards the largest eigenvectors.

Whether this is a useful result depends on your tastes (for aesthetics) and your needs (e.g. as input into machine learning)for the grey scale conversion.

*****EDIT******

Physiological Light Detection

It is also informative to consider the physiological workings of the retina when evaluating the concept presented in this paper. The retina has two main types of photodetecting cells, rods and cones. The rods detect a broad range of wavelengths of light and measure changes in intensity/brightness, while the cones detect one of (usually) 3 different colours of light - red, green or blue. See comparison of rods and cells here and contrast detection in retina

Key points relevant to the process described in the paper:

All types of photoreceptor detect light intensity, giving a nerve signal proportional to the intensity of the light.

Individual photoreceptors are hardwired together to perform image processing even before the signal reaches the optic nerve.

Rods are hardwired together to amplify light much more than cones (their role is in night vision) The retina is hardwired to detect contrast, including colour contrast between different types of cones.

These biological mechanisms explain the reason why many of the processing steps taken in the paper create a 'natural' transformation. The initial conversion from RGB to Intensity and colour contrasts reflects some of the hardwired transformations of the retina (but not all).

When converting from colour (cone detected) to greyscale you will get an effect closer to night vision if you amplify the effects in a non-linear way. This is what raising the singular value to the power of 3 does.

I did not get the impression that any of the processing steps were specifically optimised against the physiological nature of the retina and so I suspect issues such as block scaling intensity vs each of the contrasts, plus tweaking the exponent of the singular value could make the transformation even more asthetically pleasing.

*****EDIT2****** from the paper the algorithm procedure is:

1: procedure COLOR-TO-GRAY(I$_{rgb}$)

2: I$_{ycc}$ ← f(I$_{rgb}$) . according to [8]

3: I$_{ycc}$ ← I$_{ycc}$ − I$_{ycc,avg}$ . I$_{ycc,avg}$ = mean(I$_{ycc}$)

4: λ$_i$ , v$_i$ ← PCA(I$_{ycc}$) . i = 1, 2, 3

5: λ$_i$ ← λ$_i$/ ||$\lambda$||, v$_i$ ← v$_i$/ ||v$_i$|| . λ = {λ$_1$, λ$_2$, λ$_3$}

6: I$_{gray}$$\Sigma^3_{i=1} \lambda_i (v^T_i I_{ycc})$, then scaling to [0, 255]

7: if ||I$_y$ − I$_{gray}$|| > ||I$_y$ − (255 − I$_{gray}$)|| then

8: I$_{gray}$ ← 255 − I$_{gray}$

9: end if

10: end procedure

If you pay close attantion to this algorithm you will see that you should be inputting a 2D matrix, with the pixels flattened so the spatial information is a flat 1D vector but the other dimension is the channels in the new colour space.

Next you notice that step 4 is performing the PCA calculation, step 5 is weighting the eigenvalues and step 6 is projecting using the re-weighted eigenvalues (the part after the sigma term $\lambda_i (v^T_i I_{ycc})$) and summing across these channels in one step ($\Sigma^3_{i=1}$).

It is this summation that converts from YCC to greyscale - if you do not sum, then you will have an output vector of 3 channels that can be backtransformed into RGB. Note that this summation is not a PCA step - the authors have conflated PCA projection and summation into one operation which may be the source of your confusion expressed in the comments about recovery of the image after PCA.

*** FURTHER EDIT****

I have addressed a coding question on this paper here and while exploring the code have come to understand it better. There appears to be a fundamental instability in the algorithm as switching the signs of the eigenvectors changes the results. The choice of which side is up in an eigenvector is completely arbitrary and can be different from one algorithm to the next (or even iteration within an algorithm) - this makes the output unstable and unreliable.

$\endgroup$
  • $\begingroup$ Great answer, but could you please elaborate on the PCA would perfectly reconstruct the image (being underdetermined)? I cannot understand how to reconstruct a colored image from a grayscale one when PCA is employed. $\endgroup$ – nalzok May 4 at 16:49
  • $\begingroup$ Also, at the risk of getting off-topic, could you briefly describe the significance of the RGB to Y'CrBr transformation (source)? This transformation is linear, and we know luminance/intensity/brightness contains a lot of information, so I was expecting PCA to "re-invent" it. $\endgroup$ – nalzok May 4 at 17:05
  • $\begingroup$ The PCA would reconstruct the image input into it, so you would need to back transform all your processing steps, reshaping back to a 3 d matrix and converting your channels to rgb. In the paper the values of the scores were summed after the weights projection to give the greyscale image, so am extra step was required after PCA. The significance of the YCrBr is explained in the previous edit, in that it is reorganising visual information in a way analogous to how the biological eye reorganises visual input before it even reaches the optic nerve. $\endgroup$ – ReneBt May 4 at 19:01
  • $\begingroup$ Sorry, but I didn't get the answer to the first question: essentially I'm performing dimensionality reduction, from 3D RGB/YCrBr color space to 1D grayscale color space. How can a grayscale image be identical to a colored one? I mean, the chrominance information is going to be discarded, so I don't think it would be possible to "perfectly" reconstruct the original image. $\endgroup$ – nalzok May 5 at 0:32
  • 1
    $\begingroup$ I see a few issues with the accepted answer, it is not at all right. I will see if I can get time to put an answer in $\endgroup$ – ReneBt May 6 at 13:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.