What is the most accurate way of determining an object's color?

Question

I have written a computer program that can detect coins in a static image (.jpeg, .png, etc.) using some standard techniques for computer vision (Gaussian Blur, thresholding, Hough-Transform etc.). Using the ratios of the coins picked up from a given image, I can establish with good certainty which coins are which. However, I wish to add to my confidence levels and also determine if a coin that I deduce to be of type-A (from radius ratios) is also of the correct colo[u]r. The problem is that for British coins et al. (copper, silver, gold), the respective colors (esp. of copper to gold) are very similar.

I have a routine that extracts the mean color of a given coin in terms of the RedGreenBlue (RGB) 'color-space' and routines to convert this color into HueSaturationBrightness (HSB or HSV) 'color-space'.

RGB is not very nice to work with in attempting to differentiating between the three coin colors (see attached [basic] image for an example). I have the following ranges and typical values for the colours of the different coin types:

Note: the typical value here is one selected using a 'pixel-wise' mean of a real image.

**Copper RGB/HSB:** typicalRGB = (153, 117, 89)/(26, 0.42, 0.60).

**Silver RGB/HSB:** typicalRGB = (174, 176, 180)/(220, 0.03, 0.71).

**Gold RGB/HSB:** typicalRGB = (220, 205, 160)/(45, 0.27, 0.86) 

I first tried to use the 'Euclidian distance' between a given mean coin color (using RGB) and the typical values for each coin type given above treating the RGB values as a vector; for copper we would have:

$$D_{copper} = \sqrt((R_{type} - R_{copper})^{2} + (G_{type} - G_{copper})^{2} + (B_{type} - B_{copper})^{2})$$

where the smallest value of the difference ($D$) would tell us which type the given coin is most likely to be. This method has shown itself to be very inaccurate.

I have also tried just comparing the hue of the coins with the typical values of the types provided above. Although theoretically this provides a much better 'color-space' to deal with varying brightness and saturation levels of the images, it too was not accurate enough.

Question: What is the best method to determine a coins type based on color (from a static image)?

Thanks very much for your time.

Edit 1

Note: I have tried all of the ideas discussed below and have achieved next to nothing. Variance in lighting conditions (even within the same image) make this problem very tough and should be taken into consideration.

Edit 2 (Summery of Outcome)

Thank you for your answers. Further research of my own (including your answers and comments) has highlighted just how tough this problem is to deal with in the generic case of arbitrary lighting, arbitrary camera (mobile device), fluctuation in coin colour (even for same species/type) etc. I first looked at skin colour recognition (a very active field of research) as a starting point and there are still numerous problems even with the recognition of skin colour for Caucasians alone (see this paper for a review of the current techniques), and the fact that this problem contains three distinct colour objects all of which can have continuous and varying chromacities make this topic of computer vision a very hard one to classify and deal with accordingly (in fact you could do a good Ph.D. on it!).

I looked into the Gamut Constraint Method from the very helpful post by D.W. below. This was at first sight very promising as a pre-processing step to transform the image and the separate coin objects to colours that are independent of lighting conditions. However, even this technique does not work perfectly (and involves a library of images/histograms for mappings – which I don’t want to get into) and neither does the much more complex Neural Network Architecture methodologies. In fact this paper states in the abstract that:

"current machine colour constancy algorithms are not good enough for colour-based 
 object recognition.".

That is not to say that there aren’t much more up-to-date papers on this subject out there, but I can't find them and it does not seem to be a very active research area at this time.

The answer by AVB was also helpful and I have looked into LAB* briefly.

"The nonlinear relations for L*, a*, and b* are intended to mimic the nonlinear
response of the eye. Furthermore, uniform changes of components in the L*a*b* colour
space aim to correspond to uniform changes in perceived colour, so the relative 
perceptual differences between any two colours in L*a*b* can be approximated by 
treating each colour as a point in a three dimensional space."

From what I have read, the transformation to this colour space for my device dependent images will be tricky - but I will look into this in detail (with a view to some sort of implementation) when I have a bit more time.

I am not holding my breath for a concrete solution to this problem and after the attempt with LAB* I shall be neglecting coin colour and looking to sure-up my current geometric detection algorithms (accurate Elliptic Hough Transform etc.).

Thanks you all. And as a end note to this question, here is the same image with a new geometric detection algorithm, which has no colour recognition:

Answer 1

Two things, for starters.

One, definitively do not work in RGB. Your default should be Lab (aka CIE L*a*b*) colorspace. Discard L. From your image it looks like the a coordinate gives you the most information, but you probably should do a principal component analysis on a and b and work along the first (most important) component, just to keep things simple. If this does not work, you can try switching to a 2D model.

Just to get a feeling for it, in a the three yellowish coins have STDs below 6, and means of 137 ("gold"), 154, and 162 -- should be distinguishable.

Second, the lighting issue. Here you'll have to carefully define your problem. If you want to distinguish close colors under any lighting and in any context -- you can't, not like this, anyway. If you are only worried about local variations in brightness, Lab will mostly take care of this. If you want to be able to work both under daylight and incandescent light, can you ensure uniform white background, like in your example image? Generally, what are your lighting conditions?

Also, your image was taken with a fairly cheap camera, by the looks of it. It probably has some sort of automatic white balance feature, which messes up the colors pretty bad -- turn it off if you can. It also looks like the image either was coded in YCbCr at some point (happens a lot if it's a video camera) or in a similar variant of JPG; the color information is severely undersampled. In your case it might actually be good -- it means the camera has done some denoising for you in the color channels. On the other hand, it probably means that at some point the color information was also quantized stronger than brightness -- that's not so good. The main thing here is -- camera matters, and what you do should depend on the camera you are going to use.

If anything here does not make sense -- leave a comment.

Answer 2

In the spirit of brainstorming, I'll share some ideas you could try:

1. Try Hue more? It looks like Hue gave you a pretty good discriminator between silver and copper/gold, though not between copper and gold, at least in the single example you showed here. Have you examined using the Hue in greater detail, to see whether it might be a viable feature to distinguish silver from copper/gold?

   I might start by gathering a bunch of example images, which you have manually labelled, and computing the Hue of each coin in each image. Then you might try histogramming them, to see if Hue looks like a plausible way to discriminate. I might also try looking at the average Hue of each coin, for a handful of examples like the one you presented here. You might also try Saturation as well, as that looked like it might be helpful as well.

   If this fails, you might want to edit your question to show what you've tried and give some examples to concisely illustrate why this is hard or where it fails.

2. Other color spaces? Similarly, you might try transforming to rg chromacity and then experimenting to see whether the result is helpful at distinguishing silver from copper/gold. It is possible that this might help adjust for illumination variation, so it could be worth trying.

3. Check relative differences between coins, rather than looking at each coin in isolation? I gather that, from the ratios of coin sizes (radiuses), you have an initial hypothesis for the type of each coin. If you have $n$ coins, this is a $n$-vector. I suggest you test this entire composite hypothesis in a single go, rather than $n$ times testing your hypothesis for each coin on its own.

   Why might this help? Well, it may let you take advantage of the relative hues of the coins to each other, which should be closer to invariant with respect to illumination (assuming relatively uniform illumination) than each coin's individual hue. For example, for each pair of coins, you can compute the difference of their hues and check whether this corresponds to what you'd expect give your hypothesis about their two identities. Or, you could generate a $n$-vector $p$ with the predicted hues for the $n$ coins; compute a $n$-vector $o$ with the observed hues for the $n$ coins; cluster each one; and check that there is a one-to-one correspondence between hues. Or, given the vectors $p,o$, you could test whether there exists a simple transformation $T$ such that $o \approx T(p)$, i.e., $o_i \approx T(p_i)$ holds for each i. You may have to experiment with different possibilities for the class of $T$'s that you allow. One example class is the set of functions $T(x)=x+c \pmod{360}$, where the constant $c$ ranges over all possibilities.

4. Compare to reference images? Rather than using the color of the coin, you might consider trying to match what is printed on the coin. For instance, let's say that you have detected a coin $C$ in the image, and you hypothesize it is a one pound coin. You could take a reference image $R$ of a one pound coin and test whether $R$ seems to match $C$.

   You will need to account for differences in pose. Let me start by assuming that you have a head-on image of the coin, as in your example picture. Then the main thing you need to account for is rotation: you don't know a priori how much $C$ is rotated. A simple approach might be to sweep over a range of possible rotation angles $\theta$, rotate $R$ by $\theta$, and check whether $R_\theta$ seems to match $C$. To test for a match, you could use a simple pixel-based diff metric: i.e., for each coordinate $(x,y)$, compute $D(x,y) = R_\theta(x,y) - C(x,y)$ (the difference between the pixel value in $R_\theta$ and the pixel value in $C$); then use a $L_2$ norm (sum of squares) or somesuch to combine all of the difference values into a single metric of how close a match you have (i.e., $\sum_{(x,y)} D(x,y)^2$). You will need to use a small enough step increment that the pixel diff is likely to work. For instance, in your example image, the one-pound coin has a radius of about 127 pixels; if you sweep over values of $\theta$, increasing by $0.25$ degrees at each step, then you will only need to try about 1460 different rotation values, and the error at the circumference of the coin at the closest approximation to the true $\theta$ should be at most about one-quarter of a pixel, which is small enough that the pixel diff might work out OK.

   You may want to experiment with multiple variations on this idea. For instance, you could work with a grayscale version of the image; the full RGB, and use a $L_2$ norm over all three R,G,B differences; the full HSB, and use a $L_2$ norm over all three H,S,B differences; or work with just the Hue, Saturation, or Brightness plane. Also, another possibility would be to first run an edge detector on both $R$ and $C$, then match up the resulting image of edges.

   For robustness, you might have multiple different reference images for each coin (in fact, each side of each coin), and try all of the reference images to find the best match.

   If images of the coins aren't taken from directly head-on, then as a first step you may want to compute the ellipse that represents the perimeter of the coin $C$ in the image and infer the angle at which the coin is being viewed. This will let you compute what $R$ would look like at that angle, before performing the matching.

5. Check how color varies as a function of distance from the center? Here is a possible intermediate step in between "the coin's mean color" (a single number, i.e., 0-dimensional) and "the entire image of the coin" (a 2-dimensional image). For each coin, you could compute a 1-dimensional vector or function $f$, where $f(r)$ represents the mean color of the pixels at distance approximately $r$ from the center of the coin. You could then try to match the vector $f_C$ for a coin $C$ in your image against the vector $f_R$ for a reference image $R$ of that coin.

   This might let you correct for illumination differences. For instance, you might be able to work in grayscale, or in just a single bitplane (e.g., Hue, or Saturation, or Brightness). Or, you might be able to first normalize the function $f$ by subtracting the mean: $g(r) = f(r)-\mu$, where $\mu$ is the mean color of the coin -- then try to match $g_C$ to $g_R$.

   The nice thing about this approach is that you don't need to infer how much the coin was rotated: the function $f$ is rotation-invariant.

   If you want to experiment with this idea, I would compute the function $f_C$ for a variety of different example images and graph them. Then you should be able to visually inspect them to see if the function seems to have a relatively consistent shape, regardless of illumination. You might need to try this for multiple different possibilities (grayscale, each of the HSB bitplanes, etc.).

   If the coin $C$ might not have been photographed from directly head-on, but possibly from an angle, you'll first need to trace the ellipse of $C$'s perimeter to deduce the angle from which it was photographed and then correct for that in the calculation of $f$.

6. Look at vision algorithms for color constancy. The computer vision community has studied color constancy, the problem of correcting for an unknown illumination source; see, e.g., this overview. You might explore some of the algorithms derived for this problem; they attempt to infer the illumination source and then correct for it, to derive the image you would have obtained had the picture been taken with the reference illumination source.

7. Look into Color Constant Color Indexing. The basic idea of CCCI, as I understand it, is to first cancel out the unknown illumination source by replacing each pixel's R value with the ratio between its R-value and one of its neighbor's R-values; and similarly for the G and B planes. The idea is that (hopefully) these ratios should now be mostly independent of the illumination source. Then, once you have these ratios, you compute a histogram of the ratios present in the image, and use this as a signature of the image. Now, if you want to compare the image of the coin $C$ to a reference image $R$, you can compare their signatures to see if they seem to match. In your case, you may also need to adjust for angle if the picture of the coin $C$ was not taken head-on -- but this seems like it might help reduce the dependence upon illumination source.

I don't know if any of these has a chance of working, but they are some ideas you could try.

Answer 3

Interesting problem and good work.

Try using median colour values rather than mean. This will be more robust against outlier values due to brightness and saturation. Try using just one of the RGB components instead of all three. Choose the component that best distinguishes the colours. You could try plotting histograms of the pixel values (e.g. one of the RGB components) to give you an idea of the properties of the pixel distribution. This might suggest a solution that is not immediately obvious. Try ploting the RGB components in 3D space to see if they follow any pattern, for example they may lie close to a line indicating that a linear combination of the RGB components may be a better classifier than an individual one.