0
$\begingroup$

I have image data, which I can represent as one dimensional vectors. Each value represents the brightness of a pixel in a line.

eg:

(1, 12, 4, 3, 1, 4)

I want to describe the distribution of brightness levels in the vector with single digit summary statistics: how far and in which direction the "bright part" of the image is away from the centre, and how broad the "bright part" is. Together they will describe a region of interest in the underlying image. In general I expect the bright area to be a unimodal peak, or if not, then a diffuse plateau, so the description of these properties doesn't need to be complex.

The vectors will be of an unknown length, so a normalised range of values would be ideal as an output.

I thought of skewness and kurtosis as they seem conceptually similar (at least graphically), but as Nick said below these have no spatial or ordered component. I thought I could use the vector indices to transform my data, and apply a "spatial skewness/kurtosis", but I can't work out how to do this.

I would prefer to use something that seems familiar to my audience, which will largely be medical professionals who have varying amounts of statistical knowledge. In this way, a "spatial skewness" would be a recognisable idea, where another method of describing things may not be.

I am not wedded to this method however (it may not even be possible). If there is a good approach that won't be too computationally intensive I am open to that too.

edit:

additional details and questions follows Nick Cox's answer below about weighted mean positions.

OK, so having thought about this a bit more I think I can try to explain more clearly my purpose. I am trying to efficiently describe the spatial distribution of values in a vector to supply to a machine learning model. The actual vectors in my data are hundreds of numbers long, and contain a lot of noise, so I am essentially trying to smooth and perform dimensionality reduction by describing the overall distribution succinctly.

The weighted mean position is great because it describes where the "mass" of values is in the vector, but it obviously doesn't describe the distribution in other ways. In particular, the shape of the distribution is missed, whether the distribution is peaked or plateau, unimodal or multimodal etc.

In an effort to not overthink things, the solution that comes to me is to also do some binned averages ie average of first 25% of vector, average of second 25% etc.

The only issue I have with this is the inelegance of it. I am using multiple values to describe one "idea", and if I want an even more fine-grained description, I add more values by making the bins smaller. This is why I had gravitated to the concept of kurtosis initially, because it describes something about shape in a single number. I realise it is not applicable at all, but I still wonder if there is a more elegant solution to describing the shape of my distributions.

I understand this is what Nick has hinted at below, with "other weighted moment measures". I am again just running against my lack of mathematical intuition. I am not sure how to make a weighted second, third or fourth moment, let alone have a grasp of which one might be useful here.

$\endgroup$
  • $\begingroup$ Skewness is defined for a set of magnitudes. If you've seen formulas involving frequencies, they are recipes for binned data. Any decent statistical program will give you skewness directly for a vector like yours. Watch out that there are slightly different formulas. Kurtosis doesn't measure asymmetry. But -- most important -- what you want doesn't seem to be either. Averaging the positions on your line with weights given by brightness may be more like it. Thus 255, 0, 0, 0, ... would return an average of position 1 and 0, 0, 0, ..., 255 would return an average of the last position. $\endgroup$ – Nick Cox Jan 15 '16 at 12:44
  • $\begingroup$ (I am guessing very wildly at measurements 0 to 255 but the principle is the same for non-negative values.) $\endgroup$ – Nick Cox Jan 15 '16 at 12:45
  • $\begingroup$ Hmm... This is why I felt confused, I thought skew should work out of the gate. It was just giving me bad numbers when I tested it. I'll retry. Thanks $\endgroup$ – Luke_radio Jan 15 '16 at 12:52
  • $\begingroup$ Skewness won't take any account whatsoever of the positions of the measurements. For example, the skewness is the same if you reverse the vector from last to first or shuffle the elements randomly. So, it's not what you want if I understand you correctly. $\endgroup$ – Nick Cox Jan 15 '16 at 12:54
  • $\begingroup$ Concerning asymmetry, are you perhaps asking the same thing addressed in the thread at stats.stackexchange.com/questions/145296? Please note that the question of "main peak ... high and sharp" is a completely different issue. It is rare that anyone would even care about that; often sharpness of the peak is used as a visual proxy for how long the tails of the distribution are. Could you perhaps tell us in more general, non-statistical terms what features of each image you are trying to describe or learn about? $\endgroup$ – whuber Jan 15 '16 at 15:41
1
$\begingroup$

I am still sketchy on what you want but guess that you are over-thinking this, as the similarity of skewness calculation to your problem is less than it seems. Kurtosis seems even less relevant.

Given brightness along a line -- in practice measured as positive integers, but my argument applies to any non-negative measure -- you (seem to) want to quantify how far values are high on the left compared with the right, and vice versa.

Keeping for simplicity to your positions 1 to 6, and using your toy examples (small detail: one is for 7 values, not 6) it seems to me that weighted mean position will do what you want. The weights are brightness measures; what is being averaged is the position. For example, (20, 0, 0, 0, 0, 0) at positions 1 to 6 will return a mean of 1 and flipping it round will return a mean of 6. A constant brightness will return a mean of 3.5.

Rather than working through trivial calculations, let's just show how the weighted mean positions work out. Here the bars show brightness; the small gap between bars is just cosmetic and could just be removed if anyone insists fairly that pixels are to be considered as touching. The spikes show the means.

enter image description here

For a more general situation, scaling by

2 $\times$ (weighted mean position $-$ mean position) / (length of line $-$ 1)

will produce a measure from $-$1 (high values at extreme left) through 0 to 1 (high values at extreme right). Simple variations on such a theme are naturally possible. In particular, its absolute value quantifies departure from the centre regardless of left or right.

Similarly, other weighted moment measures may be useful.

EDIT: Factor of 2 introduced into displayed equation.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I suspect you are right that I am over-thinking it. I will think about the weighted mean, when you suggested it earlier I tried it briefly and thought there might be some problems with it, but I will work through it properly with my data and see if it does what I need. Cheers. $\endgroup$ – Luke_radio Jan 19 '16 at 23:42
  • $\begingroup$ I have gone through this and it definitely does part of what I am looking for, so I am happy to tick your answer. I have added a bit more to my original question about what is left of my problem if you or anyone else felt inclined to assist further. Thanks. $\endgroup$ – Luke_radio Jan 23 '16 at 11:16
  • $\begingroup$ I'll try to come back to this soon. As an interim comment, I would suggest the weighted SD of position and weighted quantiles (e.g. deciles, quartiles, median) of position as the easiest elaborations or alternatives to my first suggestion. Chopping each line into 4 parts sounds more arbitrary. $\endgroup$ – Nick Cox Jan 23 '16 at 18:12
0
$\begingroup$

I know I did not ask this question well, but I have found what I think is the start of an answer. I am posting it for feedback/review, in case it is useful to anyone else in the future. It does piggyback on skewness and kurtosis.

I would appreciate any attempts to poke holes in it. It works for some toy problems, but has some weaknesses.

I find the midpoint of the line (the middle value) and index each position relative to that. So in my above example:

x =  (1, 12, 4, 3, 1, 4)

the original index, length and midpoint are:

y =  (1, 2, 3, 4, 5, 6)
length_y = 6
midpoint_y = length_y/2 + 0.5

then:

(y - midpoint_y)

gives me a relative-to-midpoint range for my index, in this case:

new_index = (-2.5, -1.5, -0.5, 0.5, 1.5, 2.5)

I can then transform each value in the original vector with its new index position (image data is only positive integers, so this should work fine):

x * new_index = (-2.5, -18.0, -2.0, 6.5, 1.5, 10.0)

I take this result and run the normal Pearson skewness (third moment/second moment$^{3/2}$) which returns:

-0.7898

If this was normal skewness, the distribution would have a long tail to the left. That doesn't seem reassuring, because spatially the tail is to the right. But more testing shows:

input vector = (2,4,2,1,4,18,14)
output = 0.8561

input vector = (1,12,14,13,1,4)
output = -0.5278

input vector = (12,10,5,5,10,12)
output = 0

Which looks sign flipped. So if I just multiply by -1, it looks like a good start on a "spatialised skewness" measurement.

So far, looks OK. The bigger the values and the further from the middle, the bigger the output. But it falls apart again because the spatial order isn't maintained in the skewness calculation. If there are lower values outside the "peak", the output value will decrease as the end value goes up!

for example:

input vector = (1, 12, 4, 3, 1, 4)
output = 0.7898

input vector = (2, 12, 4, 3, 1, 4)
output = 0.6267

input vector = (3, 12, 4, 3, 1, 4)
output = 0.4767

which is not the output I am looking for. The overall "weight" of the numbers should be skewing the output value further, but because the further left value (when multiplied by its index position) is smaller than the 2nd from the left value, it actually pulls the skew back to the centre.

The other problem is that the output value increases as the vector length increases, meaning I have no standardised values for comparison if vector lengths are different (they are!). I'm not sure what to do about that either.

I could be barking up the wrong tree entirely here. Hopefully if nothing else, this better explains what I am trying to do.

P.S. I am ignoring kurtosis for the moment. The current system applies to that too, but likewise fails in funny ways.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.