7
$\begingroup$

Below is a scatter plot (capped at $10k) representing the average donation a project receives vs the word count of the funding request essay for all projects represented in the open Donors Choose Data.

donation amount vs essay length

There is a noticeable pattern, which I tried to characterize by fitting the curve

$$ f(x)=\left(\frac{a}{x-b}\right)^2 $$

through manual parameter manipulation. However, I'd like to know other ways to approach modeling or finding patterns/relationships in data that looks like this.


Here is the disparity that motivates my search for other methods:

In the canonical example for linear regression, the scattered points are deviations from a curve. In this example, that clearly isn't the case, as it seems the points are clustered under some area.

$\endgroup$
  • 2
    $\begingroup$ Do your data contain exact zeros in the y-variable? What about the x-variable? Why did you cap it? How much higher do values go? $\endgroup$ – Glen_b -Reinstate Monica May 5 '16 at 5:08
  • $\begingroup$ Yes, exact zeroes in both. I capped to better visualize the structure in the neighborhood depicted, exorbitant values above 10k distorted the plot. The values go as high as $\sim$ 100k $\endgroup$ – brownie_in_motion May 5 '16 at 16:39
  • 4
    $\begingroup$ The "pattern" appears illusory. Apparently, it's trying to trace out some upper envelope of the response. Although that could be done, it's not possible with such a cluttered plot, and it clearly has not been carried out correctly, because at the left the trace corresponds to extremely high percentiles but at the right it marks lower and lower percentiles. Consider performing a more illuminating exploration of the data, such as using your software to draw traces of selected percentiles for narrow bands of word counts. $\endgroup$ – whuber May 5 '16 at 20:28
  • 4
    $\begingroup$ It's hard to tell because the scatter plot is saturated, but I'd guess that the pattern, i.e. increased funding in the 500-3000 word range, is an artifact due to a higher density of data points in that range. If you try whuber's suggestion, the average funding as a function of word count might look a lot less dramatic. $\endgroup$ – R Greg Stacey May 5 '16 at 21:00
  • 1
    $\begingroup$ Agree with @whuber. I think you may be trying to apply something like a univariate density estimate to a bivariate scatterplot, but this doesn't make a lot of sense. A more appropriate tool would be a bivariate histogram or density. $\endgroup$ – dsaxton May 6 '16 at 15:40
15
+250
$\begingroup$

Just to elaborate on my comment, here's an example of how your apparent pattern could be an artifact caused by the distribution of data along the x-axis. I generated 100,000 data points. They're normally distributed in the x-axis ($\mu = 2500, \sigma =600$) and exponentially distributed in the y-axis ($\lambda = 1$).

enter image description here

Following the "visual envelope" of the scatter plot, there's a clear, although illusory, pattern: y looks maximal in the range 1000< x<4000. However, this apparent pattern, very convincing visually, is just an artifact caused by the distribution of x values. That is, there's just more data in the range 1000< x<4000. You can see this in the x-histogram on the bottom.

To prove it, I calculated the average y value in bins of x (black line). This is approximately constant for all x. If the data was distributed according to our intuition from the scatter plot, the average in the 1000< x<4000 range should be higher than the rest - but it's not. So there really is no pattern.

I'm not saying this is the whole story with your data. But I would bet it's a partial explanation.

Addendum with actual Donors Choose data.

Original scatterplot with overstriking markers:

enter image description here

Same scatterplot with reduced opacity:

enter image description here

Different patterns appear, but with 800K data points, there is still a lot of detail lost to overstriking.

Zoom, reduce opacity again and add smoother:

enter image description here

$\endgroup$
  • 1
    $\begingroup$ Sometimes it helps to use transparency on the markers to get an idea of the density. $\endgroup$ – xan May 6 '16 at 13:16
  • 2
    $\begingroup$ @xan Agreed. I have found that with this many markers (or practically any number much greater than 10K) you have to use the maximum possible transparency to see what's happening in the center of a point cloud, such as col="#00000001" in R. With almost a million points smoothing is essential. It's a good idea to make its range much shorter than one typically uses for smaller point clouds, so that it picks up more local details. $\endgroup$ – whuber May 9 '16 at 15:27
3
$\begingroup$

I would guess your variable on Y axis is exponentially distributed ($p(y) = \lambda e^{-\lambda y}$), but it seems that the rate parameter $\lambda$ is varying accordingly to the normal density probability of your variable on X axis.

I've generated random data with MatLab using normal distribution for X and exponential distribution for Y, with $\lambda = p(x)$ and I got a similar result with your data:

enter image description here

You could try machine learning to fit the parameters, changing your cost function to compare the probability density and the rate parameter for each bin on your 'histogram'. If so, don't forget to run the random generator a few times on each iteration to minimize the cost.

Here is the code I used for the plot:

% Normal distribution generation.
x = randn(10000,1);
x = x - min(x);                     % Shifting curve so every x is > 0.

% Histogram informations
k = 100;                            % Number of bins.
binSize = (max(x) - min(x)) / k;    % Width of bins.
y = 0:(k);
y = y .* binSize + min(x);          % Array with Intervals.

p = zeros(k,1);
data = [];

% For every bin...
for i = 1:k
    a = x(x >= y(i) & x < y(i + 1));    % All X values within condition.
    p(i) = size(a,1);                   % Number of occurences (or
                                        % Normal Density Probability).

    if ~isempty(a)
        for j = 1:p(i)

            % lambda = Rate parameter of exponential distribution
            % Rate parameter is varying with normal density probability.
            lambda = p(i);

            % Every X in normal distribution will have a Y
            % which was generated randomly by the exponential 
            % distribution function EXPRND.
            data = [data; a(j), exprnd(lambda)];

        end
    end
end

% Plotting normal distribution VS modified exponential distribution
scatter(data(:,1),data(:,2))
$\endgroup$
1
$\begingroup$

The question mentions regression, which typically addresses conditional expectation: $$ E[y|x] = \int y\,p(y|x)\,dy, $$ where $y$ is the average donation and $x$ is the number of words. Linear regression may be too restrictive and so one could apply a local regression approach such as Nadaraya-Watson kernel regression. The results could be sensitive to the choice of bandwidth: A wide bandwidth could mask interesting local variation.

More generally, the question of independence between $x$ and $y$ is interesting. If $x$ and $y$ are independent then $p(y|x) = p(y)$ and of course the conditional expectation is independent as well. But $y$ might depend on $x$ in intersting ways even if the conditional expectation is independent of $x$.

With so much data I would look at histograms of $y$ that all have the nearly the same value of $x$ and see how the histogram changes as the chosen value of $x$ changes. Only after such an investigation would I think about how to proceed more formally.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.