I know non-parametric relies on the median instead of the mean
Hardly any nonparametric tests actually "rely on" medians in this sense. I can only think of a couple... and the only one I expect you'd be likely to have even heard of would be the sign test.
to compare...something.
If they relied on medians, presumably it would be to compare medians. But - in spite of what a number of sources try to tell you - tests like the signed rank test, or the Wilcoxon-Mann-Whitney or the Kruskal-Wallis are not really a test of medians at all; if you make some additional assumptions, you can regard the Wilcoxon-Mann-Whitney and the Kruskal-Wallis as tests of medians, but under the same assumptions (as long as the distributional means exist) you could equally regard them as a test of means.
The actual location-estimate relevant to the Signed Rank test is the median of pairwise averages within-sample (over $\frac12 n(n+1)$ pairs including self-pairs), the one for the Wilcoxon-Mann-Whitney is the median of pairwise differences across-samples.
I also believe it relies on "degrees of freedom?" instead of standard deviation. Correct me if I'm wrong though.
Most nonparametric tests don't have 'degrees of freedom' in the specific sense that the chi-squared or the t-test of the F-test do (each of which has to do with the number of degrees of freedom in an estimate of variance), though the distribution of many change with sample size and you might regard that as somewhat akin to degrees of freedom in the sense that the tables change with sample size. The samples do of course retain their properties and have n degrees of freedom in that sense but the degrees of freedom in the distribution of a test statistic is not typically something we're concerned with. It can happen that you have something more like degrees of freedom - for example, you could certainly make an argument that the Kruskal-Wallis does have degrees of freedom in basically the same sense that a chi-square does, but it's usually not looked at that way (for example, if someone's talking about the degrees of freedom of a Kruskal-Wallis, they will nearly always mean the d.f. of the chi-square approximation to the distribution of the statistic).
A good discussion of degrees of freedom may be found here/
I've done pretty good research, or so I've thought, trying to understand the concept, what the workings are behind it, what the test results really mean, and/or what to even do with the test results; however no one seems to ever venture into that area.
I'm not sure what you mean by this.
I could suggest some books, like Conover's Practical Nonparametric Statistics, and if you can get it, Neave and Worthington's book (Distribution-Free Tests), but there are
many others - Marascuilo & McSweeney, Hollander & Wolfe, or Daniel's book for example. I suggest you read at least 3 or 4 of the ones that speak to you best, preferably ones that explain things as differently as possible (this would mean at least reading a little of perhaps 6 or 7 books to find say 3 that suit).
For the sake of simplicity lets stick with the Mann Whitney U test, which I've noticed is quite popular
It is, which is what puzzled me about your statement "no one seems to ever venture into that area" - many people who use these tests do 'venture into the area' you were talking about.
- and also seemingly misused and overused
I'd say nonparametric tests are generally underused if anything (including the Wilcoxon-Mann-Whitney) -- most especially permutation/randomization tests, though I wouldn't necessarily dispute that they're frequently misused (but so are parametric tests, even more so).
Let's say I run a non-parametric test with my data and I get this result back:
[snip...]
I'm familiar with other methods, but what is different here?
Which other methods do you mean? What do you want me to compare this to?
Edit: You mention regression later; I assume then that you are familiar with a two-sample t-test (since it's really a special case of regression).
Under the assumptions for the ordinary two-sample t-test, the null hypothesis has that the two populations are identical, against the alternative that one of the distributions has shifted. If you look at the first of the two sets of hypotheses for the Wilcoxon-Mann-Whitney below, the basic thing being tested there is almost identical; it's just that the t-test is based on assuming the samples come from identical normal distributions (apart from possible location-shift). If the null hypothesis is true, and the accompanying assumptions are true, the test statistic has a t-distribution. If the alternative hypothesis is true, then the test-statistic becomes more likely to take values that don't look consistent with the null hypothesis but do look consistent with the alternative -- we focus on the most unusual, or extreme outcomes (the ones most consistent with the alternative) - if they occur, we conclude that the samples we got would not have occurred by chance when the null was true (they could do, but the probability of a result at least that much consistent with the alternative is so low that we consider the alternative hypothesis a better explanation for what we observe than "the null hypothesis along with the operation of chance").
The situation is very similar with the Wilcoxon-Mann-Whitney, but it measures the deviation from the null somewhat differently. In fact, when the assumptions of the t-test are true*, it's almost as good as the best possible test (which is the t-test).
*(which in practice is never, though that's not really as much of a problem as it sounds)
Indeed, it's possible to consider the Wilcoxon-Mann-Whitney as effectively a "t-test" performed on the ranks of the data - though then it doesn't have a t-distribution; the statistic is a monotonic function of a two-sample t-statistic computed on the ranks of the data, so it induces the same ordering on the sample space (that is a "t-test" on the ranks - appropriately performed - would generate the same p-values as a Wilcoxon-Mann-Whitney), so it rejects exactly the same cases.
[You'd think that just using the ranks would be throwing away a lot of information, but when the data are drawn from normal populations with the same variance, almost all the information about location-shift is in the patterns of the ranks. The actual data values (conditional on their ranks) add very little additional information to that. If you go heavier-tailed than normal, it's not long before the Wilcoxon-Mann-Whitney test has better power, as well as retaining its nominal significance level, so that 'extra' information above the ranks eventually becomes not just uninformative but in some sense, misleading. However, near-symmetric heavy-tailedness is a rare situation, outside some specific applications; what you often tend to see in practice is skewness.]
The basic ideas are quite similar, the p-values have the same interpretation (the probability of a result as, or more extreme, if the null hypothesis were true) -- right down to the interpretation of a location-shift, if you make the requisite assumptions (see the discussion of the hypotheses near the end of this post).
If I did the same simulation as in the plots above for the t-test, the plots would look very similar - the scale on the x- and y-axes would look different, but the basic appearance would be similar.
Should we want the p-value to be lower than .05?
You shouldn't "want" anything there. The idea is to find out if the samples are more different (in a location-sense) than can be explained by chance, not to 'wish' a particular outcome.
If I say "Can you go see what color Raj's car is please?", if I want an unbiased assessment of it I don't want you to be going "Man, I really, really hope it's blue! It just has to be blue". Best to just see what the situation is, rather than to go in with some 'I need it to be something'.
If your chosen significance level is 0.05, then you'll reject the null hypothesis when the p-value is ≤ 0.05. But failure to reject when you have a big enough sample size to nearly always detect relevant effect-sizes is at least as interesting, because it says that any differences that exist are small.
What does the "mann whitley" number mean?
The Mann-Whitney statistic.
There's meaning in comparing its value with the distribution of values it can take when the null hypothesis is true (see the above diagram), and that depends on which of several particular definitions any particular program might use.
Is there any use for it?
Usually you don't care about the exact value as such, but where it lies in the null-distribution (whether it's more or less typical of the values you should see when the null hypothesis is true, or whether it's more extreme)
(Edit: You can obtain or work out some directly informative quantities when doing such a test - like the location shift or $P(X<Y)$ discussed below, and indeed you can work out the second one fairly directly from the statistic, but the statistic alone isn't a very informative number)
Does this data here just verify or not verify that a particular source of data I have should or should not be used?
This test doesn't say anything about "a particular source of data I have should or should not be used".
See my discussion of the two ways of looking at the WMW hypotheses below.
I have a reasonable amount of experience with regression and the basics, but am very curious about this "special" non-parametric stuff
There's nothing particularly special about nonparametric tests (I'd say the 'standard' ones are in many ways even more basic than the typical parametric tests) -- as long as you actually understand hypothesis testing.
That's probably a topic for another question, however.
There are two main ways to look at the Wilcoxon-Mann-Whitney hypothesis test.
i) One is to say "I'm interested in location-shift - that is that under the null hypothesis, the two populations have the same (continuous) distribution, against the alternative that one is 'shifted' up or down relative to the other"
The Wilcoxon-Mann-Whitney works very well if you make this assumption (that your alternative is just a location shift)
In this case, the Wilcoxon-Mann-Whitney actually is a test for medians ... but equally it's a test for means, or indeed any other location-equivariant statistic (90th percentiles, for example, or trimmed means, or any number of other things), since they're all affected the same way by location-shift.
The nice thing about this is that it's very easily interpretable -- and it's easy to generate a confidence interval for this location-shift.
However, the Wilcoxon-Mann-Whitney test is sensitive to other kinds of difference than a location shift.
ii) The other is to take the fully general approach. You can characterize this as a test for the probability that a random value from population 1 is less than a random value from population 2 (and indeed, you can turn your Wilcoxon-Mann-Whitney statistic into a direct estimate of that probability, if you're so inclined; the Mann&Whitney formulation in terms of U-statistics counts the number of times one exceeds the other in the samples, you only need scale that to achieve an estimate of the probability); the null is that the population probability is $\frac{1}{2}$, against the alternative that it differs from $\frac{1}{2}$.
However, while it can work okay in this situation, the test is formulated on the assumption of exchangability under the null. Among other things that would require that in the null case the two distributions are the same. If we don't have that, and are instead are in a slightly different situation like the one pictured above, we won't typically have a test with significance level $\alpha$. In the pictured case it would likely be a bit lower.
So while it "works" in the sense that it tends not to reject when $H_0$ is true and tends to reject more when $H_0$ is false, you want the distributions to be pretty close to identical under the null or the test doesn't behave the way we would expect it to.
