How to model this odd-shaped distribution (almost a reverse-J) My dependent variable shown below doesn't fit any stock distribution that I know of.  Linear regression produces somewhat non-normal, right-skewed residuals that relate to predicted Y in an odd way (2nd plot).  Any suggestions for transformations or other ways to obtain most valid results and best predictive accuracy?  If possible I'd like to avoid clumsy categorizing into, say, 5 values (e.g., 0, lo%, med%, hi%, 1).


 A: Are the values always between 0 and 1?
If so you might consider a beta distribution and beta regression.
But make sure to think through the process that leads to your data.  You could also do a 0 and 1 inflated model (0 inflated models are common, you would probably need to extend to 1 inflated by your self).  The big difference is if those spikes represent large numbers of exact 0's and 1's or just values close to 0 and 1.
It may be best to consult with a local statistician (whith a non-disclosure agreement so that you can discuss the details of where the data come from) to work out the best approach.
A: Methods of censored regression can handle data like this.  They assume the residuals behave as in ordinary linear regression but have been modified so that


*

*(Left censoring): all values smaller than a low threshold, which is independent of the data, (but can vary from one case to the other) have not been quantified; and/or

*(Right censoring): all values larger than than a high threshold, which is independent of the data (but can vary from one case to the other) have not been quantified.
"Not quantified" means we know whether or not a value falls below (or above) its threshold, but that's all.
The fitting methods typically use maximum likelihood.  When the model for the response $Y$ corresponding to a vector $X$ is in the form
$$Y \sim X \beta + \varepsilon$$
with iid $\varepsilon$ having a common distribution $F_\sigma$ with PDF $f_\sigma$ (where $\sigma$ are unknown "nuisance parameters"), then--in the absence of censoring--the log likelihood of observations $(x_i, y_i)$ is
$$\Lambda = \sum_{i=1}^n \log f_\sigma(y_i - x_i\beta).$$
With censoring present we may divide the cases into three (possibly empty) classes: for indexes $i=1$ to $n_1$, the $y_i$ contain the lower threshold values and represent left censored data; for indexes $i=n_1+1$ to $n_2$, the $y_i$ are quantified; and for the remaining indexes, the $y_i$ contain the upper threshold values and represent right censored data.  The log likelihood is obtained in the same way as before: it is the log of the product of the probabilities.
$$\Lambda = \sum_{i=1}^{n_1} \log F_\sigma(y_i - x_i\beta) + \sum_{i=n_1+1}^{n_2} \log f_\sigma(y_i - x_i\beta) + \sum_{i=n_2+1}^n \log (1 - F_\sigma(y_i - x_i\beta)).$$
This is maximized numerically as a function of $(\beta, \sigma)$.
In my experience, such methods can work well when less than half the data are censored; otherwise, the results can be unstable.

Here is a simple R example using the censReg package to illustrate how OLS and censored results can differ (a lot) even with plenty of data.  It qualitatively reproduces the data in the question.
library("censReg")
set.seed(17)
n.data <- 2960
coeff  <- c(-0.001, 0.005)
sigma  <- 0.005
x      <- rnorm(n.data, 0.5)
y      <- as.vector(coeff %*% rbind(rep(1, n.data), x) + rnorm(n.data, 0, sigma))
y.cen           <- y
y.cen[y < 0]    <- 0
y.cen[y > 0.01] <- 0.01
data = data.frame(list(x, y.cen))

The key things to notice are the parameters: the true slope is $0.005$, the true intercept is $-0.001$, and the true error SD is $0.005$.
Let's use both lm and censReg to fit a line:
fit <- censReg(y.cen ~ x, data=data, left=0.0, right=0.01)
summary(fit)

The results of this censored regression, given by print(fit), are
(Intercept)           x       sigma 
  -0.001028    0.004935    0.004856 

Those are remarkably close to the correct values of $-0.001$, $0.005$, and $0.005$, respectively.
fit.OLS <- lm(y.cen ~ x, data=data)
summary(fit.OLS)

The OLS fit, given by print(fit.OLS), is
(Intercept)            x  
   0.001996     0.002345  

Not even remotely close!  The estimated standard error reported by summary is $0.002864$, less than half the true value.  These kinds of biases are typical of regressions with lots of censored data.
For comparison, let's limit the regression to the quantified data:
fit.part <- lm(y[0 <= y & y <= 0.01] ~ x[0 <= y & y <= 0.01])
summary(fit.part)

(Intercept)  x[0 <= y & y <= 0.01]  
   0.003240               0.001461  

Even worse!
A few pictures summarize the situation.
lineplot <- function() {
  abline(coef(fit)[1:2], col="Red", lwd=2)
  abline(coef(fit.OLS), col="Blue", lty=2, lwd=2)
  abline(coef(fit.part), col=rgb(.2, .6, .2), lty=3, lwd=2)
}
par(mfrow=c(1,4))
plot(x,y, pch=19, cex=0.5, col="Gray", main="Hypothetical Data")
lineplot()
plot(x,y.cen, pch=19, cex=0.5, col="Gray", main="Censored Data")
lineplot()
hist(y.cen, breaks=50, main="Censored Data")
hist(y[0 <= y & y <= 0.01], breaks=50, main="Quantified Data")


The difference between the "hypothetical data" and "censored data" plots is that all y-values below $0$ or above $0.01$ in the former have been moved to their respective thresholds to produce the latter plot.  As a result, you can see the censored data all lined up along the bottom and top.
Solid red lines are the censored fits, dashed blue lines the OLS fits, both of them based on the censored data only. The dashed green lines are the fits to the quantified data only. It is clear which is best: the blue and green lines are noticeably poor and only the red (for the censored regression fit) looks about right.  The histograms at the right confirm that the $Y$ values of this synthetic dataset are indeed qualitatively like those of the question (mean = $0.0032$, SD = $0.0037$).  The rightmost histogram shows the center (quantified) part of the histogram in detail.
A: In concordance with Greg Snow's advice I've heard beta models are useful in such situations as well (see a Smithson & verkuilen, 2006, A Better Lemon Squeezer), as well as quantile regression (Bottai et al., 2010), but these seem like so pronounced floor and ceiling effects they may be inappropriate (especially the beta regression). 
Another alternative would be to considered types of censored regression models, in particular the Tobit Model, where we consider the observed outcomes to be generated by some underlying latent variable that is continuous (and presumably normal). I'm not going to say this underlying continuous model is reasonable given your histogram, but you can find some support for it as you see the distribution (ignoring the floor) has a higher density at lower values of the instrument and slowly curtails to higher values.
Good luck though, that censoring is so dramatic it is hard to imagine recovering much useful information within the extreme buckets. It looks to me like nearly half of your sample falls within the floor and ceiling bins. 
