How to normalize mortality in determining lethal concentration of toxins or pesticides? A certain set of pesticide concentrations is selected to dose the animals to determine its acute toxicity. Technically, with increasing concentrations, the mortality should increase, but that does not occur. In some cases, in higher dose category the mortality will be less than that of the preceding or earlier dose category. How to overcome this problem? If there is any way around like through mathematical formula or any paper that sheds light on this aspect please enlighten me.
The data is as follows: five treatments and seven fish per treatment

*

*First treatment: none dead

*Second treatment: two out of seven dead

*Third treatment: one out of seven dead

*Fourth treatment: all dead

*Fifth treatment: All dead

 A: tl;dr A standard probit regression analysis should be fine. The only thing "wrong" with your data is that the sample sizes are a bit low, so the outcomes are highly variable.
This graph shows the probit regression along with the binomial confidence intervals for the individual points (code included below since it's a little bit complicated):

(This analysis assumes the doses are equally spaced, or equally spaced on a log scale, or in any case should be treated as equally spaced.)
You can see that the unusual result of 1/7 (14%) dead for dose 3, although it surprises you, is not even particularly inconsistent with the probit curve; the 95% confidence interval for a binomial probability with 1/7 deaths is (0.004, 0.58), while the predicted value from the probit is very close to 0.5. (It would unsurprising even if some of the individual binomial CIs didn't overlap with the curve, because we are making multiple comparisons.)
If the sample sizes were much larger, e.g. if you had 1000/7000 deaths instead of 1/7, so that the expected level of precision was much higher, then I would worry that something had gone wrong with the experiment.
The probit model can be fitted with base R (glm(prop_dead ~ dose, data = ..., weights = n_trials, family = binomial(link="probit"))) but you might find the drc package useful (or the dose.p function from the built-in MASS package, or this link).
dd <- data.frame(dead = c(0, 2, 1, 7, 7),
                 n = 7,
                 dose = 1:5)
dd$lwr <- dd$upr <- NA
for (i in seq(nrow(dd))) {
  bb <- binom.test(dd$dead[i], dd$n[i])
  dd$lwr[i] <- bb$conf.int[1]
  dd$upr[i] <- bb$conf.int[2]
}

library(ggplot2); theme_set(theme_bw())
(ggplot(dd)
  + aes(x = dose, y = dead/n)
  + geom_pointrange(aes(ymin = lwr, ymax = upr))
  + geom_smooth(method = "glm",
                method.args = list(family = binomial(link = "probit")),
                aes(weight = n))
)

