In the case of count data where my variance is 6.34 and mean is 5.44, which would be a better fit to the data - Poisson or negative binomial? And what will be the assumptions ?
Part of this depends on what you mean by "better". Since a mixture of Poisson distributions (where $\lambda$ follows a Gamma distribution) is a Negative Binomial you will be able to fit a negative binomial to data from a Poisson, and it will almost always give a "better" fit if there are no penalty terms. Measures like "AIC" and "BIC" start with the likelihood, but also penalize for the number of parameters fit, so data that does not really benefit from the NB would probably have a better score with the Poisson fit.
Data that follows a NB may be fine in a Poisson regression, because the predictors are explaining the mixing part.
It is best to approach this from the science, does your understanding of the science that produces the data match the ideas behind a Poisson? or would it make more sense that there would be different Poissons mixed together? Or something else entirely.