I am running into a problem where an independent variable, which should have no predictive power on the dependent variable based on domain knowledge, comes out with very small p-value because the sample size is very large(~100,000). If I only use < 5000 data points, then the p-value becomes large enough to support the prior that the variable is insignificant. However, I don't think tweaking the sample size to get to the desired conclusion is a good practice. Is there any procedure to adjust for small p-value simply due to huge sample size?
You're valuing the p-value far too highly. Report the magnitude of the statistically significant effect. It should be such a small value that statistically significant is pretty irrelevant. In terms of something like Cohen's d, it takes an effect size of about 0.006 to need N's that large to be found. Talk about the effect size reasonably. That's what you should be doing for all of your effects, significant or not, expected or not.
A few years later... The paper by Naaman (2016) Almost sure hypothesis testing and a resolution of the Jeffreys-Lindley paradox seems relevant. From the abstract:
A new method of hypothesis testing is proposed ensuring that as the sample size grows, the probability of a type I error will become arbitrarily small by allowing the significance level to decrease with the number of observations in the study.
see also the Wikipedia entry for Almost sure hypothesis testing.
Keep in mind the truth that "correlation is not causation". That is, while you might not see WHY there should be a predictive relationship, there nonetheless is a strong enough correlation that you receive a small p-value.
As an example: I could perform a health & fitness study on kids who weigh between 75 - 100 lbs. If I also write down their ages & their heights, I would find correlations between both age & height to health as well as weight to health. Why? Well, if I grab a 4 year old who weighs 75 pounds, s/he is likely unhealthy. Similarly if I grab a boy 6'2" who is 100 lbs. This makes sense to you because it is intuitive. The effects in the case you're studying are likely less intuitive.
Why did this occur? Because I put a constraint on one correlated variable which inherently creates relationships with the others. (If I took all weights, then the 6'2" 100 lb boy would be drown out by other 6'2" kids who weigh much more.)
Lastly, keep in mind that the p-value you are calculating is against the null hypothesis (or so I presume). The null hypothesis is merely testing against absolutely no effect. So if you have a mild effect that is so trivial you think is uninteresting, it will still have a low p-value. So as others said, p-value alone is not the end-all-be-all. It is merely a good filter & starting point.