I'm trying to fit a xgboost model to some data and am getting the following results using a random 70/30 split of train/test data:
 train-rmse:885.071777 test-rmse:4659.329102  train-rmse:763.531128 test-rmse:4633.854980  train-rmse:669.896545 test-rmse:4616.207031  train-rmse:599.979797 test-rmse:4603.926270  train-rmse:546.986206 test-rmse:4595.299316  train-rmse:508.586578 test-rmse:4589.103027  train-rmse:479.262634 test-rmse:4584.626953  train-rmse:457.594482 test-rmse:4581.324707  train-rmse:441.262756 test-rmse:4578.702637  train-rmse:429.211090 test-rmse:4576.885742  train-rmse:419.566010 test-rmse:4575.384277  train-rmse:412.783600 test-rmse:4574.195801  train-rmse:407.107574 test-rmse:4573.338379  train-rmse:402.664185 test-rmse:4572.566895  train-rmse:399.317749 test-rmse:4572.004395
I would tend to think that this indicates over-fitting - if so what parameters could I tune to reduce the cross-validated error? No matter what parameters I change it doesn't make much difference. I have adjusted eta, min_child_weight, max_depth all with no significant impact on the cv error. Any ideas would be much appreciated.