Some things you can try:

1. Oversample your target classes. Insert duplicate records of for your other three classes to augment your training dataset

2. Undersample the negative responses. Instead of including all instances of other in your training data, only use a small portion.

3. Bootstrap undersample the negative responses. This is probably your most robust option of those I'm presenting. Start by seeding your training data with the non-other classified records. Then train each bootstrap iteration augmenting the seed training set with a different random sample (selected with replacement) from the other class. You can then either derive a confidence interval for your model's classifications from the bootstrapping procedure (as Kaushik suggested), or treat the model's you generated as an ensemble and either average their scores or have the models vote to determine your classifications. You can even implement boosting here if you want.

Some things you can try:

1. Oversample your target classes. Insert duplicate records of your other three classes to augment your training dataset

2. Undersample the negative responses. Instead of including all instances of other in your training data, only use a small portion.

3. Bootstrap undersample the negative responses. This is probably your most robust option of those I'm presenting. Start by seeding your training data with the non-other classified records. Then train each bootstrap iteration augmenting the seed training set with a different random sample (selected with replacement) from the other class. You can then either derive a confidence interval for your models classifications from the bootstrapping procedure (as Kaushik suggested), or treat the models you generated as an ensemble and combine their scores using an average or majority vote to determine your classifications. You can even implement boosting here if you want.