Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 4 down vote favorite

I have about 400 pieces of silver of different geometric dimensions. They were assigned to six groups and each group went through a series of stress tests, such as bending, pulling, putting in fire for a period of time, etc. The treatments that were given to the six groups were not the same, but fairly similar. The sizes of the six groups were not the same. The pieces either broke at some stage and that was recorded as a success or didn't, which was recorded as a failure. The time of each success was also recorded. The number of successes was about 80.

My goal is build a predictive model to determine if a piece of silver breaks based on its physical dimensions and the treatment it goes through.

I have been somewhat successful in building a model using the physical dimensions, but adding various aspects of the treatment (eg. total time spent in fire) didn't improve the performance at all. I have even tried to build features (eg.total stress on the metal in various directions, total strain on the metal, etc.) based on the physical dimensions and the treatment, for each individual piece, but even these didn't add any predictive performance.

How can I incorporate the treatment information in a way that adds to my predictive power? It is clear that the treatment is a factor in whether a piece breaks or not, and it should somehow show up somewhere.

N.B. I didn't have any control over the design of the treatment, and testing more samples with other treatments is not an option for me.

I'd very much appreciate any suggestions or comments.
Many thanks!

share|improve this question
1  
Hi there, what techniques/models have you used so far for analysing your data? Can you also show the variables you used in each model? – Michelle Mar 17 '12 at 4:56
Hi Michelle, I've tried almost every classification scheme that I could get my hands on off the shelf, but Naive Bayes seems to work best. I'm not quite sure what you mean by the list of variables. Are you referring to those coming from the treatment? For the treatment variables, I've tried everything I could think of: number hours being bent, maximum heat, maximum pressure. Anything I could think of. – user765195 Mar 17 '12 at 5:22
1  
I'd be interested to hear more about anything you may have found using logistic regression, and whether regression diagnostics could shed any light on your problem. This is a bit of a long shot, but perhaps incorporating nonlinear terms for one or more treatment predictors would help. – rolando2 Mar 18 '12 at 13:34
Hi rolando2. I got perhaps the poorest results from using logistic regression. I have no idea why. Regularized logistic regression did much better. I didn't use any nonlinear terms though. – user765195 Mar 18 '12 at 19:15
1  
I would try to go at it differently. Do another round of experiments with just one treatment (e.g. bending) and with enough variation. Make it simple enough that you know there is a rule, and that you actually know what the rule is (e.g. bending force > X => success). You can then pick features you think are reasonable and try them on this new set. You will then be able to debug your algorithm and your features until you get something that works. You know what it should look like on this simple set. You can then use these features on the old set and see how well they work. – SheldonCooper Mar 28 '12 at 5:15
show 6 more comments

1 Answer

up vote 0 down vote

You might try some tree based models, such as randomForest or GBM in R. Both models are good at picking up non-linear effects and interactions, and both also produce variable importance measures that will probably be useful in your analysis.

GBM in particular might be useful, as it fits each successive tree to the residuals of the model. In this way, after the model captures the effects of geometric dimensions, it will explore how the various treatments might be used to explain the "leftover" (or residual) variance. On the other hand, random forests require very little tuning and are harder to screw up than GBM models.

I would make sure each treatment is its set of variables, e.g. total time in fire, min/mean/median/max/cumulative bending and pulling pressure, etc. Particularly in GBM models, more variables are better, so be thorough!

How are you measuring how "good" your models are? Are you cross-validating them?

share|improve this answer
Thank you Zach. I've tried random forests (though not in R), and it doesn't work that great. In fact, none of the tree-based methods have worked that great. I haven't tried GBM though. I'm not sure what you mean by "in GBM models, more variables are better". Does that mean that overfitting doesn't happen in GBM? I'm using 10 runs of 10-fold CV to validate my models. I'm using AUC and also partial AUC as my measure of performance. I don't really think a particular algorithm is the answer. – user765195 Mar 28 '12 at 3:39
I think you should give GBM a shot. GBMs will eventually overfit, depending on how many trees you use. The GBM algorithm in R has built-in cross validation to tune the number of trees to prevent this. Try using the bernoulli and adaboost distributions. If no algorithm is working well, you need to re-examine your research question and see if there's any important variables you are missing. Did you make sure your random forest was a classification, rather than a regression forest? – Zach Mar 28 '12 at 15:14
Sure, I'll give it a shot. But I think a more fundamental approach would be to somehow amplify small differences between the treatments so that one can use them more effectively in the classification scheme. – user765195 Mar 30 '12 at 17:25
@user765195: Well, you could code the different treatments as a categorical variable (e.g. treatment 1 is 10 minutes in fire, treatment 2 is 20 minutes in fire, treatment 3 is 10 lbs of pulling pressure, etc.). If there are significant differences among treatments, this would let your model pick them up. However, if most of the failures can be explained by geometry rather than treatment, this is going to be very hard. Could you post some sample data? – Zach Mar 30 '12 at 18:04
Hi Zach. The treatments are a lot more complicated than what you describe. They're combinations of actions performed for various lengths of time, and each treatment was the same for the a group, but different groups went through different treatments. The problem is that most of the failures can't be explained by geometry (some can). The goal is to add the treatment to the classification features somehow to make the classification more accurate. – user765195 Mar 31 '12 at 3:05
show 3 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.