1
$\begingroup$

I am trying to predict auction prices from a self gathered data set.

I have titles available, and the price it was offered for. Now I noticed that whenever there is a 3 digit string of numbers in the title, there is about 48% that this is the actual price offered (I constrained to only gather data of the price between 100-800).

How can I use this information in a further model, where I will make features of other text attributes (where 1 will mean the word exists in a training sample, and 0 will mean it does not exist in a training sample)?

Is the best way to split the data up into where there exists a string in the title with numerical value between 100-800, versus where it does not exist, or is there a way to combine it in a single regression? i think what makes it different is that there is a 50% chance to be deterministic.

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
1
$\begingroup$

I think the most reasonable thing to do would be to include this as an additional feature. So in addition to your bag-of-words features you now have another feature which takes value 1 if there is a 3 digit string in the title and 0 otherwise.

This type of feature engineering can be really useful when working with textual data. It is very common for people to use features like "word starts with a capital letter", "word contains a hyphen", "looks like a url", etc. The type of features you employ will obviously depend a great deal on the task you're trying to perform.

That being said, one needs to be careful when doing this type of feature engineering. If you're looking at all your data to decide what features to include based on the relationships with the desired output, using this same data to assess the accuracy of your final model will result in biased estimates. This will be the case even if after deriving features you use a proper procedure like cross-validation to estimate metrics of interest (accuracy, precision, recall, etc).

Lastly, I think your statement there is a 50% chance to be deterministic is somewhat vacuous. For example if I have a fair coin, then the coin comes up heads 50% of the time. So I could claim there is a 50% chance the coin is deterministic (always comes up heads), but I haven't really said anything new. Maybe I misunderstood what you mean by 50% chance to be deterministic?

Improve this answer
$\endgroup$
3
  • $\begingroup$ What I meant was that this method doesn't predict 249 which is really close to 250 (the actual value), but there is around a 50% chance the error will be 0. I'm having doubts whether having it as a feature (where perhaps the estimates will just be used in 100% of the cases to make some prediction, even though the other 50% time it might be about some version number or something instead of the actual price mentioned) $\endgroup$
    – PascalVKooten
    Commented Nov 17, 2013 at 19:59
  • $\begingroup$ It seems more likely to separate the data perhaps. The actual goal at this moment is to predict the price the product was offered for (though the goal is arbitrary; it's just a fun thing). $\endgroup$
    – PascalVKooten
    Commented Nov 17, 2013 at 20:02
  • $\begingroup$ I like the first 1/2 of the answer. "looking at all your data to decide what features to include": if this approach were taken, then your point about cross-validation wouldn't apply, because this would not even be cross-validation. It would defeat the purpose of cross-validation. $\endgroup$
    – rolando2
    Commented Nov 23, 2014 at 13:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.