1
$\begingroup$

I am working with the diamonds data set from the tidyverse package in R.

library(tidyverse)
View(diamonds)

When I plot a histogram of the price variable with 300 bins there looks to be 3 groups. enter image description here

When I take the log of price these groups become much more vivid. There might even be 4 groups instead of 3! enter image description here

My question is, how do I make sure there are groups here? What are these groups? Lastly, if I do not have the data on what these groups actually are, let's say for our case the groups are continent from which the diamond was found.

Once I distinguish the groups (maybe through GMM) how do I incorporate it in a regression model, where I regress prices on carat?

Thank you kindly.

Edit** I am editing this comment due to Henry's comment. It's more of friendly rebuttal to the comment - try to explain the DV with features in the data set before trying to group the DV.

If we take a look at the iris data set (not knowing there are three different flowers) and regress Petal Width ~ Sepal Width we would conclude that Sepal Width is not a good predictor. enter image description here

However, if we originally run our GMM on the Petal Width we get back 3 groups and can visually see that Sepal Width is a good predictor. enter image description here

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ There is a curious gap in prices between $\$1455$ and $\$1545$, but the idea of groups just from the charts may be slightly spurious. There is a more understandable marketing tendency for weights to cluster at $1$, $1.5$ or $2$ carats or just above rather than just below, but that is not enough to explain the price patterns. You have the data on that and on other factors that affect price such as cut and clarity and color so you can investigate further $\endgroup$
    – Henry
    Jan 1, 2020 at 4:47
  • $\begingroup$ Hi Henry, I am a bit confused on what you think my next steps should be. Price is my DV. Are you saying I should regress the price on other variables to see if I have a variable that explains the grouping? A concern I have is if on a new data set my dependent variables do not describe the grouping. Thanks $\endgroup$
    – Jordan Wrong
    Jan 1, 2020 at 4:55
  • $\begingroup$ What I am saying is that just looking at the distribution of price may be misleading. If you have potential explanatory variables likely to affect price, as you do here, then you should investigate them before concluding there are a particular number of groups $\endgroup$
    – Henry
    Jan 1, 2020 at 4:58
  • $\begingroup$ Henry I added to my original post. Thank you for taking the time $\endgroup$
    – Jordan Wrong
    Jan 1, 2020 at 7:20

1 Answer 1

Reset to default
1
$\begingroup$

Why are you grouping price at all? Grouping continuous variables always entails loss of information and should only be done when you have a strong substantive reason for doing so, which you clearly do not.

What are you trying to find out? If, for example, you want to see if price is related to continent, then I suggest doing quantile regression with price as the DV and continent as the IV. (Or whatever IVs you are interested in). Quantile regression makes no assumptions about the residuals and lets you answer more questions than OLS regression.

Also, groups on price are very different from a group like "which continent". Continent is, by its nature, a discrete variable, price is not. And your addition of the Iris data set isn't really relevant. We know, a priori, that there are different flowers. A rose is a rose and not a daffodil.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Sorry if I am being thick, I am just really trying to understand this. In the diamond case why can I not have 3 different coloured diamonds:red, yellow and blue? A blue diamond is a blue diamond not a yellow. How does knowing that a diamond is yellow (grouping) cause a loss in information? I would think that would increase information. Thank you for answer peter $\endgroup$
    – Jordan Wrong
    Jan 1, 2020 at 14:36
  • $\begingroup$ You CAN have different color diamonds. No problem. But why are you grouping PRICE? What I am saying is that grouping price (say, into low, medium and high) loses information. $\endgroup$
    – Peter Flom
    Jan 1, 2020 at 15:18
  • 1
    $\begingroup$ Thank you peter!!! It all clicked after I read that comment and ran a regression of price on carat with the grouping low, medium, high $\endgroup$
    – Jordan Wrong
    Jan 1, 2020 at 19:47
  • 1
    $\begingroup$ Peter I also found your article on quantile regression which was very helpful. For those who want to give it a read towardsdatascience.com/… $\endgroup$
    – Jordan Wrong
    Jan 1, 2020 at 22:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.