I am working on a project whose aim is to analyze the relationship between machine elements and their price.
My data consists of thousands of machine elements, their price, as well as technical and non-technical attributes. Most of the data is categorical
The technical data contains information such as weight, dimensions, material. Each element (identified by unique code) has distinct technical attributes.
The non-technical data contain information such as supplier name, contract type, INCOTERMS, first day of validity of the contract, last day of validity, minimum order quantity, etc. In some cases the prices depend on the order quantity.
One of the problems I am facing is that often each element has a different price depending on the financial data, i.e. element X will cost 100 dollars if Incoterms is A, 200 dollars if Incoterms is B, etc. In other words, there are rows that contain price information on the same element, but one of the columns has a different value and so the price is different.
In other cases the price is 100 if 50-100 elements are ordered, 80 if 100-200 elements are ordered, 60 if 200-500, etc.
I am planning to do some correlations as well as regression. I will probably also try data mining (using Rattle and R).
I need advice on how to treat the (rather) similar observations that each object has in case of correlation, regression and in general for data mining. Should I try to select a single observation per element or do the analysis for “all in”. I guess the last option won’t work for correlation at all.
Perhaps including a sample screenshot of the data with clarify my problem. So what is visible in the picture happens quite often: one object, different attributes, different price. I am wondering whether I should (try to) have only one row per object with only one set of attributes (we don’t need to discuss the criteria) or whether I should keep the multiple rows for the same object when running a correlation analysis or a regression.
Thank you in advance.