Questions tagged [data-mining]

Data mining uses methods from artificial intelligence in a database context to discover previously unknown patterns. As such, the methods are usually unsupervised. It is closely related but not identical to machine learning. Key tasks of data-mining are cluster analysis, outlier detection and mining of association rules.

Filter by
Sorted by
Tagged with
2
votes
1answer
1k views

How to convert binary category feature to numeric feature, like 'Gender'?

In a machine learning or data mining problem, suppose I have a original feature named "Gender", and now I want to convert this feature to numeric feature, there are two ways to do that, but I realy do ...
2
votes
1answer
194 views

How can I rate the question according to the difficulty?

I have a set of problems to analyze from a programming contest. My work is to rate the question according to its difficulty (easy,medium,hard) by analyzing the ...
10
votes
1answer
16k views

K-means: How many iterations in practical situations?

I don't have industry experience in data mining or big data so would love to hear you sharing some experience. Do people actually run k-means, PAM, CLARA, etc. on a really big dataset? Or they just ...
15
votes
1answer
10k views

Relationship between Hessian Matrix and Covariance Matrix

While I am studying Maximum Likelihood Estimation, to do inference in Maximum Likelihood Estimaion, we need to know the variance. To find out the variance, I need to know the Cramer's Rao Lower Bound, ...
0
votes
2answers
462 views

How can I map data to lower dimension?

I am trying to learn data in higher space into lower space. To have a clue, I'd like to know how to transform the data in the image below into a lower dimension preserving the structure. Hope to hear ...
11
votes
1answer
784 views

Variational Inference in plain english

Having watched videos on youtube, I feel like I can't really define what variational inference is. I can follow procedures while I am watching the video lectures about it. But hard to define what ...
3
votes
1answer
14k views

Stopping condition of K-means

I know that K-means algorithm stops when the cluster assignment does not change or just changes a little. Apart from that, and defining the maximum number of iterations, is there any other stopping ...
2
votes
1answer
1k views

Why do not we use Logistic regression for large feature datasets?

I have been asked this question in interviews. But I could not figure out the logic behind that.
0
votes
1answer
1k views

Should I use dummy variables or just assign numerical values to categorical predictors in regression / PCA?

When wanting to apply PCA or Linear Regression on some data set, I understand that the explanatory variables (features) should have numerical values. My current situation is that out of 79, I have 43 ...
1
vote
0answers
24 views

How does Naive Bayes differ on these two sets?

This is likely a simple question (since it is from an introductory course) yet I'm getting myself in a muddle here: Suppose I used the Naive Bayes classifier on 5000 instances (each instance only has ...
2
votes
1answer
93 views

Determine Value of Item in Bundle

Let's say I construct a series of bundles of different sizes, containing different items, like: ...
3
votes
1answer
573 views

One-class KNN for Quality Control

I've come across this paper https://uta-ir.tdl.org/uta-ir/bitstream/handle/10106/1827/Sukchotrat_uta_2502D_10083.pdf?sequence=1&isAllowed=y] where it is described a k-Nearest Neighbors Data ...
2
votes
0answers
115 views

Can Resampling be used for estimation and goodness of fit test?

I am trying to compare my data with empirical distributions. But I don't have enough data to cut them to estimation data and validation data. I am trying a resampling approach and would like to see if ...
1
vote
0answers
113 views

What are sufficiency and parameterization invariance in the context of Maximum Likelihood estimation?

While looking at properties of MLE, the articel mentions 4 things below: sufficiency (complete information about the parameter of interest contained in its MLE estimator) consistency efficiency ...
0
votes
1answer
654 views

What hypothesis should I have for k-means clustering?

I am wondering if we can just run k-means clustering without any hypothesis to find the optimal clusters or finding the optimal clusters are the hypothesis for K-means clustering? What hypothesis ...
1
vote
0answers
261 views

Supervised cosine similarity

Suppose we have some samples, each sample is with two vectors and the corresponding label. That is, it looks like ($\mathbf{u}_i, \mathbf{v}_i, y_i$), where $y_i \in \{0, 1\}$ We can calculate the ...
0
votes
1answer
579 views

correlationAtrributeEval Weka [closed]

I am new In data mining and Weka. I am working on "Ta_Feng" data set and my Intention is to apply Pearosn's Correlation coefficient to calculate correlation between User and Item attribute in my case ...
0
votes
0answers
85 views

how to explain z score?

I am trying to understand the below result set. F20 variable binned into two groups 0 and 1-722. The question I am posing here is for bin 0. I have responders variable as population1 and non-...
3
votes
0answers
59 views

Datamining: Why use algorithms for large datasets instead of using predictions based on a sample?

A growing amount of cluster algorithms have been developed for using large datasets such as CURE or BIRCH or filtering methods for common k-means. What is the advantage of using such an algorithm ...
0
votes
0answers
40 views

How can I compute the expected value of the mean of a subset of one variable given the mean of a subset of another variable in a correlated dataset?

I have a dataset with three variables: "Category", "X", and "Y". "Category" can be one of 5 different values: say A, B, C, D, E. I know that X and Y are correlated (say their correlation coefficient ...
2
votes
2answers
681 views

Lower bound for Adjusted Rand Index?

From the original paper, it's not clear whether the adjusted rand index has a lower bound. Does it? If so, what partition yields the bound? If now, how can I construct partitions with arbitrary low ...
5
votes
2answers
1k views

Logistic Regression-Linear Features

I was reading this post on quora https://www.quora.com/What-are-the-advantages-of-different-classification-algorithms It states here that :-"Logistic regression is a pretty well-behaved ...
0
votes
1answer
456 views

SVM gamma parameter influence : 2moons data set

I'm working on the two-moons data set, in which we can achieve a perfect discrimination, the classes are separable. I'm using the RBF kernel, and as a consequence, i have to tune the gamma parameter. ...
2
votes
0answers
63 views

Looking for patterns in date time data [closed]

I am collecting a large amount of date/time statistics in a mysql database about random people connecting to my computer. Time is represented as oob_time_sec and is converted from_unix(oob_time_sec) ...
3
votes
2answers
2k views

Metric to measure consistency (or randomness) of a group of time-series data

I have hourly time series consumption data of four customers namely (F238, F262, F267, F264) for a complete month, plotted as Each line represents the consumption ...
3
votes
0answers
3k views

Chi square test for decision tree - formula meaning [closed]

I can understand higher the value of chi square more significant is the split, but cannot understand the formula used. Please explain intuitively.
3
votes
1answer
439 views

Need explanation for a form of the Chi-Squared test that differs from the standard definition

I'm doing association analysis and I'm using a Chi-Squared test on a 2x2 contingency table. The usual definition of the Chi-Squared test is given e.g. here and here, the term in the denominator is ...
0
votes
1answer
752 views

difference between IEEE International Conference on Big Data and IEEE ICDM (international conference on data mining)

I'm not sure if this is the right place to ask this question, but I didn't know where to ask. Recently, I went to the IEEE 2016 International Conference on Big Data in Washington DC, but was ...
1
vote
1answer
4k views

K modes clustering : how to choose the number of clusters?

I am looking for a proper method to choose the number of clusters for K modes.  I tried to find the optimal number of clusters by maximizing the average silhouette width though.  In k-modes,  the ...
0
votes
1answer
3k views

How can one test a neural network against new data in MatLab? [closed]

I am new with MatLab and NN. I used the Neural Network Time Series Tool (https://nl.mathworks.com/help/nnet/gs/neural-network-time-series-prediction-and-modeling.html) in order to a make prediction ...
1
vote
0answers
255 views

Instance Selection from a dataset

I am a begineer and just trying to learn more about this technique. I am trying to do Instance Selection from a dataset containing 15000 instances. I want to select 5000 instances out of those 15000 ...
0
votes
1answer
42 views

How to sample data to classify sales performance data?

I have data about sales in a year, call and appointment records, and background of salesmen. I want to apply machine learning and data mining to predict which kind of person would bring highest sales ...
2
votes
0answers
237 views

Why imbalance in the data set is an issue in data mining and machine learning? [duplicate]

I've analyzed a data set from a credit card company before and it has the famous "unbalanced classes" problem like all the other credit card companies, i.e., in the data set (the information of users ...
1
vote
1answer
169 views

Are there alternatives to MCR-ALS for determining pure response profiles of a mixture?

MCR-ALS says it is a tool for "the recovery of the pure response profiles of the chemical constituents or species of an unresolved mixture when no prior information is available about the nature and ...
1
vote
0answers
715 views

Factor analysis and logistic regression

I have done factor analysis on 18 variables and got a 5 factor solution. Denote: $X=(X_{1},X_{2},.....,X_{18})$ as the variables and $F=(F_{1},F_{2},..,F_{5})$ as the factors. Thus we have a loading ...
0
votes
1answer
1k views

Predictive maintenance data set [closed]

Currently I am working on my PhD thesis which is focused on predictive maintenance and failure prediction in industrial manufacturing processes. To test my hypothesis I would like to use real-world ...
1
vote
0answers
158 views

How to Optimize rows of a term document matrix

Suppose a $n × m$ term document matrix T. I'm looking for algorithms (preferably in R) able to automaticaly reduce row number, so only relvant terms are part of T. For example it is easy to find ...
-1
votes
1answer
848 views

How do we calculate distance and centroid for vectors that contain negative values ?

I would like to know if its possible to measure distance among vectors that contain positive and negative values such in this example : ...
2
votes
0answers
44 views

Leads and Ideas on Learning from disconnected data

Problem of learning patterns from disconnected data. I have two independent discrete variables, D1 and D2, with information similar to tables in the images. I do NOT have the broken down data for A+...
2
votes
3answers
146 views

Who is credited with inventing the term “high dimensional data”?

Title says it all, who is credited with inventing the term "high dimensional data"?
1
vote
2answers
481 views

Fat Tailed risks: do they get fatter when we linearize non-linear systems?

I'm concerned about the risks that might arise when one decide to model a complex process into a linear or non-linear polynomial or maybe a state space model. In the beginning of the century, Maxwell ...
0
votes
1answer
166 views

overall AUC of two different AUC's

I have a dataset where I divide it (based on nodule longest diameter) into two subsets of cancer features. I have got the prediction accuracy for each subset and the AUC's. I would like to combine ...
0
votes
1answer
58 views

What is or how to choose,calculate the optimal query point for Logistic regression with RBF kernel?

I have implemented a Logistic Regression with RBF kernel. I have trained the algorithm using a training set and a query point after the weights have been calculated i applied this weights to the test ...
1
vote
0answers
16 views

Wondering which percentage below it we can say the classes are suffering from under-representation and need a resampling approach [closed]

I am sure here there is someone who has experienced something like this... I have a dataset with 4 classes, one of it is 3% representation and I considered it as under-represented, so I did some ...
11
votes
2answers
27k views

Is Random Forest a good option for unbalanced data Classification? [closed]

Despite the resembling and other increasing data variability approaches, can the random forest "as an algorithm" be considered a good option for the unbalanced data classification?
1
vote
2answers
174 views

The Best Approach for the Classification of the imbalanced classes

I have data that I am going to classify to 3 classes, but one class has a few samples less than 5% of the total samples. What is the best approach to classify these imbalanced classes? I mean is ...
1
vote
0answers
86 views

Network/Graph Analysis Beginner Resources

I am a beginner to Network/Graph Analysis. I have data in the following format: Customer | CustomerSpend Anna $10 Bob $30 Charlie $50 ... .....
6
votes
1answer
2k views

How does FaceNet (Google's facerecognition) handles a new image?

I am currently researching in the facerecognition field. And I can not understand how the facenet algorithm handels a new image They use an euclidean space for image representation. Which means that ...
1
vote
1answer
111 views

Should I include all data for Apriori algorithm?

For Transaction ID 3, It contains beer only and beer does not exists in other transaction. Do I need to include transaction Id 3 when calculating the association rule by using the Apriori algorithm?
0
votes
0answers
59 views

Data mining algorithm for anomaly finding in a restaurant business

Consider running a restaurant. During each shift there are approx. 30 people working on different positions (Managers, Waiters, Cooks, etc). Day in week, daily or nightly shift,.. At the end of each ...

1
4 5
6
7 8
24