Questions tagged [data-mining]
Data mining uses methods from artificial intelligence in a database context to discover previously unknown patterns. As such, the methods are usually unsupervised. It is closely related but not identical to machine learning. Key tasks of data-mining are cluster analysis, outlier detection and mining of association rules.
1,184
questions
2
votes
1
answer
2k
views
Regression with more features than samples
I am new to the subject and I encountered this question in my text book:
Suppose that the number of features is greater than the number of samples.
e.g:
$𝑦_1 = b_1𝑋_{11} + b_2𝑋_{12} + b_3𝑋_{13}$
...
- 21
1
vote
0
answers
224
views
Comparing clustering stability over period of time [duplicate]
Suppose I am applying K-means clustering on two datasets generated by same process. How to compare clustering stability over a period of time? Let's say I have applied clustering in 2016 on a data set,...
- 323
0
votes
0
answers
267
views
scores and loadings in princomp (PCA)
I applied PCA on the dataset, but I am confused
what should I use to build the model, the components in scores or in the loadings?
@ usεr11852
...
- 11
1
vote
0
answers
120
views
use test data set after 10 Cross-validation
I applied 10 Cross-validation but I am a bit confused, I am not sure what is a correct way.
1- Should I apply 10 Cross-validation on all dataset, divide it into 10 folds and sum all the 10 matrices ...
- 11
0
votes
0
answers
584
views
How to calculate probability of Bayes network?
Hi I'm new into Bayes network
I'm trying to calculate P(F&G) but don't have the slightest clue how because everything i find online doesn't have a network that is this complex.
I will really ...
1
vote
1
answer
898
views
AdaBoost algorithm question
In the boosting algorithm,AdaBoost ,those observations which were misclassified by the classifier in the (m-1)th step have their weights increased in the mth step, and those which were correctly ...
- 552
0
votes
1
answer
68
views
Is this an unsupervised pattern recognition problem?
Consider a feature set containing values of operating condition of an automation unit such as pressure of the valve, temperature, fuel consumption. I want to discover unknown relationship between ...
- 777
1
vote
1
answer
2k
views
K nearest neighbours model complexity
My question is about the 1-nearest neighbor classifier and is about a statement made in the excellent book The Elements of Statistical Learning, by Hastie, Tibshirani and Friedman. The statement is
"...
- 552
0
votes
1
answer
1k
views
Should text pre-processing come before or after POS-tagging?
I am currently working on an assignment to produce a sentiment classifier for Twitter. One of the initial, and most important, stages in NLP classification is text pre-processing. For Twitter, this ...
- 252
2
votes
1
answer
723
views
Technique to find spurious correlations among huge number of time series datasets?
I came across this tongue-in-cheek website that lists lots of spurious correlations. It's not lost on me that the author's main point is to discourage the brute-force-search of correlations, but his ...
- 6,197
0
votes
0
answers
214
views
What is information gain ratio?
With respect to data mining, what is information gain ratio?
I'm a complete beginner to data analytics and mining, so please explain at a low level of understanding.
1
vote
1
answer
417
views
AdaBoostM1 reweighting examples
It is said Adaboost increases the weights of the misclassified examples. But if I look at step 2(b) , err is between 0 and 1. Then at step 2(c) , If err=1 , alpha = log(0)=-inf and if err=0, alpha = ...
- 552
2
votes
1
answer
193
views
Which is the state of the art regarding methods for Association rule learning?
I have a set of data about which I would like as a first step, discover some interesting or frequent relationships. For this I know that there are several algorithms, like association or sequence ...
- 231
0
votes
1
answer
57
views
How can I cluster the following data set to find out the time when ozone and pm level are high in different stations using R?
My data set contains around 630,000 rows and the data set looks like this:
date site code latitude longitude rollingo3 rollingpm2.5
1 ...
- 11
7
votes
2
answers
11k
views
How to extract specific information from text using Machine learning?
Suppose I have a text like below which usually have 2/3 sentences and 100-200 characters.
Johny bought milk of 50 dollars from walmart. Now he has left only 20 dollars.
I want to extract
Person ...
- 73
2
votes
1
answer
1k
views
What attributes to apply laplace smoothing in naive bayes classifier?
I am reading naive Bayes classifier from the book "Data mining practical machine learning tools and techniques". The example of naive Bayes is given using the below dataset.
As (Outlook=Overcast | ...
- 123
11
votes
3
answers
10k
views
Is it true that K-Means has an assumption "each cluster has a roughly equal number of observations"?
A lecturer claimed in a recent class that "K-means assumes that each cluster includes a roughly equal number of observations."
However, when I searched online, there is conflicting information ...
- 263
3
votes
1
answer
465
views
Confusion in derivation of bias variance decomposition
This question is regarding derivation of bias-variance decomposition as answered (which was accepted) in another thread. I am repeating the steps for this question:
\begin{align} \newcommand{\var}{{\...
- 175
1
vote
1
answer
172
views
Verification goal of KDD? Cases where it apply
I'm having difficulties in finding examples of a Verification Data-mining/KDD approach. Let me explain what I mean by that.
In the notorious article From Data Mining to KDD by Fayyad et al the ...
- 113
2
votes
1
answer
553
views
How to automatically find patterns in (time series) datasets?
I would like to find methods, algorithms, procedures or even concrete solutions of my problem.
The Problem:
Imagine a dataset of the historic performance and general behavior of 1000 athletes. We ...
- 123
0
votes
1
answer
23
views
significance of spliting dataset training and testing data
why is it necessary to divide the data set into training and testing set while running a particular model? what is the significance of such partition? why is it not possible to get a model for test ...
- 341
2
votes
1
answer
3k
views
k-means with binary variables
I have converted all of my features to binary variables. now I have 21 features in my data set. I am trying to cluster them with k-means. I used Hamming distance in order to measure the distance ...
- 275
1
vote
0
answers
52
views
Learning conjunctions of Horn clauses
I am studying the algorithm presented in this paper for "Learning conjunctions of Horn clauses".
The algorithm uses equivalence queries and membership queries to produce a formula that is logically ...
- 111
4
votes
1
answer
4k
views
What is the range of information gain ratio?
I am wondering what the value range of information gain ratio is. I guess it is [0,1] but am not too sure about it.
- 41
2
votes
0
answers
329
views
Lift measure for a frequent itemset
I found it's possible to add the lift measure to the quality measures of a frequent itemset in R returned by the Eclat algorithm:
...
- 161
1
vote
0
answers
2k
views
How to perform CAP curve analysis in R
I'm trying to plot in R a cumulative accuracy profile (CAP) Curve.
Additional, after I will build the curve, how am I calculate the AR.
- 61
1
vote
1
answer
39
views
Creating a predictive model based on past customer data.
I have a historical data set for customers for a particular company. Target class being Yes/NO (Would a customer subscribe to a new product.)
I need to develop a classification approach to predict ...
- 9
1
vote
2
answers
1k
views
Decision tree: where and how to split an attribute on numerical dataset?
I am new to data mining and am manually implementing decision tree classification on a dataset with all continues values. A very small sample dataset of 4 attributes (columns) would be like this:
<...
- 11
1
vote
0
answers
31
views
Are there any way to handle the derived response variable?
I have a response variable problem, there is a response variable "Y", but what I want to do is make
$$ \#(y_i>\hat{y_i})/n \rightarrow 0.5$$ (If I have n entities)
and simultaneously, $$ \hat{...
- 13
-2
votes
2
answers
165
views
how to determine the certain number of clusters your data contain? [duplicate]
I was wondering if I can find a certain way that show me the number of clusters my data has by its nature.I don't want to find the optimal number of clusters for clustering, I want to know the number ...
- 121
0
votes
1
answer
519
views
Why do linear (OLS) models often outperform other predictive models?
Recently, I've tested many prediction models on real datasets (collected by myself). But most datasets that I collected perform best using an OLS MLR hypothesis compared to many other machine learning/...
- 13
0
votes
2
answers
552
views
How to compare the distributions of variables within clusters?
I used K-means to cluster 15k data points composed of 5 quantitative features scaled between 0 and 1.
I would like to compare the distributions of the features within each cluster, and also compare ...
- 293
0
votes
1
answer
79
views
what is best data mining technique and good accuracy in time series? [closed]
i beginner in data mining in my company and i want to do some prediction in our stock , the stock contain products that have nay and sale every time ,
i want to make some prediction about best sales ...
2
votes
0
answers
546
views
Euclidean vs Manhattan distance behaviour in high dimension - curse of dimensionality
I have compared different distance functions by computing the average tf/idf distance between documents. My results show a range between $10-15$ for the Manhattan and a range between $1-1.5$ for the ...
3
votes
2
answers
4k
views
Why is DBSCAN deterministic?
Recently, I am working on DBSCAN algorithm, the original paper is
M. Ester, H. Kriegel, J. Sander, and X. Xu. A density-based
algorithm for discovering clusters in large spatial databases with
noise.
...
- 131
7
votes
1
answer
2k
views
Bayesian network vs. association rules
Apriori algorithm finds some implication rules.
Similar results are provided by Bayesian networks.
What is the essential difference? What are the specific advantages/disadvantages?
Edit:
The ...
- 2,638
3
votes
2
answers
5k
views
Finding hidden data pattern in big data set
I have a huge data set (4-5 million entries). The data is indexed by 10 values (v1, v2, ..., v10) = IDENTIFIER.
The Identifier is not unique, there are many ...
- 33
4
votes
0
answers
4k
views
Intuitive meaning behind support, confidence, lift and conviction
I'm learning about association rules and came across the common interestingness measures support, confidence, lift and conviction.
I'm interested in the intuition behind your decision-making process ...
- 161
2
votes
0
answers
411
views
Clustering of high dimensional data
I am having a data set with 54 independent variables .Most of them are having zeros it resembles like sparse matrix .How to cluster this kind of data and is there any data pre processing like Box cox ...
- 131
-2
votes
1
answer
285
views
Classification problem and Associative rule mining [closed]
Imagine, you are solving a multiclass classification problem with highly imbalanced class. The distribution of the classes is such that, you observed the majority class 99% of the times in the ...
0
votes
0
answers
30
views
quick way to check all variables, both continuous and categorical, for systematic differences between two groups in R?
I have a dataset of about 3 million observations with around 1000 duplicate cases/rows (found simply by using the duplicated() function). I'm trying to figure out why these cases might have been ...
- 113
0
votes
1
answer
64
views
What is the name for this type of model?
My boss wants me to evaluate a rules based model that a former employee designed. The problem is that I'm not sure what type of model it is or how to evaluate its performance.
The model shows which ...
- 133
2
votes
2
answers
2k
views
Why are k-means and k-NN considered simple algorithms in machine learning?
We all know the k-means clustering algorithm and the k-nearest neighbors algorithm: the former is an unsupervised clustering method, and the latter is a supervised learning technique in machine ...
- 712
1
vote
1
answer
730
views
Using Naïve Bayes to predict disease occurence
I have a dataset concerning patients with information about their diseases and symptoms. I want to estimate probability of $P(disease_i = TRUE|symptom_j = TRUE)$. My intuition is that I should use a ...
- 11
19
votes
7
answers
7k
views
Biased Data in Machine Learning
I am working on a Machine Learning project with data that is already (heavily) biased by data selection.
Let's assume you have a set of hard coded rules.
How do you build a machine learning model to ...
- 1,962
1
vote
2
answers
234
views
Android Recommendation App Algorithm [closed]
I have this project proposal entitled "Android Based Program Recommendation App". (This application is for those college students who wants to shift to other programs).
The app will find a program ...
- 11
3
votes
2
answers
1k
views
Feature Hashing in Machine learning
I've been interested in feature hashing in machine learning (the hashing trick) I can't seem to find worked examples of it being used in action for an actual algorithm. Can anyone provide an example ...
- 551
0
votes
0
answers
26
views
How can we inject external background knowledge into data stream mining algorithm?
If we want to include the context as a background knowledge, how can we do this when using a data stream mining algorithm. What are the possible forms of the background knowledge and how can they be ...
- 1
2
votes
1
answer
556
views
Predicting column use in a table
I have a set of tables $\mathcal{T} = \{T_1, ..., T_n\}$, where each $T_i$ is a collection of named columns $\{c_0 .. c_{j_{i}}\}$. In addition, I have a large sequence of observations $\mathcal{D}$ ...
- 101
2
votes
1
answer
446
views
How to check cross validation scores for market basket analysis?
If I have a large set of transactions where in each I buy a set of goods and I want to do market basket analysis using either A-priori or FP Growth or any other data mining method, you typically get ...
