Questions tagged [data-mining]

Data mining uses methods from artificial intelligence in a database context to discover previously unknown patterns. As such, the methods are usually unsupervised. It is closely related but not identical to machine learning. Key tasks of data-mining are cluster analysis, outlier detection and mining of association rules.

Filter by
Sorted by
Tagged with
1
vote
1answer
2k views

k-means with binary variables

I have converted all of my features to binary variables. now I have 21 features in my data set. I am trying to cluster them with k-means. I used Hamming distance in order to measure the distance ...
1
vote
0answers
23 views

Learning conjunctions of Horn clauses

I am studying the algorithm presented in this paper for "Learning conjunctions of Horn clauses". The algorithm uses equivalence queries and membership queries to produce a formula that is logically ...
4
votes
1answer
3k views

What is the range of information gain ratio?

I am wondering what the value range of information gain ratio is. I guess it is [0,1] but am not too sure about it.
2
votes
0answers
271 views

Lift measure for a frequent itemset

I found it's possible to add the lift measure to the quality measures of a frequent itemset in R returned by the Eclat algorithm: ...
1
vote
0answers
2k views

How to perform CAP curve analysis in R

I'm trying to plot in R a cumulative accuracy profile (CAP) Curve. Additional, after I will build the curve, how am I calculate the AR.
1
vote
1answer
33 views

Creating a predictive model based on past customer data.

I have a historical data set for customers for a particular company. Target class being Yes/NO (Would a customer subscribe to a new product.) I need to develop a classification approach to predict ...
1
vote
1answer
746 views

Decision tree: where and how to split an attribute on numerical dataset?

I am new to data mining and am manually implementing decision tree classification on a dataset with all continues values. A very small sample dataset of 4 attributes (columns) would be like this: <...
1
vote
0answers
25 views

Are there any way to handle the derived response variable?

I have a response variable problem, there is a response variable "Y", but what I want to do is make $$ \#(y_i>\hat{y_i})/n \rightarrow 0.5$$ (If I have n entities) and simultaneously, $$ \hat{...
-2
votes
2answers
89 views

how to determine the certain number of clusters your data contain? [duplicate]

I was wondering if I can find a certain way that show me the number of clusters my data has by its nature.I don't want to find the optimal number of clusters for clustering, I want to know the number ...
0
votes
1answer
357 views

Why do linear (OLS) models often outperform other predictive models?

Recently, I've tested many prediction models on real datasets (collected by myself). But most datasets that I collected perform best using an OLS MLR hypothesis compared to many other machine learning/...
0
votes
2answers
192 views

How to compare the distributions of variables within clusters?

I used K-means to cluster 15k data points composed of 5 quantitative features scaled between 0 and 1. I would like to compare the distributions of the features within each cluster, and also compare ...
0
votes
1answer
67 views

what is best data mining technique and good accuracy in time series? [closed]

i beginner in data mining in my company and i want to do some prediction in our stock , the stock contain products that have nay and sale every time , i want to make some prediction about best sales ...
2
votes
0answers
456 views

Euclidean vs Manhattan distance behaviour in high dimension - curse of dimensionality

I have compared different distance functions by computing the average tf/idf distance between documents. My results show a range between $10-15$ for the Manhattan and a range between $1-1.5$ for the ...
3
votes
2answers
3k views

Why is DBSCAN deterministic?

Recently, I am working on DBSCAN algorithm, the original paper is M. Ester, H. Kriegel, J. Sander, and X. Xu. A density-based algorithm for discovering clusters in large spatial databases with noise....
7
votes
1answer
2k views

Bayesian network vs. association rules

Apriori algorithm finds some implication rules. Similar results are provided by Bayesian networks. What is the essential difference? What are the specific advantages/disadvantages? Edit: The ...
3
votes
2answers
5k views

Finding hidden data pattern in big data set

I have a huge data set (4-5 million entries). The data is indexed by 10 values (v1, v2, ..., v10) = IDENTIFIER. The Identifier is not unique, there are many ...
3
votes
0answers
3k views

Intuitive meaning behind support, confidence, lift and conviction

I'm learning about association rules and came across the common interestingness measures support, confidence, lift and conviction. I'm interested in the intuition behind your decision-making process ...
2
votes
0answers
367 views

Clustering of high dimensional data

I am having a data set with 54 independent variables .Most of them are having zeros it resembles like sparse matrix .How to cluster this kind of data and is there any data pre processing like Box cox ...
-2
votes
1answer
191 views

Classification problem and Associative rule mining [closed]

Imagine, you are solving a multiclass classification problem with highly imbalanced class. The distribution of the classes is such that, you observed the majority class 99% of the times in the ...
0
votes
0answers
27 views

quick way to check all variables, both continuous and categorical, for systematic differences between two groups in R?

I have a dataset of about 3 million observations with around 1000 duplicate cases/rows (found simply by using the duplicated() function). I'm trying to figure out why these cases might have been ...
0
votes
1answer
59 views

What is the name for this type of model?

My boss wants me to evaluate a rules based model that a former employee designed. The problem is that I'm not sure what type of model it is or how to evaluate its performance. The model shows which ...
2
votes
2answers
2k views

Why are k-means and k-NN considered simple algorithms in machine learning?

We all know the k-means clustering algorithm and the k-nearest neighbors algorithm: the former is an unsupervised clustering method, and the latter is a supervised learning technique in machine ...
2
votes
1answer
653 views

Using Naïve Bayes to predict disease occurence

I have a dataset concerning patients with information about their diseases and symptoms. I want to estimate probability of $P(disease_i = TRUE|symptom_j = TRUE)$. My intuition is that I should use a ...
18
votes
7answers
7k views

Biased Data in Machine Learning

I am working on a Machine Learning project with data that is already (heavily) biased by data selection. Let's assume you have a set of hard coded rules. How do you build a machine learning model to ...
1
vote
2answers
214 views

Android Recommendation App Algorithm

I have this project proposal entitled "Android Based Program Recommendation App". (This application is for those college students who wants to shift to other programs). The app will find a program ...
3
votes
2answers
957 views

Feature Hashing in Machine learning

I've been interested in feature hashing in machine learning (the hashing trick) I can't seem to find worked examples of it being used in action for an actual algorithm. Can anyone provide an example ...
0
votes
0answers
22 views

How can we inject external background knowledge into data stream mining algorithm?

If we want to include the context as a background knowledge, how can we do this when using a data stream mining algorithm. What are the possible forms of the background knowledge and how can they be ...
2
votes
1answer
542 views

Predicting column use in a table

I have a set of tables $\mathcal{T} = \{T_1, ..., T_n\}$, where each $T_i$ is a collection of named columns $\{c_0 .. c_{j_{i}}\}$. In addition, I have a large sequence of observations $\mathcal{D}$ ...
2
votes
1answer
294 views

How to check cross validation scores for market basket analysis?

If I have a large set of transactions where in each I buy a set of goods and I want to do market basket analysis using either A-priori or FP Growth or any other data mining method, you typically get ...
1
vote
0answers
16 views

How to check for stratified points?

I originally posted this question on MathExchange but did not receive any answer. I have discovered now this site dedicated to data mining, and think it is a more appropriate site to ask this. In $\...
0
votes
0answers
70 views

Domain knowledge or data dredging?

I have a binary classification problem where I'm required to classify transactions as anomalous or normal (1 or 0 respectively), with anomalies being the rarer instance. With what I know to be true ...
1
vote
0answers
75 views

Data Science: Using Inferential Statistics to label train dataset

Lack of High Schools in remote areas is a problem for students in developing country. Students in some locations are better than that in other. So, I have to find those locations. Now, the main ...
1
vote
2answers
6k views

Machine Learning model on dataset with mainly zeros

I have a text field dataset. Each observation counts the number of appearances of that particular word, and the columns (variables) are most frequently appeared words. Within each column, zeros ...
0
votes
2answers
975 views

machine learning model true negative rate is too low while true positive rate is too high

I am using the tm package and h2o package to do text mining using neural network. Here I have a data frame of 100 most frequent words in the text. These variables only have values of non-negative ...
0
votes
2answers
487 views

Normalization ,standardization, or do nothing

I have medical data with max value 500 along with values like age and binary values for sex (0 or 1). I will use clustering to find the number of clusters. Which is the best approach among three....
3
votes
1answer
139 views

Data Science Interview Question - Build a model to Predict the Class of the Output [closed]

I recently attended an Interview and I was asked this question by the Interviewer. Question : There are 100 books, 90 of the books belong to category A and 10 of the books belong to category B. ...
1
vote
1answer
2k views

guide for text classification using weka

I have a set of 2000 small texts (each less than 500 words) that I manually categorized. All the texts are in the same main subject, and I want to separate them into distinct groups based on their ...
2
votes
1answer
3k views

why don't we test error on autoencoders?

we are not suppose to use test set when we r fitting the model to our data. but I noticed in this blog https://blog.keras.io/building-autoencoders-in-keras.html which is very established it uses <...
0
votes
0answers
21 views

Algorithm to handle positive rating if there is huge difference between the number of reviews

We have two products P1 and P2. P1 has total 1000 number of reviews and p2 has 200 reviews. Positive rating of P1 is 85% (85% users are saying it's a good product) Positive rating of P2 is 90% (90% ...
1
vote
1answer
239 views

Recommendations for textbooks covering current data mining/machine learning techniques for fraud detection?

I work in the health insurance field, but a general treatment of fraud detection methodologies would still be helpful. So far I've discovered brief articles outlining particular techniques used in ...
0
votes
1answer
1k views

Designing a simple tag-based recommender system

I want to design a system for product recommendation, where I have the following: ~1000 products that are tagged with 1 to ~10 descriptive tags by professional product specialists. ~200000 users, ...
1
vote
4answers
248 views

How to treat incomplete variable values

I'm trying to analyze some fairly sparse data on a recurrent medical symptom, and I don't know what to do with two entries where my data is incomplete. My overall goal is a bit vague: it's to find a ...
1
vote
0answers
105 views

Supervised Multi-Classification with small dataset

I've got 700 data rows with about 6 features and there are 300 different class labels. Most of the classes have got only 2 data rows (ca. 250 classes). I know there are too many classes for this data ...
4
votes
2answers
1k views

Are XGBoost probability outputs based on the number of examples in a terminal leaf

I am trying to replace a c4.5 tree that someone else implemented with a boosted tree (XGBoost). The data is extremely skewed and the company wants the new model to output similar distributions. c4.5 ...
0
votes
0answers
582 views

a question about splitting data into TRAIN/TEST set

Can we "trust" about this method when we have small data set ? Is there a minimum size required to partition the data ? e.g. N=200 ?
3
votes
0answers
1k views

Algorithm to detect time series anomalies (outliers) (using Apache Spark)

I am currently new to machine learning and I will be working on a project that involves using a Machine Learning library to detect and alert about possible anomalies. I will be using Apache Spark and ...
0
votes
1answer
414 views

Extracting Part of Speech (Source and Destinations) using text mining/NLP?

I need to extract the source and destination terms from the text documents using text mining/NLP/Information Retrieval ? ex : i am travelling from New York to London. i am heading towards ...
1
vote
1answer
67 views

Is it possible to forget data when computing sufficient statistics on a stream?

My understanding is when we are computing sufficient statistics on a stream, when a new instance arrives, the value of the new instance is added to the already computed sufficient stats. so there is ...
0
votes
0answers
22 views

build a model capturing related features for multiple events

There is a data set, which has about 20000 data points. Each data point has about 100 features. At the same time, there is an instance set, including 60 different instances. Right now, there is a ...

1 2 3
4
5
24