Data mining uses methods from artificial intelligence in a database context to discover previously unknown patterns. As such, the methods are usually unsupervised. It is closely related but not identical to machine learning. Key tasks of data-mining are cluster analysis, outlier detection and mining ...

learn more… | top users | synonyms

1
vote
0answers
19 views

Cross-validation for parameter tuning in data mining process (KDD)

In my project I want to compare different classification algorithms to solve a specific problem with a specific dataset. To do this, I divided the dataset in 2 parts. With the first (bigger) part I ...
0
votes
0answers
16 views

Are there differences between Delta TF-IDF and TF-IDF?

Are there differences between both algorithm or not ? i mean if i implement for ex Delta TF-IDF in a project instead of ...
0
votes
1answer
18 views

What are distinctive terms?

Here $n$ is the number of distinctive terms in document $d$. What is the meaning of distinctive? My guess is that it's terms that remain after filtering document from terms that aren't necessary, ...
1
vote
1answer
20 views

Choosing a k-value for Local Outlier Factor (LOF) detection analysis

I have a set of three-dimensional data, and I'm trying to use Local Outlier Factor analysis to identify the most unique or strange values. How does one decide the k-value to use in LOF analysis? I ...
4
votes
4answers
95 views

Econometric Model and deciding the frequency of data collection

I am looking to build an econometric model and I am wondering if using annual data vs monthly or quarterly data is going to produce a less accurate model. If the dependent variable is affected by ...
-1
votes
1answer
31 views

guidance and help required on improving open source ML/Data Mining Libraries

We would like to crawl a bunch of websites for specific information like the about us,company,technology pages of start-ups and enable sharing it across a social network which my organization is ...
0
votes
1answer
33 views

The performance of the classifier went bad after adding extra two features

I'm solving a classification problem using 10 features and logistic regression. The performance of the classifier is fine when I use the 10 features only, however when adding another 2 features, the ...
0
votes
1answer
40 views

Are there any data mining methods with highest specificity by default?

I am solving a problem of binary classification with up to 50 continuous and categorical predictors, where the class of interest is quite rare (1-5%). In addition, I would like to be very specific ...
0
votes
0answers
18 views

Generalised itemset mining?

As far as I understand, the common approaches to itemset mining assume the following: An itemset is a conjunctive logical clause (which in some approaches allow negation) All items are in principle ...
2
votes
1answer
45 views

What statistics could an online book store use? [closed]

This is a theoretical question. If I had an online bookstore what kind of statistics would I keep. The number of times a book was viewed is one example. Another example may be the number of visitors ...
1
vote
0answers
31 views

Predicting problem hardness based on feature vector

Given some optimization problem. For each instance of this problem I can obtain a number of features, say [p1, p2,..., pn]. Features could be for example: number of customers, number of vehicles, ...
0
votes
0answers
33 views

how to interpret “lift” and “odds ratio” in association rules?

How can I interpret "lift" and "odds ratio" in association rules? if there is a rule such that, X=>Y and lift = 3.48 & odds ratio = 10.09
1
vote
0answers
27 views

Feature space reduction for tag prediction

[x-post] from stackoverflow. I am writing a ML module (python) to predict tags for a stackoverflow question (tag + body). My corpus is of around 5 million questions with title, body and tags for ...
0
votes
1answer
25 views

IBM Synthetic Data generator

I am implementing some algorithms in association rule hiding. I need to apply these algorithms in different data sets. I searched for IBM synthetic data generator through Google but I could not find ...
0
votes
1answer
17 views

Decision Tree English Rules and Dependency Network in MS SSAS

I created a Decision Tree model in Microsoft Analysis Services (SSAS, Visual Studio 2010). There are two tabs in the Mining Model Viewer tab: (1) Decision Tree that shows a tree itself, and (2) ...
1
vote
1answer
41 views

Is it better to use MAE or MSE for perfomance measure? [duplicate]

My data set is about forest fires in Portugal. I want to define a model that can predict better wildfires. In my data set, the outliers are entries referring to big fires. What is the best performance ...
-1
votes
0answers
41 views

When to use K mean clustering and hierarchical clustering algorithm? [closed]

Can you please tell me when to use the K-mean clustering and hierarchical clustering algorithm and what is the different between them... Regards, Rahul
1
vote
0answers
41 views

How to compare and cluster sets of daily time series?

I have multiple dataframes each representing traffic speed for each day of the year (366 dataframes for 366 days of the year). The raws of the dataframe are timestamp from 00:00 to 23:55 at 5 minute ...
2
votes
1answer
38 views

What does it actually mean for classes to be balanced?

I saw the following statement when reading Kuhn's APM: "The classes are fairly balanced; there are 111 samples in the first class and 97 in the second..." I thought balance would require the ...
0
votes
1answer
56 views

Summer school on data mining & ML [closed]

I'm a PhD student in Physics and this summer I'd like to attend a one/two weeks summer school on data mining and machine learning. Do you have one to recommend? Thanks!
0
votes
1answer
73 views

Statistically Approximating Clicks From Limited Data

Assume a business started in January 2014. I have the following daily data (from June 2014 to December 2014): 1. Number of people who joined the website; 2. Number of people who left the website; ...
0
votes
1answer
31 views

Can you add the probabilities of a classifier to better predict an outcome?

Say I am interested in predicting the TOTAL number of people that survive the titanic disaster, NOT each individual who died. Is it possible to run a probabilistic classifier on the data getting a ...
0
votes
2answers
44 views

Data Sampling while preserving the underlying distribution

I have a large 10-15 dimensional data set with close to 10 million points. I want to test some algorithms over a chunk of this data. But I don't want the character of this data to be lost by selecting ...
2
votes
0answers
31 views

How can I predict based on several time series of many different projects?

I want to predict the time that a client takes to pay for a service that has already been received. We are talking about a construction company, so the payments are always overdue since the company ...
0
votes
0answers
23 views

How much can I restrict my data with outliers?

I know there are tons of questions on CV.SE about outliers, but I didn't find a solution to my specific case. I have a dataset that I'm analyzing where in order to achieve "good" results, the data ...
0
votes
0answers
19 views

adaboost with multiple classification algorithms

Up to now I saw that all adaboost implementations use single classification algorithm and a training dataset as input and then creates multiple classification models by re-sampling dataset and uses ...
0
votes
1answer
27 views

MinHashing and Hash Digests/Signatures

Nilsimsa and SimHash are two similarity hash algorithms that produce a single hash digest/signature per document. Comparing two documents is as simple as computing the hamming distance between two ...
0
votes
0answers
86 views

Gower's dissimilarity measure and Ward's clustering method

I have read some topics in this web side that, it is not true to use Gower's dissimilarity matrix for Ward's clustering algorithm. I have mixed type variables, first I had a dissimilarity matrix ...
1
vote
0answers
22 views

A paper that proves using the latent features of RBM as input to logistic regression?

I'm looking for a paper that includes a proof that simply training a Restricted Boltzmann Machine and then using the latent features as input to a logistic regression classifier is a correct thing and ...
0
votes
0answers
22 views

Does a Restricted Boltzmann Machine model any distribution as a Gibbs distribution?

What i know is that a Restricted Boltzmann Machine (RBM) is a Markov Random Field (MRF) and that the joint distribution of an MRF represents a Gibbs distribution. What I also know is that our goal of ...
0
votes
1answer
58 views

Which statistical melhods could I use to determine if a price is good, based on a history of prices?

I have the following scenario: A history of prices of a specific product; The current price of the same product. The history of prices should contain prices with a certain amount of discount, very ...
3
votes
2answers
104 views

What does “Virgin Data” mean?

I am using RTextTools, which has a function to create container with following syntax: create_container(matrix, labels, trainSize=NULL, testSize=NULL, virgin) ...
0
votes
1answer
27 views

sklearn.tree.export_graphviz values do not add up to samples

When I run tree.export_graphviz() after training a sklearn.ensemble.RandomForestClassifier() on my data, I get some leaf nodes where the samples count doesn't match the value array, like this: ...
0
votes
1answer
39 views

When to use Gini impurity and when to use information gain?

Can someone please explain to me when to use Gini impurity and information gain for decision trees? Can you give me situations/examples of when is best to use which?
0
votes
1answer
42 views

Concept of p-vector and matrix

I have worked with vectors and matrices but the following paragraph from The Elements of Statistical Learning by Trevor Hastie et al is little confusing (online edition, page 10) Matrices are ...
1
vote
0answers
14 views

How to Mine Tree Structures?

To learn similarities/differences between different instances (that are in the form of tree), what are the suitable methods/approaches? I know kernel methods and particularly tree kernels, but would ...
0
votes
0answers
28 views

Model for the response of campaign in generalized linear model

Part of this question concern actual case and other is hypothetical case. Suppose that agent calls list of leads and offers them particular product. Agent records response in the CRM system which ...
1
vote
0answers
36 views

Time series classification and auto correlation

I am trying to understand time series data and data mining. I am trying to classify EEG data set. The classes are known in advance for the data set and the algorithm is trained on the example data ...
1
vote
1answer
16 views

Incremental improvement for boosting

By adding additional factors, will the fitting result of a boosting algo (say Ada boosting) guaranteed to be improved? From my experiment, adding additional factors could make the prediction accuracy ...
0
votes
1answer
32 views

Association rules or classifier for product modeling for queries

I have a set of products P {1...n} which are rated on a goodness scale G ={1...100} (G10 is more good than G5). Each product has a set of features F {1....m}, now I want to learn a model for ...
1
vote
0answers
81 views

Alternatives to Non-Linear Regression

I'm not a professional statistician but I frequently work in the area of data analysis using R and Python, and frequently use linear regression models (OLS) or quantile regression, and tree based ...
1
vote
1answer
33 views

Method for solving problem with variable number of predictors (repost from Data Science)

REPOST from Data Science: I've been toying with this idea for a while. I think there is probably some method in the text mining literature, but I haven't come across anything just right... What ...
0
votes
0answers
16 views

Edge prediction from Live Journal data in SNAP

I have downloaded data for the Live Journal graph from SNAP . The dataset looks like this From_Node_Id ->To_Node_Id ...
1
vote
0answers
36 views

difference between confidence interval and prediction interval in the context of regression analysis and predictive modeling

When building prediction models, I always see the following concept 1) Confidence interval for regression model 2) Prediction interval 3) Confidence interval for predicted value I can understand ...
1
vote
1answer
107 views

Quick Exploratory Analysis of Categorical Data

Does anyone know of a tool (preferably free) that does quick analysis of exploratory data mainly categorical with date. Using R and Python I can create time series and histograms, perform tests such ...
0
votes
0answers
9 views

Find input image (ID,passport) in imagesDB based on similarity

I would like to decide if image is exists on DB images (pictures of IDs,passport,Stu. card,etc) I thought of KNN alghorithem that will plot the K closest images. Options for distance metric: 1) sum ...
0
votes
1answer
42 views

After Clustering, how can I evaluate which features had the biggest impact?

I've just performed unsupervised clustering (using DBSCAN) on a dataset for which I have no expert knowledge on. I'm interested in working out which features had the greatest impact on my clustering. ...
0
votes
1answer
26 views

identify nature of missingness for categorical variables

could you please give me some hints for identifying the nature of missingness for categorical variables' missing value? I mean, I gave a fast search on google scholar but I didn't find anything ...
0
votes
0answers
39 views

Unsupervised learning algorithems to detect anomaly in waves

I have a sample of graphs (more then 10000...). that look like in the image below: I am searching an Unsupervised learning algorithems thet can help me to detect Anomaly observations. Here what i ...
2
votes
2answers
71 views

Best algorithm for association rule mining

I am working on an application where I have to extract or identify association / correlation between different sets of items. An example would be say if a person buys shoes at a store, would he/she ...