Questions tagged [data-mining]

Data mining uses methods from artificial intelligence in a database context to discover previously unknown patterns. As such, the methods are usually unsupervised. It is closely related but not identical to machine learning. Key tasks of data-mining are cluster analysis, outlier detection and mining of association rules.

303 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
5
votes
0answers
192 views

Determining the effect of number of likes

Let's say I have marketing data and I need to determine how effective the marketing is. The marketing strategy is to publish facebook posts at inconsistent intervals. The goal is to see how the ...
5
votes
0answers
83 views

(Nominal) raters with no gold standard

A friend of mine took a document and broke it up into parts, then asked 5 subject matter experts to classify each part into nominal category A, B, C, D, or E. (I'm not sure yet, but D may be "All of ...
5
votes
1answer
140 views

Regarding the size of training data for building classifier

When we build a classifier, like SVM or Naive Bayesian, are there any generic rules or theoretical derivations on the size of training data set? For example, to train a SVM-based classifier, what ...
5
votes
0answers
1k views

What are some useful data visualisation or data mining techniques to investigate horse racing forms?

I have a dataset of 13k horse races from four different tracks with an average of 11 runners per race. In all, there are 26k unique runners in the data. For each race, I know who came first, second, ...
4
votes
0answers
392 views

How to test the result of cSpade

I'm new working with sequences rules and I am little lost about what is the next step. I already generate the rules but I'm not sure how to test them in a new dataset, in order to verify them. Do I ...
4
votes
1answer
108 views

Finding related words

I have several files, each of which contains unique terms which are related to each other(without sentence structure). So for finding the word relationships I created a dictionary of bi-grams for ...
4
votes
0answers
172 views

Record linkage when sources have different fields

I have read a little about record linkage, but it seems to me that a requirement is that all fields in both sources can be compared. For example, with sources A and B, an assumption is that we can ...
4
votes
1answer
341 views

Generating M/U Probabilities in Fellegi-Sunter Record Linkage

I'm working on merging records from several databases that cover the same entities, but share no reliably deterministic fields, leaving us with fields such as name and address to resolve identity. In ...
4
votes
0answers
2k views

SVM classifier (with soft-margin) implementation in R, gamma value and quadprog

I'm trying to implement a Support Vector Machine classifier in R and I have to solve the optimization problem using the quadprog R package which solves problems of the form : $$min_b \frac{1}{2} b^...
4
votes
0answers
2k views

How do I interpret this Weibull plot?

I was exploring Weibull analysis for understanding reliability of two specific specimen. I used the R package, "weibulltoolkit" and "survival" to get the plot in question. The dataset is big so I am ...
3
votes
0answers
3k views

Intuitive meaning behind support, confidence, lift and conviction

I'm learning about association rules and came across the common interestingness measures support, confidence, lift and conviction. I'm interested in the intuition behind your decision-making process ...
3
votes
0answers
1k views

Algorithm to detect time series anomalies (outliers) (using Apache Spark)

I am currently new to machine learning and I will be working on a project that involves using a Machine Learning library to detect and alert about possible anomalies. I will be using Apache Spark and ...
3
votes
0answers
59 views

Datamining: Why use algorithms for large datasets instead of using predictions based on a sample?

A growing amount of cluster algorithms have been developed for using large datasets such as CURE or BIRCH or filtering methods for common k-means. What is the advantage of using such an algorithm ...
3
votes
0answers
616 views

How to compare growth rates

I'm analyzing the growth rate of a data set, and am unsure what is the best approach to make sure the growth is independent from another data set. Taking a simple (fictive) example, I'd like to ...
3
votes
0answers
65 views

Several questions about using PCA on large data

I am using PCA on a square matrix of pairwise distances between 6000 elements, where the columns can be viewed as variables and rows as observations. Here are some of my questions and concerns: 1) ...
3
votes
0answers
132 views

Distributions of eigenvalues of random matrices: what can they be used for in data mining?

I've accidentally come across some papers discussing distributions of principal components of the sample covariance matrices. An example of such a paper is Johnstone, 2001, On the distribution of the ...
3
votes
0answers
365 views

Is Mean Deviation better than Standard Deviation for data mining?

I just read this article and it says Mean Deviation(MD) is more efficient than Standard Deviation(SD) when there are some errors in observations. Like in real practise. I don't know what 'efficient' ...
3
votes
0answers
231 views

Hyperparameter optimization in 6-dimensional continuous space

I am using Random Forest and Stochastic Gradient Boosting to predict a categorical target variable exhibiting severe between-class imbalance. I am using oversampling to make sure that the models do ...
3
votes
0answers
393 views

Estimating probability distribution function of data stream

Although a similar question exists, I couldn't find my answer. I'm not a statistician hence please neglect if some terminologies aren't correct and let me know if I am interpreting something wrong. ...
3
votes
0answers
93 views

Data Mining Methods with Nested Data

I have three general questions that I am really struggling to answer: a) When we have nested data (e.g. employees nested in departments which are nested in companies which are in specific ...
3
votes
0answers
5k views

3-4-5 Rule How to partition the sets?

In Data Mining course, we are taking 3-4-5 Rule to segments the data uniformly. I'm trying to understand these lines below, and how they are linked to the graph below too. If an interval covers 3, 6, ...
3
votes
0answers
79 views

Which technique to build a model returning a vector of values (in R)

In my current project I need to build a model returning a vector of actions for each observation. I need a suggestion which statistical technique is used in general in such cases. In a project, I ...
3
votes
0answers
151 views

Parameter space exploration

I do realise this question is quite specific and practical, but I seek for some general help which helps me progress further in my analysis. Let $y(\boldsymbol{x})\in\mathbb{R}$ be the function I'd ...
3
votes
0answers
332 views

Mutual information/pointwise mutual information for measuring prediction

I want to measure how well I predict a vector $Y$ (vector not a label) for observation $X$. Both $X$ and $Y$ have the same set of features ($1\times n$). For that, I thought of "scoring" the ...
3
votes
0answers
234 views

Concerns about housing data mining

Can anyone suggest me good papers related to housing data mining. I have googled for a while and couldn't find any recent paper. I am assuming that people have stopped doing any mining related to ...
3
votes
0answers
55 views

Do we need to apply the same transformation of predictors on a test dataset?

I first divide the dataset into training (75%) and test (25%). Then fit a logistic regression model on training data set. When fitting the model, I did some modification on independent variable, such ...
3
votes
0answers
1k views

Weka J48 decision tree problem

I have a CSV dataset which contains mean (Numeric), spread (Numeric), review (string), ...
3
votes
0answers
4k views

Validation error less than training error — implications?

I am running a neural net to predict used car prices, sample size is 800. Using both 10-fold cross validation (10 times) and 1/3 holdback (10 times), the $R^2$ for training is about 0.60 and for ...
3
votes
0answers
51 views

Can a different indexing approach improve the relevance and efficiency of search engine results?

I am creating a search engine, with the corpus consisting of websites crawled through a webcrawler (Apache Nutch). I need the query searches to be both fast and relevant. So far, I have been trying to ...
3
votes
0answers
365 views

Data prep / variable creation for predictive models

I was reading a couple of the write ups from a Kaggle challenge: Here is one and another and it got me wondering about variable creation in data mining and why there seems to exist so few texts or ...
3
votes
0answers
86 views

Determine optimal (recurring) frequency of discrete values

So I'm trying to determine a semi-efficient way to calculate the optimal (recurring) frequency of a set of data. The data only exists at random time periods, but it is assumed that the data set is ...
2
votes
1answer
23 views

Visualizing shared instances of p-values<alpha across large numbers of treatments

Assume a data table that presents the p-values of a large number of independent runs of a statistical hypothesis test. Each run represents a single test with two possible hypotheses (i.e., null and ...
2
votes
0answers
202 views

Is there a well established algorithm to match two documents on a semantic level?

I have a set of documents from a wide variety of topics and I would like to retrieve the ones that are more similar to a new document provided. A search based on common words is not good enough, so ...
2
votes
0answers
79 views

Decision tree ,information gain and overfitting

If i use the information gain in order to evaluate the best split in a decision tree, why using a binomial split reduces the risk of overfitting ? Is the information gain test misleading if we have a ...
2
votes
0answers
293 views

Is Multiple Correspondence Analysis applicable to Multi-valued Categorical Variables?

I have a data-set containing only Categorical Variables. I needed to do Principal Component Analysis on the data set. Eventually, I found Multiple Correspondence Analysis and learnt it. But, in MCA, ...
2
votes
0answers
271 views

Lift measure for a frequent itemset

I found it's possible to add the lift measure to the quality measures of a frequent itemset in R returned by the Eclat algorithm: ...
2
votes
0answers
456 views

Euclidean vs Manhattan distance behaviour in high dimension - curse of dimensionality

I have compared different distance functions by computing the average tf/idf distance between documents. My results show a range between $10-15$ for the Manhattan and a range between $1-1.5$ for the ...
2
votes
0answers
368 views

Clustering of high dimensional data

I am having a data set with 54 independent variables .Most of them are having zeros it resembles like sparse matrix .How to cluster this kind of data and is there any data pre processing like Box cox ...
2
votes
1answer
294 views

How to check cross validation scores for market basket analysis?

If I have a large set of transactions where in each I buy a set of goods and I want to do market basket analysis using either A-priori or FP Growth or any other data mining method, you typically get ...
2
votes
0answers
476 views

Why StackingRegressor doesn't catch the trend?

I just reviewed very good example of fitting StackingRegressor from mlxtend package. ...
2
votes
0answers
501 views

How to do feature engineering of real time data?

I have made a good linear regression model with following step: Data Integration Data normalization/scaling(data preprocessing & feature engineering) Model Building(using linear regression with ...
2
votes
0answers
111 views

Can Resampling be used for estimation and goodness of fit test?

I am trying to compare my data with empirical distributions. But I don't have enough data to cut them to estimation data and validation data. I am trying a resampling approach and would like to see if ...
2
votes
0answers
43 views

Leads and Ideas on Learning from disconnected data

Problem of learning patterns from disconnected data. I have two independent discrete variables, D1 and D2, with information similar to tables in the images. I do NOT have the broken down data for A+...
2
votes
0answers
164 views

What is the difference between Gradient Boosting Machines and Gradient Boosted Regression Trees?

I would like to know if Gradient Boosting Machines and Gradient Boosted Regression Trees are the same algorithm or different. If they are different algorithms what is the main difference between them?
2
votes
0answers
142 views

Libraries to search sentences which include semantically similar phrases to the query sentence's

Could you recommend libraries to search sentences which include semantically similar phrases to the query sentence's? More semantically common phrases included, more scores. I think I should be able ...
2
votes
1answer
33 views

How do I choose the appropriate numbers of customers to be considered for cluster analysis?

I am currently doing a customer segmentation project in SAS. I have identified 2700 customers who are have made a purchase in each of the 4 years I am analysing. For the cluster analysis the more ...
2
votes
0answers
164 views

How to find the correct spatial scale in landscape ecology?

I am currently studying the effect of organic farming on honeybee colonies. I have calculated the percentage of organic land in several buffer areas around the hives (from 100m to 3000m in 100m steps) ...
2
votes
1answer
383 views

Selecting the number of hashes for minhash? Working with extremely sparse data and want more collisions

I'm attempting to use minhash to generate clusters and similarities, and I am primarily using ideas from these resources. http://www2007.org/papers/paper570.pdf https://chrisjmccormick.wordpress.com/...
2
votes
0answers
219 views

Using entropy to imputing missing value based on grey relational analysis and clustering

This algorithm contain three techniques : 1-fuzzy c-mean clustering 2-Grey relational theory 3-Entropy multiple imputation The frame work of this algorithm is as follows : My questions are about ...
2
votes
0answers
63 views

Finding randomly excluded words in hundreds of documents

I have a problem that I am trying to solve using data mining techniques. What is known: There is 253 1 page documents that belong to 4 exclusive topics "clustering" "classification" "frequent ...

1
2 3 4 5
7