Data mining uses methods from artificial intelligence in a database context to discover previously unknown patterns. As such, the methods are usually unsupervised. It is closely related but not identical to machine learning. Key tasks of data-mining are cluster analysis, outlier detection and mining ...

learn more… | top users | synonyms

0
votes
0answers
13 views

forecasting household size using longitudinal survey (panel-data)

I have a multidimensional longitudinal survey dataset that gathered general demographic characteristics (~100 variables) of ~10,000 households over a 10 year time period (3 surveys with 5 years ...
1
vote
0answers
36 views

How would you modify your model according to that confusion matrix?

How would you proceed with your model, if the confusion matrix looks like on the picture below? Classes 2, 3 and 4 get misclassified a lot among each other. EDIT: The Confusion Matrix was created ...
-1
votes
0answers
19 views

Is it possible to use WEKA in a web based application? [closed]

I am building a web based text mining application. For a word that user enters, the application has to: search it in google gather the documents pre-process by using Bag of words model cluster ...
0
votes
0answers
16 views

Appropriate cluster method for 7-point scale data of 58 variables and 635 observations

My research buddy and I are conducting cluster analysis on survey data using a 7-point relevance scale (1=Not relevant, 7=Extremely Relevant). We have 58 variables, arranged in 10 groups of ...
0
votes
1answer
43 views

Fisher Projection vs Linear Discriminant Analysis [closed]

Basically, I am confused between Fisher and LDA. Looking for differences between the two. How is the Fischer projection computed in R?
2
votes
0answers
36 views

Boosting: why is the learning rate considered a regularization parameter?

I understand that the learning rate parameter ($\nu \in [0,1]$) in Gradient Boosting shrinks the contribution of each new base model -typically small trees- that is added in the series. It was shown ...
1
vote
0answers
23 views

Requirement on data distribution when using certain data mining classifiers

I am a little bit confused on whether data mining classifiers make any assumptions about the distribution of the data. I myself have a computer science background, but when I have discussions with an ...
0
votes
1answer
18 views

Evaluation of binary approach to one vs all multi-class classification

I'm working on a multi-class problem which I have redefined as a series of binary problems (i.e. a one vs all classification problem). However, each observation can belong to more than one class. For ...
2
votes
0answers
42 views

Stories about statistics, machine learning and data mining [closed]

I'm looking for interesting stories about statistics, machine learning, etc. Examples of stories I find interesting are the following: https://en.wikipedia.org/wiki/Carl_Friedrich_Gauss developed the ...
0
votes
1answer
31 views

Meaning of Dummy Variable

I am working on a case study from Kaggle: https://www.kaggle.com/c/liberty-mutual-fire-peril Here, in the variables, there is a variable: "Dummy" about which the description says: (dummy: Nuisance ...
1
vote
1answer
53 views

How to approach a regression problem? [closed]

I have to solve a regression problem involving 302 variables. How do we select appropriate models or ensemble of models to work well ? Does this decision of choosing models come from the ideas ...
2
votes
1answer
35 views

Explaining the steps of a visualization tool

this is my first post on CrossValidated. I've done, for academic purpose, a web tool doing this process: web scraping from various sites pre-process the responses (cleaning, error and redundance ...
3
votes
2answers
95 views

Practical Question about the Assumptions of Support Vector Machines

As far as I know, the only assumptions of support vector machines are independent and identically distributed data. I am planning to train and run a SVM on a number of variables that aren't naturally ...
0
votes
0answers
15 views

Data Cleaning in R [migrated]

I have a csv file which I want to extract only the timestamp of the sentences which contain toward plus the fruit name in that sentence. How can I do this in R (or ...
0
votes
2answers
32 views

Text Pattern Recognition - Model building using R

I have a training dataset which has two columns which has around 70 values. “PNRNo” whose values like UT767G, CADA, 4I9I59, 4BH5TW…(typical PNR number patterns) I have created one more factor ...
1
vote
0answers
13 views

more appropriate domain in data mining for person less interested in maths [closed]

am doing mtech in computer science and i was thinking to submit my final thesis in data mining. but when i read about it i found that it includes stats and maths, and am not interested in them. i ...
0
votes
0answers
58 views

How to deal with imbalance data in , for example, neural network

Do we usually discard this issue, just train the neural network then compute the AUC, or can we use weighted version of loss function, for example, in binary classification problem, can we use ...
1
vote
0answers
11 views

How can I create a globally adjusted sampling distribution based on the regions of Morocco?

I have a table of population statistics from Burundi with the following: Province -- Population -- Sample Distribution -- Global Sample (Distribution Adjusted) The table continues with each province ...
0
votes
0answers
21 views

What value of (adjusted) McFadden R square or other pesudo R square means good fitting

I got adjusted MaFadden R square for logistic regression: 0.918772 , 0.6135568 , 0.3407252 respectively, which value is good? I just heard the value between 0.2 and 0.4 is good for McFadden R square. ...
1
vote
1answer
21 views

Is multiple stage binary classification a good idea if you have very few positives?

The problem is the following: We have a set of, say 5000 documents, with a single binary label. Say that 4900 documents are negative and only 100 are positive. I built a binary classifier while ...
3
votes
1answer
75 views

CART: Selection of best predictor for splitting when gains in impurity decrease are equal?

My question deals with Classification trees. Consider the following example from the Iris data set: I want to manually select the best predictor for the first split. According to the CART ...
1
vote
0answers
38 views

Clustering transactional data with R [closed]

I have a dataset with 3.205 observations and 6 variables about viewing habits from Rio and Sao Paulo in Brazil. They're supposed to have different habits so I need these data to form natural clusters ...
3
votes
1answer
42 views

What is the origin of the term confusion matrix?

The confusion matrix is a fundamental term in machine learning. However, that name itself is a bit, well, confusing. I tried to trace its origin but I'm not sure I'm correct, and I'd like to hear what ...
3
votes
4answers
158 views

Machine Learning Book (Python)

I'm in search of a good book about Machine Learning. Requirements: Good theoretical depth (while being a ML beginner, I hold a degree in mathematics), but accompained with good examples and plenty ...
3
votes
0answers
66 views

Advantages and disadvantages of machine learning hyperparameter optimizers [closed]

What are the respective advantages/disadvantages of the following optimization algorithms for ML applications? (that is, to optimize the hyperparameters of a SVM, RForest, Boosting model, etc.). In ...
0
votes
1answer
58 views

How to get probability from the confidence score in SVM

In liblinear library we can get confidence score (the distance between decision hyperplane) in SVM solver for a binary classification problem, but if i want a probability value for membership in any ...
0
votes
1answer
44 views

Is ROC or PR curve only the overall performance measure for classification

We can use ROC or PR curve to access the performance of the classifier,especially on imbalance data. But it is a curve with parameter threshold, even if we get a high ROC or PR performance, which ...
3
votes
1answer
80 views

What's the measure to assess the binary classification accuracy for imbalanced data?

Now I have binary classification problem with positive samples roughly 100 times the number of negative samples. In this case the normal accuracy measure (predict == label) is not a good measure. What ...
2
votes
0answers
119 views

Finding idf only for text mining

We find tf-idf for training phase in text mining, however, in test phase, we need the tf for each element in test set, but should use idf in train set, so is there any api in python that can calculate ...
0
votes
0answers
23 views

tf-idf in text mining

I used sklearn of Python for getting tf-idf attribute in text analysis, but the problem is: I have about 78000 words in train_set, but the tf-idf matrix only has 39000 words. What is the problem ...
0
votes
0answers
21 views

IKAnalyzer in text mining

Does anyone use IKAnalyzer for word segmentation in the preprocess for text mining? I have never loaded my own extended dictionary or stopword dictionary successfully. The following is the ...
0
votes
1answer
13 views

Value of support in frequent pattern matching

I'm working with an extremely small & sparse dataset (~20k lines of transaction data) where about 92% of the lines contain a single item and the remaining 8% contain between 2 and 7 items. I want ...
0
votes
0answers
23 views

What is a good non cryptographic Hash for string feature translation?

What would be a good non cryptographic Hash function to use for converting string features to a numerical representation for feeding into machine learning algorithms? To explain the scenario my ...
0
votes
0answers
37 views

Different Central Tendency Measures for Describing Groups or Individuals

Background: I have records with various metrics for many users. I want to develop a profile for each and also group similar users using clustering. Question: When developing a profile for each user, ...
0
votes
1answer
40 views

Regression model for edge-sensitive data set

I have data sets in which important information is allocated in the edges, which are also very sensitive to inaccuracies. I would like to find a regression model based on edge recognition that brings ...
4
votes
1answer
125 views

Relative variable importance for Boosting

I'm looking for an explanation of how relative variable importance is computed in Gradient Boosted Trees that is not overly general/simplistic like: The measures are based on the number of times a ...
0
votes
1answer
39 views

Data analysis and prediction algorithm recommendation

I need some help. I'm a programmer but I'm not familiar with data science or analysis. I've been given a project which I have to do a research with a list of CSV data files. I converted some of those ...
3
votes
1answer
67 views

Does Hashing data affect linear separability

What would be a good way to obfuscate sensitive information and store e-commerce transaction data, to later perform fraud analysis on it. One idea that crossed my mind, was to hash each sensitive ...
1
vote
1answer
31 views

How to choose sampling method for imbalanced data?

I have an imbalanced dataset with 4995:5 ratio as well as other datasets with less imbalanced ratios. I split this 4995:5 ratio into training and testing for about 2/3 training and 1/3 testing. I also ...
0
votes
0answers
40 views

Unsupervised Anomaly Detection with Mixed Numeric and Categorical Data

I am working on a data analysis project over the summer. The main goal is to use some access logging data in the hospital about user accessing patient information and try to detect abnormal accessing ...
0
votes
0answers
15 views

Clustering Standardized Mortality Ratio

I am bit new to the whole clustering idea. I have a data set that gives information about the SMR in the different states of America from 1995-2000. Hence I want to apply clustering techniques ...
0
votes
0answers
7 views

Recommended Study Area For Processes

I am looking for a machine learning area that deals in processes for logistics. If anyone can show me some use cases or even point me in the direction of a couple of algorthims. Im currently using R ...
0
votes
0answers
44 views

Downside to scaling and centering?

Bottom line up front: is there any reason not to center and scale continuous variables prior to model fitting for the sake of conducting model comparison? I'm conducting a model comparison on a large ...
0
votes
0answers
66 views

Getting an error message “Error in if(reached.threshold < min.reached.threshold)…” while training network using neuralnet package

I'm using R to create train and test a neural network on a time series (the annual sales of a company over a large period of time). As using the package's default learning algorithm (resilient ...
1
vote
1answer
57 views

Feature selection with genetic algorithm in R [closed]

I'm looking for a R-package that does feature selection using a genetic optimization algorithm. I couldn't find one on CRAN and I wonder whether there is a free one. I would be very appreciative for ...
0
votes
0answers
8 views

Literature about co occurence events in statistics

I was reading an article titled "Risk Mining in Medicine: Application of Data Mining to Medical Risk Management" there the authors Tsumoto and Hirano, mention the term "cooccurence events" and they ...
2
votes
1answer
102 views

What balancing method can I apply to a imbalanced data set?

I'm trying to solve one classification problem from the UCI database repository. Unfortunately (or fortunately), I've noticed that my dataset is imbalanced. I've structured the data as 5 classes, ...
0
votes
1answer
47 views

Which features are most relevant to each class in neural for network binary classification?

I designed a neural network for binary classification in MATLAB R2015a. What are differences between two classes? How system detects a sample is from class 1 or ...
0
votes
0answers
8 views

Finding the ratio of value factors in telecom

Before reading this i know the question i asked can be answered based on our strategies, but i'd like to know the methods of quantifying these strategies into numbers. I'm doing subscriber profiling ...
0
votes
0answers
24 views

whats the difference between Confidence and certainty factor?

please I need to know whats the difference between Confidence and certainty factor ? As I apply on generated Association rule to validate it and I notice that certainty factor =1 only if ...