Data mining uses methods from artificial intelligence in a database context to discover previously unknown patterns. As such, the methods are usually unsupervised. It is closely related but not identical to machine learning. Key tasks of data-mining are cluster analysis, outlier detection and mining ...

learn more… | top users | synonyms

0
votes
1answer
38 views

How to get probability from the confidence score in SVM

In liblinear library we can get confidence score (the distance between decision hyperplane) in SVM solver for a binary classification problem, but if i want a probability value for membership in any ...
0
votes
1answer
42 views

Is ROC or PR curve only the overall performance measure for classification

We can use ROC or PR curve to access the performance of the classifier,especially on imbalance data. But it is a curve with parameter threshold, even if we get a high ROC or PR performance, which ...
3
votes
1answer
32 views

What's the measure to assess the binary classification accuracy for imbalance data

Now I have binary classification problem with positive samples roughly 100 times the number of negative samples. In this case the normal accuracy measure (predict == label) is not a good measure. What ...
-2
votes
0answers
11 views

Does theano support multi-machine version [closed]

We use theano in the GPU in a single machine, I am just wondering if it supports multi-machine programming so that the speed will raise up?
2
votes
0answers
33 views

Finding idf only for text mining

We find tf-idf for training phase in text mining, however, in test phase, we need the tf for each element in test set, but should use idf in train set, so is there any api in python that can calculate ...
0
votes
0answers
21 views

tf-idf in text mining

I used sklearn of Python for getting tf-idf attribute in text analysis, but the problem is: I have about 78000 words in train_set, but the tf-idf matrix only has 39000 words. What is the problem ...
0
votes
0answers
19 views

IKAnalyzer in text mining

Does anyone use IKAnalyzer for word segmentation in the preprocess for text mining? I have never loaded my own extended dictionary or stopword dictionary successfully. The following is the ...
0
votes
1answer
12 views

Value of support in frequent pattern matching

I'm working with an extremely small & sparse dataset (~20k lines of transaction data) where about 92% of the lines contain a single item and the remaining 8% contain between 2 and 7 items. I want ...
0
votes
0answers
21 views

What is a good non cryptographic Hash for string feature translation?

What would be a good non cryptographic Hash function to use for converting string features to a numerical representation for feeding into machine learning algorithms? To explain the scenario my ...
0
votes
0answers
34 views

Different Central Tendency Measures for Describing Groups or Individuals

Background: I have records with various metrics for many users. I want to develop a profile for each and also group similar users using clustering. Question: When developing a profile for each user, ...
-2
votes
0answers
52 views

What types of job opportunities are out there for data mining and machine learning? [closed]

I have asked a question about skills which lead to data mining/machine learning field (What are Ideal IT skills/programming languages for data mining/machine learning/AI field?), but I feel like my ...
-4
votes
0answers
44 views

Data Mining question - Computer Science [closed]

Assume we have two 'prime number detectors' called A and B. detector A uses the following equation: $z = 2x^2 + 29$ detector B uses the following equation: $z = x^4 + 29x^2 + 101$ test both of ...
-3
votes
0answers
23 views

Data Mining - Computer Science [closed]

Assume I have a test to detect whether a person has disease X. I get 10 people, 4 of which actually HAVE disease X and 6 who do not. I give them my test. My test says 6 people with the disease (of ...
-1
votes
0answers
27 views

Automatic identification of numeric vs categorical variable

I am trying to build an automatic data preparation pipeline. One of the capability which I want to include is the ability to tag a variable as Numerical or categorical. Currently the idea I have is to ...
0
votes
1answer
36 views

Regression model for edge-sensitive data set

I have data sets in which important information is allocated in the edges, which are also very sensitive to inaccuracies. I would like to find a regression model based on edge recognition that brings ...
4
votes
1answer
79 views
+50

Relative variable importance for Boosting

I'm looking for an explanation of how relative variable importance is computed in Gradient Boosted Trees that is not overly general/simplistic like: The measures are based on the number of times a ...
0
votes
1answer
33 views

Data analysis and prediction algorithm recommendation

I need some help. I'm a programmer but I'm not familiar with data science or analysis. I've been given a project which I have to do a research with a list of CSV data files. I converted some of those ...
0
votes
0answers
9 views

Large scale hierarchical text classification - real world data [closed]

I have recently found website dealing with large scale text classification competitions http://lshtc.iit.demokritos.gr/LSHTC4_CALL. I can see that the data provided there are quiet obfuscated. I ...
3
votes
1answer
63 views

Does Hashing data affect linear separability

What would be a good way to obfuscate sensitive information and store e-commerce transaction data, to later perform fraud analysis on it. One idea that crossed my mind, was to hash each sensitive ...
1
vote
1answer
24 views

How to choose sampling method for imbalanced data?

I have an imbalanced dataset with 4995:5 ratio as well as other datasets with less imbalanced ratios. I split this 4995:5 ratio into training and testing for about 2/3 training and 1/3 testing. I also ...
0
votes
0answers
32 views

Unsupervised Anomaly Detection with Mixed Numeric and Categorical Data

I am working on a data analysis project over the summer. The main goal is to use some access logging data in the hospital about user accessing patient information and try to detect abnormal accessing ...
0
votes
0answers
13 views

Clustering Standardized Mortality Ratio

I am bit new to the whole clustering idea. I have a data set that gives information about the SMR in the different states of America from 1995-2000. Hence I want to apply clustering techniques ...
0
votes
0answers
7 views

Recommended Study Area For Processes

I am looking for a machine learning area that deals in processes for logistics. If anyone can show me some use cases or even point me in the direction of a couple of algorthims. Im currently using R ...
0
votes
0answers
39 views

Downside to scaling and centering?

Bottom line up front: is there any reason not to center and scale continuous variables prior to model fitting for the sake of conducting model comparison? I'm conducting a model comparison on a large ...
0
votes
0answers
34 views

Getting an error message “Error in if(reached.threshold < min.reached.threshold)…” while training network using neuralnet package

I'm using R to create train and test a neural network on a time series (the annual sales of a company over a large period of time). As using the package's default learning algorithm (resilient ...
1
vote
1answer
55 views

Feature selection with genetic algorithm in R [closed]

I'm looking for a R-package that does feature selection using a genetic optimization algorithm. I couldn't find one on CRAN and I wonder whether there is a free one. I would be very appreciative for ...
0
votes
0answers
8 views

Literature about co occurence events in statistics

I was reading an article titled "Risk Mining in Medicine: Application of Data Mining to Medical Risk Management" there the authors Tsumoto and Hirano, mention the term "cooccurence events" and they ...
2
votes
1answer
71 views

What balancing method can I apply to a imbalanced data set?

I'm trying to solve one classification problem from the UCI database repository. Unfortunately (or fortunately), I've noticed that my dataset is imbalanced. I've structured the data as 5 classes, ...
0
votes
1answer
44 views

Which features are most relevant to each class in neural for network binary classification?

I designed a neural network for binary classification in MATLAB R2015a. What are differences between two classes? How system detects a sample is from class 1 or ...
0
votes
0answers
7 views

Finding the ratio of value factors in telecom

Before reading this i know the question i asked can be answered based on our strategies, but i'd like to know the methods of quantifying these strategies into numbers. I'm doing subscriber profiling ...
0
votes
0answers
17 views

whats the difference between Confidence and certainty factor?

please I need to know whats the difference between Confidence and certainty factor ? As I apply on generated Association rule to validate it and I notice that certainty factor =1 only if ...
0
votes
0answers
12 views

Automated Structured Data Extraction

I am following a paper on Structured Data Extraction titled : Finding and Extracting Data Records from Web Pages, M. Alvarez, A. Pan et al. I am writing code for same in ruby. The data region ...
0
votes
1answer
274 views

SVD interpretation of Climate data

I was going through my homework. I have a question to interpret the plot which i obtain from first and second left singular vectors after computing SVD. What can i exactly interpret, is it specific to ...
0
votes
0answers
18 views

Reconstructing individual rankings from popularity scores

I have the scores from lots of voters voting top N choices from >100 choices, and I want to reconstruct the rankings of all voters based on this distribution, to predict the coming results. What ...
0
votes
0answers
17 views

Bagging Improvement for classification problems

Breiman proved that bagging improves the accuracy of regression. How could this be proven for a classifier model?
1
vote
0answers
15 views

How to data mine google for most popular things out of a set?

Let's say I want to get the list of the top 5000 most searched biomolecules on google - how would I approach such a problem? One possible solution is to fetch the whole pubchem/pdb database and run ...
-1
votes
1answer
25 views

Data mining : I want to automate data scrapping process in ruby. Which approach would be better. Supervised or unsupervised?

I've read that the problem with supervised approach is it fails to generalize if the structure of web page changes. But sites rarely change their structure. Problem with unsupervised is it will ...
0
votes
1answer
30 views

Data Mining - Preprocessing [closed]

If in an attribute I have the following possible values: "low, low-medium, medium-high, high" how should I create my attribute. If I enter the strings as mentioned above and make my attribute ...
-1
votes
1answer
25 views

Decision tree learning

Suppose we want to know whether Joe will play football today. We have a dataset containing information about weather such as outlook, wind, and humidity and the decision he made (to play or not to ...
0
votes
0answers
37 views

How to qualify which features to use in a predictive model?

When building a predictive model, it's well established that picking/building the right features will draw the line between failure and success in the forecasting task. That's said, some people ...
0
votes
0answers
21 views

How to build kpi on employee<>customer<>sales

For further analysis i want to build a KPI which describes the "critically" of a employee<>customer relationship. To make it clear, some example data: Employee - Customer - ...
0
votes
0answers
12 views

reliefF vs Information Gain for feature selection

I have a mixed dataset (binary labels) - continuous, ordinary & discrete attributes and I want to perform feature selection. The objective is to understand most important factors influencing the ...
0
votes
0answers
22 views

Forming relational graph for a noun by mining web

I want to find a relational/relevance graph for any noun, by mining the web. For example the graph of sushi may be like : sushi -> fish(seafood),rice-> Japanese -> Food. PS : I may be missing some ...
0
votes
0answers
29 views

fuzzy clustering and multi-label classification

I’m working on a clustering problem that I would like to extend to multi-label classification. Basically, I want to generate a number (x) of clusters using something like fuzzy c-means and using the ...
1
vote
0answers
16 views

What are few techniques that use sequential pattern mining for predicting location?

I found many sequential pattern techniques like GSP Algorithm Sequential PАttern Discovery using Equivalence classes (SPADE) Apriori algorithm etc But they all work for data with certainty like ...
4
votes
0answers
135 views

Determining the effect of number of likes

Let's say I have marketing data and I need to determine how effective the marketing is. The marketing strategy is to publish facebook posts at inconsistent intervals. The goal is to see how the ...
0
votes
0answers
24 views

Prediction - important features from a new element

On my data I did LSI and got a large matrix (>200000 samples, >6000 features, very sparse). I do SVD on it, keeping only 150 dimensions. When I get a new element, do a folding-in, calculate cosine ...
3
votes
0answers
55 views

What challenges can data science help with in small businesses

Data science can solve/help with many challenges. But it usually looked upon as something for multi-million dollar corporations. I'm looking for examples of challenges data science helped with in ...
0
votes
1answer
66 views

How to implement knn in r with missing values?

I have this data set from https://archive.ics.uci.edu/ml/machine-learning-databases/credit-screening/crx.names which gives a good summary of the attributes im using. Some of the observations are ...
2
votes
0answers
48 views

Estimating probability distribution function of data stream

Although a similar question exists, I couldn't find my answer. I'm not a statistician hence please neglect if some terminologies aren't correct and let me know if I am interpreting something wrong. ...