Questions tagged [data-mining]

Data mining uses methods from artificial intelligence in a database context to discover previously unknown patterns. As such, the methods are usually unsupervised. It is closely related but not identical to machine learning. Key tasks of data-mining are cluster analysis, outlier detection and mining of association rules.

Filter by
Sorted by
Tagged with
1
vote
1answer
1k views

What is the confidence for rule for $\emptyset \implies A $?

I'm looking at a question for association rule mining and this comes up: What is the confidence for $\emptyset \implies A $? What is the confidence for $ A \implies \emptyset $? Given: $$\...
2
votes
1answer
42 views

Privacy preserved record linkage

What is the best way for privacy preserved record linkage in data mining. I am final year university student. My thesis topic is related to data mining. I am working with data and I need to privacy ...
0
votes
0answers
29 views

Alternatives to classification for natural text data

I'm working at a project that would like to do topic extraction / classification from data, consisting of various NL sources (tweets, social network updates, pastebin). Data are very diverse in ...
1
vote
1answer
794 views

Multiway Array Aggregation

I'm having trouble understanding how the calculations for multiway array aggregations work using the example used in https://docs.google.com/file/d/0B5Ju2x50v6l5X0p3WDEwNkNTVW8/edit at page 165. ...
0
votes
0answers
38 views

Building a Diagnostic System for Psychiatric Disorders

I am an undergraduate with near-infinite passion to the theoretical machine learning and ML applications. Inspired from my challenging mental disorders, I am really interested in building a ...
0
votes
1answer
131 views

visualize a multi column matrix

I would like to visualize a 2000*100 matrix to find how different variables change based on values. Can you suggest a way doing it using R or Matlab? I have data from patients who use an ECG device ...
1
vote
1answer
2k views

Classifier for continuous data?

I am trying to classify a topographic cross section (profile) using a machine learning method. The classification consists of 2 main classes (scarp, no scarp) and 3 sub-classes (cls1, cls2, and cls3) ...
0
votes
0answers
407 views

Goodness-of-fit vs maximum likelihood for logistic regression?

From what I understand, maximum likelihood is used to estimate a parameter alpha in a way that maximizes the probability P(Y=|x,alpha) for example. It is used for logistic regression in order to get ...
0
votes
1answer
563 views

How do I calculate test error for adaboost?

I've calculated an adaboost algorithm for 20 iterations with a decision tree as my weak learner. I want to make a graph that plots the training error and the testing error. I have the training error,...
1
vote
0answers
248 views

Why do we convert data from long-form to wide-form for analysis?

I convert data from long-form to wide-form for my thesis. It makes sense intuitively for me to do so as I can easily see the relationship between every variable. However I was wondering is there a ...
1
vote
1answer
123 views

Becoming a data scientist using only StackExchange sites - what questions should I look at? [closed]

If I had a maths degree with a little foundation in statistics, what would be the top $100$ questions/posts on CrossValidated or MathStackExchange or MathsOverflow or Stack Overflow that I would have ...
1
vote
2answers
156 views

Is it reasonable to use rules to label positive samples when doing fraud detection in machine learning?

We use supervised learning algorithm to detect fraud, but we have fewer or even no positive samples, is it reasonable if we use rules to label positive samples? if so, is the supervised learning model ...
4
votes
2answers
7k views

What's the difference between Leave-One-Out and K-Fold Cross validation?

As far as I know in K-fold cross validation the samples are split into k sets and at round k-1 of these are used for the training of the model and the last one is used for testing the model and ...
1
vote
1answer
66 views

What kind of analysis can someone perform given a swap rate curve of a currency pair at different times?

The dataset that I have looks like the one attached - any ideas on how data like this can be used to extract anything meaningful? I would appreciate any help or ideas you might have. Thanks in ...
0
votes
1answer
3k views

How do weka classifiers deal with missing values? [closed]

I tried using a training set that has missing values. I applied filters (like replace missing data) and then after there were no more missing data I applied naive bayes, trees etc... I thought this ...
0
votes
0answers
22 views

What Statistics Can I Get from This Grading Procedure

I am a mathematician, not a statistician. So, I am hoping this question is not too broad. I'll delete if the community wishes. I recently read on Terence Tao's blog about a way to give partial ...
1
vote
1answer
57 views

How can I rate the difficulty of a question in an adaptive learning platform?

I'm working on an adaptive learning platform with some friends for a project that helps students to prepare their university admission test. We provide different kind of practice sessions and we have ...
0
votes
1answer
89 views

How to determine recurring use numbers for my app

A few months ago, I wrote an Android App. I am getting two key pieces of data from this app regarding its usage. The first metric, is number of downloads per day. The second metric, is basically, ...
1
vote
3answers
3k views

binary and multiclass classifiers

I have a simple yes/no problem so I was naturally inclined towards using a binary classifier because I was reading the book, A Course in Machine Learning by Hal Daumé III and I quote from it: [ Binary ...
0
votes
1answer
2k views

Classification and mixed categorical and numeric variables

I've been working a little with weka and so far I haven't made my own database to apply a classifier but I've tried to look at the already existing files and from what I've seen there is absolutely no ...
2
votes
1answer
2k views

Naive Bayes and independence

In every example I see(spam, negative vs positive tweet , weather study...) there is always the assumption that the input features (or variables) are independent. In order for me to be able to ...
1
vote
0answers
130 views

What is the best way to make use of numerical variables together with text mining for classification purposes?

I have a document-term matrix and I was able to classify the documents into their respective classes using svm. Now, I have numerical variables associated with the documents which I think might ...
0
votes
1answer
201 views

what are mixed variables in data mining?

I read that neural networks, SVM and neuro-fuzzy don't support "mixed variables." So what are those exactly? Does it refer to mixed types (numeric and non-numeric)? And if so, does that mean the ...
1
vote
1answer
203 views

What would be the final hypothesis like? if our unknown target is a distribution rather than a function?

The above picture is about building model, it seems a bit difficult to understand the meaning of "plus noise", and what would the final hypothesis look like? if the unknown target changes from ...
0
votes
0answers
557 views

Classifiers for small data sets and low dimensional features?

I have a small data set (under 100 samples) and 5 input features. I often hear about how neural networks are prone to overfitting under such conditions and that naive Bayes is likely to underfit. ...
0
votes
1answer
26 views

How do data mining classifiers behave when we add training samples to previously established training models?

I have yet to choose a classifier to get my model but I wanted to see which possible classifiers fit what I'm looking for. In fact, I need an algorithm to establish offline training this way: The ...
-1
votes
1answer
82 views

How do big companies apply machine learning? [closed]

I was wondering how recommendation on youtube work for example? How are the algorithms applied, because every user gets different recommendations depending on his location, his past liked videos etc......
1
vote
1answer
188 views

Classification vs regression machine learning?

While studying data mining methods I have come to understand that there are two main categories: -Predictive methods: classification Regression -Descriptive methods: Clustering Association rules ...
2
votes
0answers
476 views

Why StackingRegressor doesn't catch the trend?

I just reviewed very good example of fitting StackingRegressor from mlxtend package. ...
-1
votes
1answer
56 views

How to determine which columns/values in a dataset/dataframe can be Predicted which cannot?

I have a data-set/data-frame with columns Description, Department Name, Priority, Doctor name & Location.Description data comes as free text from from the UI. Based on historical description's in ...
4
votes
1answer
4k views

Understanding NIPALS algorithm for PLS

I've found a nice presentation describing PLS1 and PLS2 algorithms (pages 16-19). It's pretty clear but there is a thing confusing me. For PLS1. Let's look at the algorithm. The first steps are $w = ...
1
vote
0answers
488 views

Improving forecasting output using tbat's method

is tbats method the best model to use for my example ? for a weekly seasonality ? i want the predicted data to be like the initial data but that's not the result obtained so here's my data and the ...
1
vote
2answers
56 views

lower-bound of data dimension when using a deep learning architecture

I have a (X,Y)=(100,5) dataset (non-image) that I used with a deep linear classifier on Tensorflow to train and evaluate. At the ...
0
votes
1answer
767 views

What is the good method to select initial seeds in K-means? [duplicate]

in text documents clustering when k-means using as base algorithm, and VSM is a matrix for doc-term weighted by tf-idf, what is the best metric can be used for select an optimal initialization points (...
4
votes
2answers
417 views

lifetime of fraud detection models

Suppose we are building/testing a fraud detection model for a specific credit card/ or a quick cash loan business. We have a lot of data to play with (say past 5years), and after careful preprocessing,...
8
votes
1answer
8k views

What is the difference between patch-wise training and fully convolutional training in FCNs?

In the paper of fully convolutional neural network, the authors mention both patch wise training and fully convolutional training. My understanding for the training set construction is as follows: ...
1
vote
0answers
212 views

ML Datasets for Telecommunications Networking

I am working on a telecommunications networking project and I am interested in datasets which contains the following features: source/destination IP packet size. protocol. Port number. I have been ...
1
vote
1answer
61 views

Finding the attribute that affects the outcome the most

Lets say I have a medical data set of cause of deaths. ...
2
votes
1answer
315 views

Random forest feature construction: use two separate features or the ratio?

I am constructing a random forest model to predict a dependent variable Y. Two features are X_1 and ...
2
votes
0answers
501 views

How to do feature engineering of real time data?

I have made a good linear regression model with following step: Data Integration Data normalization/scaling(data preprocessing & feature engineering) Model Building(using linear regression with ...
1
vote
1answer
42 views

What will happen if features are correlated to output?

I was doing CS109 lab. There I saw this written:- "By the way, there is a problem with pre-doing feature selection before doing cross-validation. Ideally one should be doing the feature selection ...
5
votes
1answer
3k views

How to derive the time computational complexity of k-medoids (PAM) clustering algorithm?

I have read that the time complexity of k-medoids/Partitioning Around Medoids (PAM) is O(k(n-k)^2). I am trying to understand how this algorithms translates into this time complexity. As per my ...
18
votes
3answers
44k views

What is the difference between bagging and random forest if only one explanatory variable is used?

" The fundamental difference between bagging and random forest is that in Random forests, only a subset of features are selected at random out of the total and the best split feature from the subset ...
3
votes
1answer
1k views

Does correlation between variables and class label affect building a good classifier?

If all variables that I tested have low correlation with the class variable, would it be possible to build a good classifier? And if there was a high correlation between some variables and class ...
1
vote
0answers
46 views

C-statistic as a performance measure for binary classifiers

As c-statistic represents the area under the ROC curve, can we report c-statistic for any binary classifier or it must be a logistic regression and for the rest we can use the AUC under the ROC curve?
2
votes
1answer
1k views

How to convert binary category feature to numeric feature, like 'Gender'?

In a machine learning or data mining problem, suppose I have a original feature named "Gender", and now I want to convert this feature to numeric feature, there are two ways to do that, but I realy do ...
2
votes
1answer
191 views

How can I rate the question according to the difficulty?

I have a set of problems to analyze from a programming contest. My work is to rate the question according to its difficulty (easy,medium,hard) by analyzing the ...
10
votes
1answer
16k views

K-means: How many iterations in practical situations?

I don't have industry experience in data mining or big data so would love to hear you sharing some experience. Do people actually run k-means, PAM, CLARA, etc. on a really big dataset? Or they just ...
15
votes
1answer
10k views

Relationship between Hessian Matrix and Covariance Matrix

While I am studying Maximum Likelihood Estimation, to do inference in Maximum Likelihood Estimaion, we need to know the variance. To find out the variance, I need to know the Cramer's Rao Lower Bound, ...
0
votes
2answers
440 views

How can I map data to lower dimension?

I am trying to learn data in higher space into lower space. To have a clue, I'd like to know how to transform the data in the image below into a lower dimension preserving the structure. Hope to hear ...

1
3 4
5
6 7
24