Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Data mining uses methods from artificial intelligence in a database context to discover previously unknown patterns. As such, the methods are usually unsupervised. It is closely related but not identical to machine learning. Key tasks of data-mining are cluster analysis, outlier detection and mining ...

0
votes
0answers
9 views

SVM Regression n R

Hi The following graph represents original data, linear regression and Support vector regression. I would like to know if this is a decent plot of SVR and not understanding how to predict for a new ...
0
votes
1answer
20 views

Calculating lift

I was reading about lift for frequent itemsets and came across this exercise here which was taken from a data mining textbook. I tried plugging in the numbers, but could not get the 0.89 value for ...
0
votes
0answers
15 views

Identifying patterns

We have logs coming from a system that indicates the number of items bought in a particular transaction and the classes that are involved in handling this took how much time in each system. From the ...
0
votes
0answers
12 views

what to do if the missing data in one column is based on some value/condition in another column in r?

I have a dataset with 20,000 observations and 19 variables. To start off with I have a gender column which has three levels namely 'M', 'F' and 'U' where U can be taken as not disclosed. Whenever ...
3
votes
1answer
93 views

Too many dummy variable in regression model

we have about 50000 models of mobile phone (like Galaxy S7, iPhone 9) in database and the size of data is about 3 million. We want to find the mobile phones that have the least call success rate ( ...
0
votes
1answer
20 views

Why does creating training sets with replacement lead to better performance?

Pasting generally suffers from lower performance than bagging, because it training sets without replacement. Why does creating training sets with replacement lead to better performance?
5
votes
1answer
68 views

Functional Data Analysis and Splines Regression

I'm new to Functional Data Analysis so my question will'be very simple for an expert in this topic. Operatively, when I fit a model like this, in R : model<- lm( y ~ bs( x, df = k, degree = l ) )...
0
votes
1answer
25 views

Statistical time series analysis using Percentile and Empirical Rule

I analysing time-series to see the range of an asset. I have data for GBPUSD so i could only analyze this so far. I have used both empirical rule and percentile. Of course they give different ...
0
votes
0answers
4 views

Supervised ML opinion mining of medical entities: tools?

I would like to extract relations between the entities of 'quality of life' and 'health interventions' out of a corpus of medical texts via supervised machine learning. I wonder whether there is a ...
1
vote
0answers
15 views

Log mining machine learning approach

I am looking for guidance on a log mining problem I am encountering. My dataset comprises of lines like this with low veracity: I am trying to predict whether or not there is going to be a delay on ...
0
votes
0answers
32 views

Data Mining and Multiple Testing

I have a dataset A which consist of 100000 data points. I am trying to find hidden relations in the data. I am following the steps below: Step 1. I partition data in 5 different random sets (20000 ...
0
votes
0answers
14 views

can focal loss function works for text classification problem?

I am working on a relation extraction and classification problem. The data is in the form of text files. The data is imbalanced. I want to use focal loss function to address class imbalance problem in ...
0
votes
0answers
5 views

Combining 2 different spatial datasets

I've tried to keep the formulation as general as possible. Problem: For a given location (e.g. a county, country), I have 2 different datasets (D1, D2) where the rows are of the form D1: ...
-2
votes
1answer
57 views

Why isn't Matlab's Classification Learner the End All? [closed]

After building some machine learning models in Python, R and Matlab, I found the latter's Classification Learner App to be immensely powerful. In the time it took me to build a single model in Python ...
0
votes
2answers
24 views

What are possible approaches for learning causal DAG of events?

I have historical data of event logs. Each event has an associated contextual Id, which can be used to tell that event A happened first in some context, then event B happened in same context and then ...
0
votes
1answer
38 views

Violating “compactness” in single linkage hierarchical agglomerative clustering

While I was studying Hierarchical Agglomerative Clustering in the book Elements of Statistical Learning in the chapter Unsupervised Learning, I came through the following : The statement The ...
1
vote
1answer
38 views

Clustering similar user reviews

I am using Amazons Video Games dataset ( 1.3 mil user reviews ) in order to make a recommendation system using LSTM networks. What I would like to do is to cluster ...
0
votes
1answer
40 views
0
votes
0answers
22 views

Formulating an Anomaly Detection method for a dataset based on time

I have the following data (temporal feature) for n videos of given length: start times of all the users who watched the video duration of the video viewed Now, given a video (given the start times ...
3
votes
1answer
149 views

When is weighted average of $F_1$ scores $\simeq$ accuracy in classification?

Example where accuracy $\simeq$ weighted average of $F_1$ scores I have a classifier which classifies between 2 classes, $\mathbf{A}$ and $\mathbf{B}$. Let's say that we have the confusion matrix ...
0
votes
1answer
79 views

classification With ordinal data in R ?

I have a survey data which is Likert scale ordinal data . I want ordinal classification with R . I try several data mining algorithm with this data set but my accuracy is too low . How can i improve ...
1
vote
0answers
178 views

Information Gain vs Gain Ratio

In the building of a decision tree, when it's better to prefer the information gain criterion to the gain ratio criterion ? And why ?
2
votes
0answers
29 views

Decision tree ,information gain and overfitting

If i use the information gain in order to evaluate the best split in a decision tree, why using a binomial split reduces the risk of overfitting ? Is the information gain test misleading if we have a ...
0
votes
0answers
11 views

What are the characteristics of a good cluster?

I am analysing a data set using Clustering Algorithm in SAS Enterprise Miner. I have tried to change various settings to get the best results. But,I dont really know how to decide whether the model ...
1
vote
0answers
57 views

Simplistic Write-Up of XGBoost Algorithm

I was wondering whether anybody could check through my very simplistic (e.g., no detail of greedy split finding algorithm) write-up of the XGBoost algorithm for binary classification. Algorithm: ...
0
votes
0answers
158 views

Where could I find pseudocode for XGBoost?

I was wondering whether anybody knew where I could find pseudocode for the XGBoost algorithm?
1
vote
1answer
1k views

What is the “binary:logistic” objective function in XGBoost?

I am reading through Chen's XGBoost paper. He writes that during the $\text{t}^{\text{th}}$ iteration, the objective function below is minimised. $$ L^{(t)} = \sum_{i}^n l(y_i, \hat{y}_i^{(t-1)} + ...
2
votes
2answers
121 views

How to scale predictions from a neural network in R when the output is not a part of the dataset

I've been using a neural network to make predictions. So my training data is in one .csv file which I read-in and then scale. My test data is in another file that I read-in and is also scaled. However,...
1
vote
0answers
105 views

Why the contour of lasso and ridge regression are drawn only at that position and how do they draw like that?

Why all of the contour were drawn only at this kind of position and how do this contour drawn. Thanks
0
votes
0answers
15 views

Discretize numeric attribute values

I have a numeric attribute (e.g. revenue attribute), I must discretize its values before I use WEKA tool, but based on what I must divide them into categories? Since this may affect the classification ...
2
votes
0answers
51 views

What data mining methods can I use for a large (800x100000) matrix that don't require a super computer? [closed]

What data mining methods can I use for a large (800x100000) matrix that don't require a super computer? Because upon trying to do data mining on such data set, even basic PCA in sklearn runs into ...
3
votes
1answer
84 views

How to make a regression plot more insightful and engaging to an end user

I have an interview soon for a graduate position. I will to do 10-15 presentation. So, I was sent out a regression plot by the recruiter and I have to present how I can make the regression plot more ...
0
votes
1answer
68 views

Does it make sense to apply feature selection on a data set that has undergone principle component analysis?

I have a data set that consists of credit card transactions. This data set contains only numerical values which are the result of principle component analysis. I'm currently trying improve my models ...
1
vote
0answers
66 views

What is a good percentage value for SMOTE in Weka

I am using the oversampling technique named as SMOTE to balance my dataset in Weka. To balance my dataset I needed to use 1300 as my -p (percentage) value. The default -p (percentage) value is 100. ...
0
votes
0answers
33 views

Using LASSO to do the feature selection [duplicate]

When we do a LASSO feature selection, do we have to standardize each feature?
3
votes
1answer
527 views

Mix of text and numeric data

I have to train a classification model with 15 classes based on data which contains both textual and numeric data. For instance: product description(textual), product length'(numeric). I have ...
1
vote
1answer
220 views

Anomaly detection across repeated time-series with time delay

I am considering repeated time series (for example on day 1, day 2, ..., day(n)), and I want to find the anomalous points in the day(n+1) given that some delay between samples across different days is ...
0
votes
0answers
34 views

Best method for clustering product arrival dates

I work for a retailer, and I am trying to analyze the performance of each "collection". It is not an attribute we have formally defined in our systems, so I am trying to create it via a Clustering ...
0
votes
0answers
16 views

Determine whether feature discriminates between two classes

I have a large dataset of documents for which I have calculated the normalized occurrence rate of each word (i.e., normalized term-frequency) and have a label. Each document is labeled as either "HIGH"...
0
votes
0answers
29 views

am I doing the sampling techniques correctly and how would I analyse different algorithms?

I'm doing an undergrad project in which I compare the performance of various classification algorithms which are logistic regression, decision trees, and support vector machines. The data set I chose ...
2
votes
0answers
29 views

Is Multiple Correspondence Analysis applicable to Multi-valued Categorical Variables?

I have a data-set containing only Categorical Variables. I needed to do Principal Component Analysis on the data set. Eventually, I found Multiple Correspondence Analysis and learnt it. But, in MCA, ...
1
vote
1answer
60 views

Regression with more features than samples

I am new to the subject and I encountered this question in my text book: Suppose that the number of features is greater than the number of samples. e.g: $𝑦_1 = b_1𝑋_{11} + b_2𝑋_{12} + b_3𝑋_{13}$ ...
0
votes
0answers
5 views

grouping and categorizing differently spaced out events

I have data of client's visits to a food shelter. For each day, over 10 years, there is a 1 if the person went to the pantry and 0 if they didn't. I'm trying to group people whose pattern of use is ...
1
vote
0answers
37 views

Comparing clustering stability over period of time [duplicate]

Suppose I am applying K-means clustering on two datasets generated by same process. How to compare clustering stability over a period of time? Let's say I have applied clustering in 2016 on a data set,...
0
votes
0answers
125 views

scores and loadings in princomp (PCA)

I applied PCA on the dataset, but I am confused what should I use to build the model, the components in scores or in the loadings? @ usεr11852 ...
1
vote
0answers
48 views

use test data set after 10 Cross-validation

I applied 10 Cross-validation but I am a bit confused, I am not sure what is a correct way. 1- Should I apply 10 Cross-validation on all dataset, divide it into 10 folds and sum all the 10 matrices ...
0
votes
0answers
134 views

How to calculate probability of Bayes network?

Hi I'm new into Bayes network I'm trying to calculate P(F&G) but don't have the slightest clue how because everything i find online doesn't have a network that is this complex. I will really ...
0
votes
0answers
16 views

Difference in definition of support for frequent itemsets and association rules

My professor at university made the statement, that there's a difference between the definition of "support" for FISs and ARs, because "when looking for FIS, support relates to the set of products ...
0
votes
1answer
141 views

AdaBoost algorithm question

In the boosting algorithm,AdaBoost ,those observations which were misclassified by the classifier in the (m-1)th step have their weights increased in the mth step, and those which were correctly ...
0
votes
1answer
39 views

Is this an unsupervised pattern recognition problem?

Consider a feature set containing values of operating condition of an automation unit such as pressure of the valve, temperature, fuel consumption. I want to discover unknown relationship between ...