Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

A partition is an assignment of every element of a set into 1 & only 1 subset w/ no empty subsets. A common instance of partitioning in statistics is the partitioning of sums of squares for F-tests.

0
votes
1answer
42 views

Statistical test of fairness of partitioning

I have a question about a statistical test, but I lack the proper terminology so I'll try to explain in my own words. Say I have a collection of $N$ objects. The collection is partitioned into $p$ ...
1
vote
0answers
57 views

Variation partition analysis: morphology-climate correlation, controlling for geography

I'm working with three datasets with informations about individuals: morphological data, climatic data of the individual sampling site and geographical data. My objective is to assess whether ...
1
vote
0answers
10 views

Cluster/partition by time

I have a dataset of different events in time. I want to group/cluster/partition the data by datetime. A small example of the time of the data might be: [19-09-2018 12:00, 19-09-2018 12:01, 19-09-2018 ...
0
votes
1answer
26 views

Estimation of partitioning error in next-generation sequencing experiments

[Edited: explanation of the partitioning error] I would like to estimate how the initial number of molecules (or the level of gene expression) affects reproducibility between technical replicates of ...
0
votes
1answer
97 views

Adjusted Rand Index with different size inputs

Forgive me if my terminology is off, this concept is somewhat new to me. I'm trying to run an ARI calculation between two partitions which contain a different number of clusters, where each cluster ...
0
votes
0answers
9 views

Appropriateness of explicitly modelling sums in hierarchical models

My lab is interested in morphological measures that have a partition structure. A simple example that has the same property would be studying effects on arm lengths. Each person has an armspan which ...
0
votes
0answers
35 views

Tools for Creating Balanced Data Partitions

How can I create balanced partitions of my data? By balanced partitions I mean divisions of my data that have distributions of all covariates and the response that are as similar as possible within ...
2
votes
1answer
105 views

Appropriate partition region in a decision tree?

I have the following partition region for a decision tree, but I do not think it is a valid region: $R=(\vec{X}=(X_1, X_2, ..., X_p)^T\in \mathbb{R}^p: X_1X_2 \geq 3)$ I feel that this is not a ...
3
votes
2answers
169 views

Predictors in random Forest

I am building a random forest to predict a binary variable y. I have several predictors named x1..n. One predictor, lets say x1, is a very strong predictor of y but only in some cases (see below) ...
0
votes
0answers
198 views

Using predictor variable to create data partition

I noticed that in the documentation for the createDataPartition function of the 'caret' package in R that they suggest the possibility of splitting data into training/test sets based on predictor ...
2
votes
0answers
53 views

Estimates for k-d tree partition cells and for Voronoi partition cells

Let us have a $k$-d tree in a unit cube in ${\mathbb R}^k$. Let the points of the tree in the cube be i.i.d. with some density bounded away from 0. How can I estimate the volumes of partition ...
0
votes
0answers
217 views

PCA in each fold of cross-validation for hyper-parameter tuning?

I performed PCA separately on the train set and applied the same transformation to the test set so as to avoid "leakage" of information as mentioned here and here. I'm using XGBoost for a ...
0
votes
1answer
123 views

How to partition set of items into subsets with similar mean, variance and number of elements - looking for some help

Short version: Given I have a list of 120 traits (happy, sad, etc.) evaluated on a number of scales ranging from -5 (for example highly negative) to +5 (for example highly positive), how can I group ...
1
vote
1answer
130 views

Possible explanations for Imputation before train-test split?

I'm working on a real world data set containing missing information. I understand imputing missing values before data partitioning can lead to leakage of information. I'm using this R package MissMech ...
0
votes
0answers
30 views

Statistical method to partition variability

I have a rather simple question, but I am not sure if it can be answered. I have rainfall totals from 1981-2015. I have used these 35 values to calculate an interannual variance. Each year is ...
-1
votes
1answer
44 views

Clustering restaurant dishes

I have data about restaurants, their dishes and customer's orders. Then I want to cluster dishes in restaurant's menu by customer order. So clusters will be dishes in order and criteria of similarity ...
0
votes
2answers
43 views

Cluster non-vector data based on similarity rating

I have a list $S$ of $N$ non-vector elements (in this case words or concepts), and I'd like to partition them into k subsets of similar words. For each pair $(a,b)$ with $a,b \in S$, I have a number ...
0
votes
1answer
49 views

What statistical methods should i use to identify segments in a population according to a response variable?

Assuming I have $p$ uncorrelated explanatory variables $(X_i)_{i=1...p}$ and one binary response variable $Y$, such that : $$\log\left(\dfrac{P(Y=0|X)}{1-P(Y=0|X)}\right)=1_{\{X_1>0\}}\sum_{i=2}^p\...
-1
votes
1answer
33 views

How is graph clustering performed? [closed]

When we do graph partitioning or graph clustering how does one give the graph as input? Is it by means of an adjacency matrix? What if the graph is very large?
2
votes
1answer
211 views

Is it compulsory to partition data in training and testing for time series modelling?

Is it compulsory to partition data in training and testing for time series modelling ? The reason for asking this question: Lets say training data= first 80% observation of actual time series data(...
1
vote
0answers
46 views

Smart sampling to learn a space partitioning

Given a compact feature space, I want to partition the space according to some classification. The first method that comes to mind is to use SVMs. For the classification I have in mind, the space ...
0
votes
0answers
54 views

Valid to supply unique features to each fold of k-fold Cross Validation?

I have a dataset of n = 55 samples with 300,000 - 600,000 features for each sample. I am trying to train a model (ksvm classification) to predict the class of each sample. N = 26 of the samples are ...
0
votes
1answer
760 views

Partitioning with cross validation

I am new to data analytics having only started exploring the field this week. I have downloaded KNIME and am working with a single dataset to try out different classification algorithms. I am ...
0
votes
2answers
41 views

How to combine partitioned dataset to single dataset?

I have been doing some Machine learning research on Microarray dataset. So, I downloaded breast cancer dataset from "http://datam.i2r.a-star.edu.sg/datasets/krbd/BreastCancer". The problem here is , ...
2
votes
0answers
55 views

How do loan companies set interest rate tiers?

What statistical or machine learning methods do companies like Lending Club use to segment their customer base into loan grades A1-G5? What would a reasonable partitioning method look like after ...
1
vote
1answer
387 views

Calculating covariances

Currently I am preparing for my exam. Therefore, we have a sample exam with the answers included. Unfortunately, even with the answers it not all makes sense to me. This is the question with the ...
0
votes
0answers
106 views

Partition Z-Scores Across Multiple Factors in R

Basic question Is there a way to create z-scores across multiple factors quickly? And how would you do this without creating multiple subsets and rebinding them? Details Basically I am comparing ...
1
vote
2answers
350 views

Lower bound for Adjusted Rand Index?

From the original paper, it's not clear whether the adjusted rand index has a lower bound. Does it? If so, what partition yields the bound? If now, how can I construct partitions with arbitrary low ...
0
votes
0answers
52 views

how do you calculate the number of combinations recursive partitioning uses in decision trees?

So I was wondering why random forests limit the levels of a variable to 32 levels, and I found an answer here : R's randomForest can not handle more than 32 levels. What is workaround? My ...
2
votes
2answers
307 views

Dividing a sample based on the value of y would be problematic?

Is it a well known fact in basic econometric that, if one divides a sample based on the value of Y (Dependent variable), it creates a number of problems? My dependent variable is financial rating ...
1
vote
0answers
57 views

Comparison of two different models usig the same CV partition

I am trying to compare two different models with cross validation and I am not sure whether I choosed an effective way to split my dataset into equal sized partitions. My data frame is called ...
1
vote
0answers
64 views

Assigning values to time dependent variables in Train/Test sets

In a dataset with history of customer visits, I have a variable denoting 'Days since last visit'. I have done a stratified train/test split of the data. How should I assign values to this variable? ...
2
votes
1answer
1k views

Dividing data into two groups as similar as possible

In this simplified example, I have five variables for each individual. The data is normalized, with mean 0 and sd 1 in each variable. I would like to divide this data into two groups with equal size ...
5
votes
1answer
123 views

Visualizing differences between alternative clusterings?

Does anyone know of a good way to visualize differences between alternative clusterings? I know there are some measures which can be used to quantify the similarity between two partitionings, e.g.: ...
0
votes
1answer
2k views
2
votes
3answers
946 views

Repeatedly split data in training (0.75) and test (0.25) for cross validation

What kind of cross validation is it called when we randomly split the data into 0.75 training and 0.25 test data set. And this split is done 1000 times.
1
vote
0answers
190 views

How to calculate VPC in glmer?

For a specific analysis I want to calculate the variance partition coefficient (VPC). I am using the following formula: ...
0
votes
2answers
928 views

How to partition leave-one-subject-out (not leave-one-example-out) cross-validation in MATLAB?

I am currently extracting 16 features from 7 samples all of different length. Now I would like to apply the data using multiple classification algorithms with cross validation. I already done this by ...
0
votes
1answer
284 views

Random Forests with modified partitioning criteria

Here is the context of my question : I'm doing binary classification with unbalanced classes. The measure of performance I'd like to maximise is a modified F-measure : $$ F_{\alpha} = \frac{1}{\frac{\...
3
votes
2answers
175 views

Cluster real numbers

I have a set of precise measurements, and what I want to do is count the frequency (how many time it appears) for each value. The problem is that these are very precise measurements and with a naive ...
0
votes
1answer
58 views

How to choose the relative sizes of training and validation sets?

When I work with the methods of data mining, the data is split in training and validations data samples (and sometimes test). I know training + validation = 100%. Which criteria can I use to find a ...
4
votes
3answers
2k views

How to build a decision tree with a constraint on sensitivity?

I am trying to develop a classification model on a sample of people which will discriminate between "Type A" and "Not-Type A" people. Due to external factors, the minimum sensitivity for this ...
0
votes
1answer
47 views

Least Squared Regressions on Partitioned Data

Say I have 800 (X,Y) data points, and I do a LSQ fit and get y = mx+b Then I think to myself, of the 800 data points, 500 are males and 300 are females, so I ...
1
vote
1answer
1k views

What are the differences between Lloyd's, MacQueen's and Hartigan's algorithms for K-Means?

There are three distinct algorithms for the K-Means function in R. These are: Lloyd's MacQueen's Hartigan's I believe I understand how Lloyd's works. 1. The cluster centers are chosen. 2. Points are ...
7
votes
0answers
424 views

What approaches use multiple eigenvectors in graph spectral clustering?

Background: In Newman's PNAS 2006 paper Modularity and community structure in networks, the first eigenvector splits the graph in two clusters, and then each cluster can be further divided by ...
1
vote
0answers
480 views

Is K-Medoids really better at dealing with outliers than K-Means? (with example showing the contrary)

K-Medoids and K-Means are two popular methods of partitional clustering. The consensus is that K-Medoids is better at clustering data when there are outliers (source). This is because it chooses data ...
4
votes
0answers
53 views

Efficient ways to partition rows of augmented design matrix $[X|y]$ into subsets with similar regression results?

Imagine I have $n$ observations on a regression model; are there any reasonably efficient methods for partitioning that into two (or more) roughly equally sized groups which almost reproduce the ...
4
votes
0answers
167 views

Catalan numbers and gambler's ruin paths

Edit I've edited this question several times, and in the end I essentially answered my own question. No need to answer it yourself. I made it slightly less long and rambling, in case anyone else is ...
1
vote
0answers
84 views

Comparing two graphs/markov chains by comparing their clusters

I have an undirected graph representation of my system (a dynamical system), i.e. I have some labelled nodes and bi-directional edge weights, so everything is in a Markov matrix form. Now I can form ...
-1
votes
1answer
54 views

Clustering a list of restaurant dishes

If I have a large list of restaurant dishes that all have the same cuisine... (Pulled Pork, BBQ chicken, 1/2 Ribs, Pork Sliders, Slow Smoked Pork, Full Chicken Special....) What would be a good ...