How to detect and remove the duplicate samples Suppose $X = \{(x_1,y_1),\cdots,(x_N,y_N)\}$ is our samples. Now if I currently copy paste $X$ one time, then our sample data will become $(X,X).$ And we know that the half of samples, the copy pasted part is meanless. So in pre-process of data, how to detect and deal with such samples?
Since we know that such samples will have the impact. For example linear regression, copy pasting the sample will not affect the coefficient of estimation and R-squared, however it will affect the estimation variance and T test.
 A: Usually, you have to identify if the duplication is intended or at least an unintended artifact of a systematic procedure. Duplication can arise also naturally. One example would be data truncation. Suppose that you measure some lengths and truncate the result to centimeters. Human heights truncated to centimeters will provide enough duplicates. What I usually do, as part of data cleaning before approaching a problem is to inspect the data available for systemic duplicates. It is hard to decide sometimes, but one can follow various lines of reasoning and some cases are clear. If I detect systemic duplication I simply remove it, otherwise, I leave it there because I destroy the structure of the signal. It is a similar situation with missing data, where filling missing values could be different for random missing values and systemic one.
A: To detect (near-)duplicates, you can compute nearest-neighbour distances.  If you have a lot of variables this could be slow; one speed-up is to do it for a fixed or random subsample of variables first.
(A speed-up with better theoretical guarantees is to take a random low-dimensional projection and compute nearest-neighbours for the projected variables, but that's more complicated)
