Pretty much every learning algorithm I know could handle a dataset with 10,000 rows. Random Forests, SVMs, boosted trees, penalized linear models, knn, etc. etc. I've fit all of these models on datasets of that size on a pretty standard laptop (4GB ram, core i5 processor).
You can start to hit computational limits on a dataset of that size when you start re-sampling and cross-validating to avoid over-fitting.
I guess the answer to your question is "How much time do you have?" Maybe waiting a few hours to cross-validate a 10x10 grid of parameters is more than you can spare.
Here's some example code in R. Input is a random matrix with 5 columns and 2000 rows (10000 values). Output is a binary vector with 2000 values. I tried a random forest, an SVM, a penalized linear model, a KNN model, and a neural network. All the models fit the entire dataset in under 1.5 seconds. Tuning the models using bootstrapped re-samples took up to ~2 minutes each (although some models were quicker).
Here's the code:
#Setup
rm(list = ls(all = TRUE))
set.seed(1)
#Generate an input matrix with 10,000 values
#2,000 rows, 5 columns
X <- data.frame(replicate(5, rnorm(2000)))
#Construct Y using X
Y <- runif(1)*X[,1]*X[,2]^2+runif(1)*X[,3]/X[,4]
#Convert Y to binary
Y <- as.factor(ifelse(sign(Y)>0,'X1','X0'))
#Create bootstap samples for fitting models
library(caret)
tmp <- createResample(Y,times = 25)
myCtrl <- trainControl(method = "boot", index = tmp, timingSamps = 10)
#Fit models
RFmodel <- train(X,Y,method='rf',trControl=myCtrl,tuneLength=1) #Random Forest
SVMmodel <- train(X,Y,method='svmRadial',trControl=myCtrl,tuneLength=3) #Support Vector Machine
GLMmodel <- train(X,Y,method='glmnet',trControl=myCtrl,tuneLength=10) #Penalized linear model
KNNmodel <- train(X,Y,method='knn',trControl=myCtrl,tuneLength=10) #Nearest-neighbors
NNmodel <- train(X,Y,method='nnet',trControl=myCtrl,tuneLength=3, trace = FALSE) #Neural network
#Assess re-sampled (out of sample) accuracy
resamps <- resamples(
list( RF = RFmodel,
SVM = SVMmodel,
GLMnet = GLMmodel,
KNN = KNNmodel,
NN = NNmodel
))
summary(resamps)
dotplot(resamps, metric = "Accuracy")
#Assess run-time (in seconds)
resamps$timings
And here's the results:
Accuracy :
Min. 1st Qu. Median Mean 3rd Qu. Max.
RF 0.8972 0.9062 0.9261 0.9205 0.9280 0.9490
SVM 0.8313 0.8437 0.8527 0.8546 0.8634 0.8800
GLMnet 0.6613 0.6808 0.6862 0.6918 0.6995 0.7295
KNN 0.8158 0.8244 0.8344 0.8368 0.8492 0.8661
NN 0.7592 0.7943 0.8003 0.8047 0.8231 0.8352
Run Time (seconds):
Everything FinalModel Prediction
RF 33.80 1.28 0.03
SVM 110.76 1.17 0.01
GLMnet 104.21 0.05 0.01
KNN 17.09 0.00 0.02
NN 76.51 0.51 0.00
As a graph:
