Questions tagged [spark-mllib]

The Apache Spark distributed machine learning library.

Filter by
Sorted by
Tagged with
1 vote
1 answer
37 views

Wrong time series detection

I have a problem and I need help. I have a time series and I need to know if the data is correct. Let me explain with an example. Suppose I have data generated by an atmospheric pressure sensor. The ...
Rirro Romeu's user avatar
0 votes
0 answers
14 views

Spark Pipeline - Chain Regressors Together

I'm running this on databricks, using python, spark, pipeline, mlflow, etc. Can use whichever library I need to though I have a simple Linear Regression script. I separately have a Random Forest ...
Adam12344's user avatar
  • 527
0 votes
0 answers
16 views

Correlate terms with a scalar value

There's a list of tuples Double -> Set[String]. score tems 0.1 [foo, bar, baz] 0.9 [foo, baz] 0.3 [foo, bar] 0.1 [foo, bar] 0.8 [foo] The size of the ...
Sergey Romanovsky's user avatar
0 votes
1 answer
4k views

What happens if a random forest max bins is set much higher than the number of categorical values?

What happens if a random forest "max bins" hyperparameter is set too high? When training a sparkml random forest with maxBins set roughly equal to the max ...
lampShadesDrifter's user avatar
0 votes
0 answers
222 views

SparkNLP, Tensorflow, Pytorch for NLP

I used Tensorflow, Keras and Pytorch in the past for NLP related works. New company uses Hadoop, which I only have basic concepts. I looked around and found the NLP package associated with the ...
MeiNan Zhu's user avatar
1 vote
1 answer
1k views

xgboost predictions are all 0? [closed]

I ran xgboost4j for classification (in scala-spark), but when I did a sanity check on my predicted values, I got all zeroes. How do I change the threshold? I'm assuming there's a way to map ...
Kashif's user avatar
  • 507
0 votes
0 answers
188 views

How do I interpret K-means model output from Pyspark ml. clustering library?

I am new to PySpark and learning how to build models using PySpark's machine learning libraries. I build a k-means clustering algorithm based on the code of this website. Now, I fed my data into the ...
DataBach's user avatar
  • 155
2 votes
0 answers
125 views

MLeap and Spark ML SQLTransformer [closed]

I have a question. I am trying to serialize a PySpark ML model to mleap. However, the model makes use of the SQLTransformer to do some column-based transformations e.g. adding log-scaled versions of ...
femibyte's user avatar
  • 131
2 votes
0 answers
31 views

If I can put all my data in memory, why do I need frameworks like Spark? [closed]

Just wondering - if my organisation's data never runs into sizes than are bigger than my instances' memory size, why do I need something like Spark? I can scale the memory up using cloud instances, ...
lppier's user avatar
  • 73
1 vote
0 answers
110 views

XGBoost model in Spark --> Missing value treatment [closed]

Unlike python, where missing value is handled internally by the XGBoost algorithm, While building XGBoost model in SPARK, the missing values are implicitly converted to 0.0(float?!). Is this okay ? ...
Anjala Abdurehman's user avatar
2 votes
0 answers
1k views

Logistic Regression Class Imbalance and the use of weighting and undersampling

I have been working on a machine learning model using Spark (binomial) LogisticRegression. The dataset has what I think is a high degree of imbalance - roughly 1% of rows are labelled as events. The ...
user2711693's user avatar
1 vote
1 answer
1k views

Is it possible to have recall and precision of 0 while having an area under PR ~0.5?

As the title suggests, I am running a Random Forest classifier using Scala. To evaluate this classifier (and since I am handling highly imbalanced classes), I used the ...
Toutsos's user avatar
  • 157
0 votes
1 answer
43 views

How can i know that my dataset is being well distributed with K-means?

I'm trying to make an anomaly detection system using Spark Mlib an its K-means implementation but i'm struggling to decide when should i stop searching for K. I'm following Chapter 5 of the Advanced ...
Rubén Figueredo's user avatar
0 votes
0 answers
28 views

Are there specific Machine Learning Algorithms that are more indicated for Real Time Analytics?

As the title suggests, I am wondering if there are specific ML algorithms that are more suitable for real time learning. In my case, I am working on deploying a stacking algorithm on Spark Streaming ...
Odisseo's user avatar
  • 674
1 vote
0 answers
438 views

Differing feature importances after saving/loading GBT Classification model

I am using the GBTClassifier in Spark's ml.classification package. Right after training my model, I save it, and then I grabbed the feature importances, using ...
Green's user avatar
  • 123
1 vote
0 answers
160 views

PR curve and confusion matrix members for multiclass and multilabel classification problems [closed]

I am looking for any libraries which provide out of box support for calculating PR curve and the confusion matrix items(not just count but the items which contributed to the count as well) for ...
sachi1325's user avatar
-1 votes
2 answers
94 views

How to visualise kmeans clustercenters

Asking here as I can't find a tutorial anywhere, and am new to this topic. I've run a kmeans algorithm in spark Scala on some data, and have a prediction object that contains clusterCenters, how can ...
user124123's user avatar
0 votes
1 answer
1k views

Should I expect inertia from a K-Means solution on counts to be substantially lower than for a similar solution on percentages?

During exploratory clustering with K-Means on agents with a range of events, I created two sets of models across clusters with K in {2,..,9}. In one set, the model is fit using raw counts of five ...
MisterJT's user avatar
  • 151
1 vote
0 answers
101 views

Logistic regression on HDFS, what's the algorithm?

How does Spark (or something similar) estimate a logistic regression model, or any statistical model that is estimated by an optimization algorithm, when the data are stored in a distributed ...
Glen's user avatar
  • 7,100
7 votes
1 answer
1k views

Difference Between Linear Regression in Machine Learning and Statistical Model

I had the understanding that the major difference between machine learning and statistical model is, the later "assumes" certain type of distribution of data & based on that different model ...
Beta's user avatar
  • 6,304
1 vote
1 answer
199 views

can collaborative filtering recommendations provide a match percentage metric?

I followed a couple tutorials about matrix factorisation with Spark (https://gerardnico.com/wiki/data_mining/collaborative_filtering one of them). I'm clear that I'm building a dataframe that fills in ...
javaNoober's user avatar
1 vote
0 answers
336 views

How to model preference strength in Spark ALS with implicit feedback?

I am trying to use Spark MLib ALS with implicit feedback for collaborative filtering and I have two questions: according to this paper it seems that I may need to provide both 0- and non-0 preference ...
Motumbo's user avatar
  • 11
1 vote
1 answer
218 views

Positive-Only Training Classification in Spark ML in Scala [closed]

I've been reading the Spark ML documentation, but I can't seem to find any insights into how best to deal with training data that only has positives. In my use-case, I'm attempting to use a list of ...
darkfrog's user avatar
  • 113
0 votes
1 answer
1k views

How to fit an Autoregression model in Spark? [closed]

I'm having a look at the implementation of Autoregression model in Scala https://github.com/sryza/spark-timeseries/blob/master/src/main/scala/com/cloudera/sparkts/models/Autoregression.scala Now if I ...
antonioACR1's user avatar
4 votes
2 answers
4k views

Understanding and interpreting the output of Spark's TF-IDF implementation

I am currently trying to understand what the example code provided as part of Spark's TF-IDF implementation is doing. Given the example code block (taken from Spark's Github repository) ...
Jesús Zazueta's user avatar
3 votes
1 answer
5k views

how to handle sparse data problem in unsupervised learning .i'm going to use k means on data set

how to handle sparse data problem in unsupervised learning .i'm going to use k-means on the dataset. I have 200 variables, nearly in each column have 70% zeros. how can I handle without discarding any ...
Newbie's user avatar
  • 141
-1 votes
1 answer
88 views

Optimize a regression forest (Better parameters and how to obtain them)

I'm currently working on sales forecasting. I'm using a Regression Forest to make my forecast. (with MLLib from Spark on Databricks) I'm trying to find what features are useful in my forecasting. ...
KIToRe's user avatar
  • 25
1 vote
1 answer
1k views

Problem with classification machine learning: prediction is always wrong

I am working on a current project where I need to classify and detect data that doesn't follow the standards and parameters required. I have past data which is the data that I will train my model with....
Guillermo Herrera's user avatar
0 votes
2 answers
2k views

Spark MLLib Gaussian Mixture Model feature or bug

Is this expected from Gaussian Mixture Model? Given a perfectly homogenous dataset, the cluster center is not exactly the same as the data point? ...
Joe Nate's user avatar
0 votes
1 answer
937 views

How to partition data with multiple categorical features?

I have a data set with size of 30k+, and it has several categorical features and also several numerical features. When I try to split the data into training/testing data set, I need to answer a ...
Ryan's user avatar
  • 109
0 votes
1 answer
404 views

Hot Encoding in production

I am using spark to perform random forest classification in a data frame with following structure ...
Anubhav Dikshit's user avatar
1 vote
0 answers
78 views

Expected Value of a Decision Based On Binary Machine Learning Model

I have developed a model to predict the failure of equipment in the field where, in the event of an unplanned failure, the cost of repair and loss of revenue is high. On the other hand, if a ...
dmbaker's user avatar
  • 121
1 vote
1 answer
990 views

Is it good to limit number of instances per node decision tree? Why?

I am running Spark MLLib's Decisioin Tree model. While parameter tuning, I came across the minInstancesPerNode but I am not sure of the implications of setting it too low or too high. Can anyone help ...
disha's user avatar
  • 13
3 votes
1 answer
779 views

Linear Regression in Spark's MLLib gives a seeming incorrect result

I am running the example found here. The training data for the model can be found in this CSV, where the first column is the response variable and the second column is a space separated list of ...
Jon Claus's user avatar
  • 605
2 votes
0 answers
1k views

How to perform data quality check on large number of features using Spark?

I am used to work with manageable number of features. I usually print some descriptive statistics and visualise the histograms of each feature using Python and Pandas or R. I check for outliers and if ...
amrakm's user avatar
  • 21
4 votes
1 answer
2k views

How to estimate most important dimensions of the clusters after performing k-means?

I need to cluster customers of retail shops based on the products that they purchased. Therefore, I need to obtain, as results, both the customers belonging to each cluster and in each cluster the ...
Nko's user avatar
  • 41
7 votes
5 answers
9k views

K-Means Cluster has over 50% of the points in one cluster. How to optimize it?

I am running a clustering algorithm in Spark and I have to choose between K-Means and Bisecting-Kmeans. However the only thing that differes between the two is the runtime because the performance is ...
Mnemosyne's user avatar
  • 173
0 votes
0 answers
693 views

Spark ALS Recommendation engine with prior clustering

I'm trying to develop a recommendation engine, but since the dataset is too big, I'm trying to divide users in clusters and run the ALS recommender on each single cluster. To do this, I'm trying to ...
Vektor88's user avatar
  • 103
-1 votes
1 answer
37 views

What is the Regression algo to Use for this case? [closed]

Having this Data : ...
Nassim MOUALEK's user avatar
3 votes
1 answer
14k views

How to apply word2vec for k-means clustering?

Background: I am new to word2vec. With applying this method, I am trying to form some clusters based on words extracted by ...
mlee_jordan's user avatar
6 votes
2 answers
9k views

How to use Kullback-leibler divergence if mean and standard deviation of of two Gaussian Distribution is provided?

With Apache Spark MLLib library I am trying to find Clusters within a dataset by using Gaussian Mixture Model (number cluster =3) . Now it returns 3 different values of mean and standard deviation. I ...
Avik Dutta's user avatar
2 votes
1 answer
2k views

Interpreting importance of features in logisitic regression model [closed]

I am trying to find the importance of a specific feature or how much impact a specific feature has on a model by looking at feature weights. To my knowledge the feature weights are not scaled in ML ...
Vibhor Agrawal's user avatar
1 vote
1 answer
2k views

What is a corpus in topic modeling?

I am new to machine learning and I am trying to implement LDA using Spark Mllib but I am confused about corpus. What exactly is a corpus of a document? Is it created document-wise or as a whole for ...
amit_kumar's user avatar
1 vote
0 answers
74 views

Title and Occupation Correlation Model

I have a large set of data that contains both a person's title (e.g., "CEO", "Bank Teller", "Fireman", etc.). and occupation codes (e.g, "A1": Managerial, "A2": Laborer, etc.). I need to build an ...
Joel's user avatar
  • 111
2 votes
3 answers
1k views

How should new variables to be added in logistic regression model in production? [duplicate]

We have built a LR model (online SGD ) in Spark. There are more that 15 categorical variables only as independent variables. At run time new values come in few of the columns. We have used ...
Arpit Sisodia's user avatar
5 votes
1 answer
1k views

Given a topic distribution over words from LDA model how to calculate document distribution over topics for new document?

I'm using Spark 1.6.2 via the Python API. It seems that as of when this post is being written, the only data available from the LDA (latent Dirichlet allocation) model calculations is a topic ...
thecity2's user avatar
  • 1,945
0 votes
1 answer
1k views

preprocessing and input format for gradient boosted trees

I'm using MlLib library de Spark and trying to perform Gradient-Boosted Trees algorithm on my data, that has mostly categorical features (and just two numerical features). in the example given in ...
Scana's user avatar
  • 15
5 votes
0 answers
678 views

(Cross) Correlation of time series with very different sampling intervals (sec. vs days)

This is my first post on Cross-Validated. I read a lot of question related to my problem, but no one was completely satisfying. I have two time series that are sampled at very different time ...
McKracken's user avatar
  • 161
1 vote
1 answer
241 views

Does PCA do something else apart from selecting features with the most variance?

While experimenting with Spark library MlLib, I questioned myself if I understood well the mechanism of PCA algorithm, because output of MlLib algorithm was not what I expected to get. so for given ...
Scana's user avatar
  • 15
1 vote
1 answer
3k views

true negative is 0% whereas true positive is 100% correctly classified

I used Naive Bayes from Spark's MlLib to train a model and test it on the data (in the form of an RDD). The results were confusing. the data and results are as follows: The problem is a binary ...
Preetham Madeti's user avatar