What machine Learning Algorithm can I use for stemming What are some ways of implementing machine learning into stemming/lemmatizing tasks? While well-known algorithms like Porter Stemmer exist, I wanted to know if there was any means of implementing it as a machine learning task.
 A: People tend not to take an ML approach to stemming, because it’s meant to be a cheap and quick hack for text normalization. Why throw a complex model at something that can be handled by a few well-chosen string operations?
In other words, you probably want to stick to the Porter or Snowball stemmers. Not everything requires (or benefits) from ML. But for completeness, here are other options.
Lemmatization
By contrast, lemmatization is well defined and principled. The goal is to convert words into their citation form (“lemma”).
This is better defined cross-lingually than stemming is. Both are forms of text normalization. But (in part because this also has value to computational linguists) lemmatization garners more attention.
Modeling
Either one is easy to operationalize, if you have a training set of (word, normalized) pairs. You can use any off-the-shelf model of $p(y \mid x)$ where $x$ and $y$ are string-valued random variables. You then seek the argmax of that expression over all $y$s for your word $x$.
Two major directions for this string-to-string modeling are statistical modeling using Bayes’s rule (e.g. EGYPT, GIZA++) and neural network “sequence-to-sequence” models that model the direct probability. These approaches both stemmed from the machine translation literature, but they can apply to stemming and lemmatization if you treat the input and output “sentences” as the sequence of letters in your input and output words.
A: Given a decent dictionary containing for a few thousand terms the stemmed or lemmatized version, you can transform your textual data into a bag of words and then use k nearest neighbors based on a string distance such as Levensthein or cosine.
Lemmatization requires certainly many more training data points due to the fact that short terms such as "good" and "better" sharing the same lemma are not easily detected based on the sole string distance.
This resource discusses stemming and lemmatization algorithms more closely. That said, I understand your use case as a purely experimental one (e.g. for an uncommon language), given the efficiency of existing algorithms.
