# Difference between Factorization machines and Matrix Factorization?

I came across the term Factorization Machines in recommender systems. I know what Matrix Factorization is for recommender systems but never heard of Factorization Machines. So what's the difference?

Matrix factorization is a method to, well, factorize matrices. It does one job of decomposing a matrix into two matrices such that their product closely matches the original matrix.

But Factorization Machines are quite general in nature compared to Matrix Factorization. The problem formulation itself is very different. It is formulated as a linear model, with interactions between features as additional parameters. This feature interaction is done in their latent space representation instead of their plain format. So along with the feature interactions like in Matrix Factorization, it also takes the linear weights of different features.

So compared to Matrix Factorization, here are key differences:

1. In recommender systems, where Matrix Factorization is generally used, we cannot use side-features. For a movie recommendation system, we cannot use the movie genres, its language etc in Matrix Factorization. The factorization itself has to learn these from the existing interactions. But we can pass this info in Factorization Machines.
2. Factorization Machines can also be used for other prediction tasks such as Regression and Binary Classification. This is usually not the case with Matrix Factorization

The paper shared in previous answer is the original paper that talks about FMs. It has a great illustrative example too as to what FM exactly is.

Edit: A note on side features that can be used in Factorization Machines but not Matrix factorization:

Matrix Factorization is solely a collaborative filtering approach which needs user engagement on the items. So it doesn't work for what is called "cold start" problems. Think of a new movie released on Netflix. As no one would have watched it, matrix factorization doesn't work for it. But as Netflix would know the genre, actors, director etc, Factorization Machine can kick-start the recommendations for this movie from day 1 itself, which is a crucial component for many websites that use recommendation systems.

Just some extension to Dileep's answer.

If the only features involved are two categorical variables (e.g., users and items), then the (nature of the interaction terms of) FM is equivalent to the matrix factorization model. But FM can be easily applied to more than two real-valued features.

• what do you mean by "equivalent"? would they really have the same model equation in that case? Commented Sep 13, 2020 at 3:30
• @dontloo: can you elaborate more on the "equivalence" please? Commented Aug 24, 2021 at 16:30
• @ZaidGharaybeh I made an edition and wonder if the equivalent works or not now? Commented Apr 27, 2022 at 15:04

Matrix factorization is a different factorization model. From the article about FM:

There are many different factorization models like matrix factorization, parallel factor analysis or specialized models like SVD++, PITF or FPMC. The drawback of these models is that they are not applicable for general prediction tasks, but work only with special input data. Furthermore their model equations and optimization algorithms are derived individually for each task. We show that FMs can mimic these models just by specifying the input data (i.e. the feature vectors). This makes FMs easily applicable even for users without expert knowledge in factorization models.

From libfm.org:

"Factorization machines (FM) are a generic approach that allows to mimic most factorization models by feature engineering. This way, factorization machines combine the generality of feature engineering with the superiority of factorization models in estimating interactions between categorical variables of large domain."

Let me take a force-march through a simple item-user example below, where there are only two categorical variables for the user and item scenario, and hence both matrix factorization and factorization machines work(inspired by @dontloo's answer).

Let's say we have two users: $$u_1$$ and $$u_2$$, and two items: $$i_1$$ and $$i_2$$. We initialize two vectors with very low dimension of 2 for the two users($$u_1$$ and $$u_2$$ respectively): [$$v_{11}$$, $$v_{12}$$], [$$v_{21}$$, $$v_{22}$$], and two low dimension vectors for the two items$$i_1$$ and $$i_2$$ respectively: [$$w_{11}$$, $$w_{12}$$], [$$w_{21}$$, $$w_{22}$$].

And our observations can be represented as the following matrix:

The three values, namely 2, 4, and 1, are ratings of $$u_1$$ for $$i_1$$, $$u_1$$ for $$i_2$$, and $$u_2$$ for $$i_2$$.

To train a model and predict using matrix factorization, we do the following steps:

1. preparing training data using the three cases:
a. multiply the vector of $$u_1$$ and $$i_1$$ for the label rating 2: [$$v_{11}$$, $$v_{12}$$] $$\cdot$$ [$$w_{11}$$, $$w_{12}$$] = 2
b. multiply the vector of $$u_1$$ and $$i_2$$ for the label rating 4: [$$v_{11}$$, $$v_{12}$$] $$\cdot$$ [$$w_{21}$$, $$w_{22}$$] = 4
c. multiply the vector of $$u_2$$ and $$i_2$$ for the label rating 1: [$$v_{21}$$, $$v_{22}$$] $$\cdot$$ [$$w_{21}$$, $$w_{22}$$] = 1

2. use Stochastic Gradient Descent(SGD) or Weighted Alternating Least Squares(WALS) (with some regularization methods) to get the vectors for $$u_1$$, $$u_2$$, $$i_1$$, and $$i_2$$, by minimizing the true results of the three dot productions and their corresponding labels: 2, 4, 1.

3. use the trained vectors of $$u_2$$ and $$i_1$$ to predict the missing value in the matrix in the above image: [$$v_{21}$$, $$v_{22}$$] $$\cdot$$ [$$w_{11}$$, $$w_{12}$$]

However the process of training and predicting differs for factorization machines:

1. We kind of flatten the users and items to make the training data matrix.
a. we add four parameters(which should be learned like the w's and v's in the matrix factorization example) for the first order feature combinations($$u_1$$, $$u_2$$, $$i_1$$, and $$i_2$$): $$k_{u1}$$, $$k_{u2}$$, $$k_{i1}$$, and $$k_{i2}$$, and then we multiply the four parameters with their values for the three observations:
$$\text{ }$$i. For label 2, it relates only two $$u_1$$ and $$i_1$$: $$k_{u1}$$ * 1 + $$k_{u2}$$ * 0 + $$k_{i1}$$ * 1 + $$k_{i2}$$ * 0； 1 for relating to the item or user and 0 otherwise
$$\text{ }$$ii. For label 4, it relates only two $$u_1$$ and $$i_2$$: $$k_{u1}$$ * 1 + $$k_{u2}$$ * 0 + $$k_{i1}$$ * 0 + $$k_{i2}$$ * 1； 1 for relating to the item or user and 0 otherwise
$$\text{ }$$iii. For label 1, it relates only two $$u_2$$ and $$i_2$$: $$k_{u1}$$ * 0 + $$k_{u2}$$ * 1 + $$k_{i1}$$ * 0 + $$k_{i2}$$ * 1； 1 for relating to the item or user and 0 otherwise
b. To deal with the second-order feature combinations, we multiply the vectors for each user and item for the three observations: $$u_1$$ and $$i_1$$ for 2, $$u_1$$ and $$i_2$$ for 4, and $$u_2$$ and $$i_2$$ for 1; in this case, we don't need three additional parameters, we use the dot product of each user and item vector pairs:
$$\text{ }$$i. $$u_1$$ and $$i_1$$: [$$v_{11}$$, $$v_{12}$$] $$\cdot$$ [$$w_{11}$$, $$w_{12}$$] * 1 + [$$v_{11}$$, $$v_{12}$$] $$\cdot$$ [$$w_{21}$$, $$w_{22}$$] * 0 + [$$v_{21}$$, $$v_{22}$$] $$\cdot$$ [$$w_{11}$$, $$w_{12}$$] * 0 + [$$v_{21}$$, $$v_{22}$$] $$\cdot$$ [$$w_{21}$$, $$w_{22}$$] * 0
$$\text{ }$$ii. $$u_1$$ and $$i_2$$: [$$v_{11}$$, $$v_{12}$$] $$\cdot$$ [$$w_{11}$$, $$w_{12}$$] * 0 + [$$v_{11}$$, $$v_{12}$$] $$\cdot$$ [$$w_{21}$$, $$w_{22}$$] * 1 + [$$v_{21}$$, $$v_{22}$$] $$\cdot$$ [$$w_{11}$$, $$w_{12}$$] * 0 + [$$v_{21}$$, $$v_{22}$$] $$\cdot$$ [$$w_{21}$$, $$w_{22}$$] * 0
$$\text{ }$$iii. $$u_2$$ and $$i_2$$: [$$v_{11}$$, $$v_{12}$$] $$\cdot$$ [$$w_{11}$$, $$w_{12}$$] * 0 + [$$v_{11}$$, $$v_{12}$$] $$\cdot$$ [$$w_{21}$$, $$w_{22}$$] * 0 + [$$v_{21}$$, $$v_{22}$$] $$\cdot$$ [$$w_{11}$$, $$w_{12}$$] * 0 + [$$v_{21}$$, $$v_{22}$$] $$\cdot$$ [$$w_{21}$$, $$w_{22}$$] * 1
c. add the two above terms as the final predicted value for each observation.
$$\text{ }$$i. $$u_1$$ and $$i_1$$ for 2: $$k_{u1}$$ * 1 + $$k_{u2}$$ * 0 + $$k_{i1}$$ * 1 + $$k_{i2}$$ * 0 + [$$v_{11}$$, $$v_{12}$$] $$\cdot$$ [$$w_{11}$$, $$w_{12}$$] * 1 + [$$v_{11}$$, $$v_{12}$$] $$\cdot$$ [$$w_{21}$$, $$w_{22}$$] * 0 + [$$v_{21}$$, $$v_{22}$$] $$\cdot$$ [$$w_{11}$$, $$w_{12}$$] * 0 + [$$v_{21}$$, $$v_{22}$$] $$\cdot$$ [$$w_{21}$$, $$w_{22}$$] * 0 = 2
$$\text{ }$$ii. $$u_1$$ and $$i_2$$ for 4: $$k_{u1}$$ * 1 + $$k_{u2}$$ * 0 + $$k_{i1}$$ * 0 + $$k_{i2}$$ * 1 + [$$v_{11}$$, $$v_{12}$$] $$\cdot$$ [$$w_{11}$$, $$w_{12}$$] * 0 + [$$v_{11}$$, $$v_{12}$$] $$\cdot$$ [$$w_{21}$$, $$w_{22}$$] * 1 + [$$v_{21}$$, $$v_{22}$$] $$\cdot$$ [$$w_{11}$$, $$w_{12}$$] * 0 + [$$v_{21}$$, $$v_{22}$$] $$\cdot$$ [$$w_{21}$$, $$w_{22}$$] * 0 = 4
$$\text{ }$$iii. $$u_2$$ and $$i_2$$ for 1: $$k_{u1}$$ * 0 + $$k_{u2}$$ * 1 + $$k_{i1}$$ * 0 + $$k_{i2}$$ * 1 + [$$v_{11}$$, $$v_{12}$$] $$\cdot$$ [$$w_{11}$$, $$w_{12}$$] * 0 + [$$v_{11}$$, $$v_{12}$$] $$\cdot$$ [$$w_{21}$$, $$w_{22}$$] * 0 + [$$v_{21}$$, $$v_{22}$$] $$\cdot$$ [$$w_{11}$$, $$w_{12}$$] * 0 + [$$v_{21}$$, $$v_{22}$$] $$\cdot$$ [$$w_{21}$$, $$w_{22}$$] * 1 = 1

2. use Stochastic Gradient Descent(SGD) (with some regularization methods) to get the values or vectors for $$k_{u1}$$, $$k_{u2}$$, $$k_{i1}$$, $$k_{i2}$$, $$u_1$$, $$u_2$$, $$i_1$$, and $$i_2$$, by minimizing the true results of the three predictions in 1.c and their corresponding labels: 2, 4, 1.

3. use the trained values and vectors of $$u_2$$ and $$i_1$$ to predict the missing value in the matrix in the above image: $$k_{u1}$$ * 0 + $$k_{u2}$$ * 1 + $$k_{i1}$$ * 1 + $$k_{i2}$$ * 0 + [$$v_{11}$$, $$v_{12}$$] $$\cdot$$ [$$w_{11}$$, $$w_{12}$$] * 0 + [$$v_{11}$$, $$v_{12}$$] $$\cdot$$ [$$w_{21}$$, $$w_{22}$$] * 0 + [$$v_{21}$$, $$v_{22}$$] $$\cdot$$ [$$w_{11}$$, $$w_{12}$$] * 1 + [$$v_{21}$$, $$v_{22}$$] $$\cdot$$ [$$w_{21}$$, $$w_{22}$$] * 0

The above are just two toy examples, and in real problems there would be much more users and much more items(and more additional features for the factorization machines) and hence much more observations, making the ratio of parameter to observation much lower. Usually we would try models with less parameters by reducing the vector size for the users and items or perhaps other additional features(for FM).

And the connection between FM and MC lies in that they both use dot product of two vectors to reduce the number of parameters in modeling: $$u \cdot i$$

References: