27
$\begingroup$

Suppose I have $n$ independent sources, $X_1, X_2, ..., X_n$ and I observe $m$ convex mixtures: \begin{align} Y_1 &= a_{11}X_1 + a_{12}X_2 + \cdots + a_{1n}X_n\\ ...&\\ Y_m &= a_{m1}X_1 + a_{m2}X_2 + \cdots + a_{mn}X_n \end{align}

with $\sum_j a_{ij} = 1$ for all $i$ and $a_{ij} \ge 0$ for all $i,j$.

What's the state of the art in recovering $X$ from $Y$?

PCA is out of the question because I need the components to be identifiable. I've looked at ICA and NMF - I can't find any way to impose nonnegativity of the mixing coefficients for ICA, and NMF doesn't seem to maximize independence.

$\endgroup$
14
  • 3
    $\begingroup$ I would think that this should be called "nonnegative independent component analysis", but it seems that this name has been used for ICA with the nonnegativity constraint on the sources $X$, not on the mixing matrix $A$ (eecs.qmul.ac.uk/~markp/2003/Plumbley03-algorithms-c.pdf). So this does not apply to your case. Interesting question. $\endgroup$
    – amoeba
    Commented Dec 29, 2014 at 22:51
  • $\begingroup$ Don't you want the sums to run over j instead of i? Can you assume the sources are approximately gaussian? if they are unimodal and have sufficiently fast decay, it is possible that fitting a GMM would suffice. $\endgroup$
    – Yair Daon
    Commented Jan 6, 2015 at 22:16
  • 1
    $\begingroup$ What ICA algorithms have you tried? I'm a little rusty, but think the non-negativity assumption of the mixing coefficients can be imposed in some algorithms that assume certain models for the signals like the Weights-Adjusted Second Order Blind Identification (WASOBI) algorithm, as it assumes you can model the signals as AR processes and, thus, you can impose conditions in the coefficients. $\endgroup$
    – Néstor
    Commented Jan 10, 2015 at 23:10
  • 1
    $\begingroup$ The sources are all supported on the set {1,2,...,96} $\endgroup$
    – anonymous
    Commented Jan 12, 2015 at 17:13
  • 1
    $\begingroup$ This is not unique. link.springer.com/article/10.1007/s11634-014-0192-4 Mixture distributions are ubiquitous and also problematic for those organic applications that I have examined. I have yet to see a circumstance in which they are preferred over dependent processes like convolutions of components upon testing. $\endgroup$
    – Carl
    Commented Feb 27, 2018 at 3:17

4 Answers 4

1
$\begingroup$

What you are looking for is Non-negative Independent Component Analysis (nICA). nICA is a variant of ICA that incorporates non-negativity constraints, hence it maintains the desirable properties of ICA (maximization of statistical independence of the reconstructed sources) while also imposing non-negativity of the mixing coefficients. If would like to have both nonnegativity constraints and linear equality constraints on the mixing coefficients then you would need to use a quadratic programming solver (e.g. osqp) to calculate the updates, though I haven't seen an off-the-shelf implementation. See this article and references therein and Chapter 11 in this book. Here is a Matlab implementation, and here is a Python one of nICA, which would be a good start and with some modification could be made to also allow for linear equality constraints.

$\endgroup$
1
$\begingroup$

This is just ICA with a constraint on the mixing matrix. Look up an ICA book or survey paper for cost functions associated with ICA, using fixed or adaptive source distributions depending on the distribution of the components. Then add a constraint that the rows of the mixing matrix satisfy the equality and inequality constraints, and use the update rule as the direction in a non-linear programming routine.

$\endgroup$
3
  • $\begingroup$ This is correct, but the OP would be better helped if you could give a good link to "an ICA book or survey paper for cost functions associated with ICA"... You also need to be a bit more specific than "and derive update rule using an optimization method". What optimization method in this case? Quadratic programming? $\endgroup$ Commented Jun 19, 2023 at 5:38
  • 1
    $\begingroup$ The ICA book by Hyvarinen would be a good start. The update rules for Extended Infomax or FastICA, formulated in terms of the mixing matrix rather than the unmixing matrix, represent a direction related to the gradient, for the vectorized mixing matrix. This can be used in a standard non-linear programming routine with equality and inequality constraints. The cost function, which is basically a log likelihood, has to be formulated as well. The Hyvarinen chapter on Maximum Likelihood methods should provide a likelihood function and corresponding (natural) gradient update. $\endgroup$
    – japalmer
    Commented Jun 20, 2023 at 6:10
  • 1
    $\begingroup$ I think the BSS book by Cichocki has natural gradient updates for the mixing matrix if the Hyrvarinen book doesn't. $\endgroup$
    – japalmer
    Commented Jun 20, 2023 at 7:14
-1
$\begingroup$

It could be achieved by using an exponential non-linearity instead of the typical/default tanh(), if X is also non-negative.

Formula 40 in https://www.cs.helsinki.fi/u/ahyvarin/papers/NN00new.pdf and available in most implementations.

E.g. in sklearn simply use fun='exp' https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.FastICA.html

$\endgroup$
2
  • 3
    $\begingroup$ Welcome to Stats.SE. Can you please edit your answer and expand it in order to explain the key steps of the links you provide? This way, the information is searchable in here (and sometimes links break). You may want to take a look at some formatting help. While you're at it, you can use LaTeX / MathJax. $\endgroup$ Commented May 11, 2019 at 16:17
  • $\begingroup$ I don't think this would work - with transformation one also wouldn't have independence on the original scale - typically one would like hard nonnegativity constraints in problems like this... Nonnegative independent component analysis is what you would need here, see Chapter 11, google.be/books/edition/Blind_Source_Separation/…. And my answer below. $\endgroup$ Commented Jun 17, 2023 at 10:37
-2
$\begingroup$

Transformation of variables is a good option when that linearizes the problem. That procedure can be used to increase the correlations, reduce the residuals, and decrease the number of parameters needed to produce a good fit to the data.

For example, $\ln Y=a_0+a_1\ln X_1+a_2\ln X_2\to Y=e^{a_0}X_1^{a_1}X_2^{a_2},$ might be vastly superior to $Y=a_0+a_1X_1+a_2X_2$. A hint as to what to do is often provided by examination of the data or its residuals. For example, if one has fan shaped heteroscedasticity of parameters, like in this paper enter image description here

That particular type of log-log transform may be of interest. More generally, there are lots of transforms to consider: taking square roots, exponentiation, taking reciprocals, and so forth. Another indication of how one should treat the data comes from considering the physics of the problem. For example, if a regression problem cannot take negative $Y$-values, that problem may not be linear, as lines may become negative.

$\endgroup$
8
  • $\begingroup$ I don't think this would work - one also wouldn't have independence on the original scale - typically one would like hard nonnegativity constraints in problems like this... Nonnegative independent component analysis is what you would need here, see Chapter 11, google.be/books/edition/Blind_Source_Separation/…. And my answer below. $\endgroup$ Commented Jun 17, 2023 at 10:36
  • $\begingroup$ @TomWenseleers Independence on the original scale is irrelevant when the original scale is unmotivated. The data above are mass (W) and volume (V) which are definite positive AND are inappropriately dimensioned to be properly related on the original scale. Frankly, I don't follow what you are saying, it doesn't compute. $\endgroup$
    – Carl
    Commented Jun 17, 2023 at 17:29
  • $\begingroup$ Well that in this example the aim is to recover a set of orthogonal sources X_i. But what was required was orthogonality on the original untransformed scale X_i, not orthogonality on some log-log scale. And that's typically tackled using nonnegativity constrained independent component analysis (nnICA). Also, the aim is to decompose the signal in additive components, but a log transforms would decompose the signal in multiplicative components, which is not what is asked for here... $\endgroup$ Commented Jun 17, 2023 at 17:36
  • $\begingroup$ You would be claiming that a nonnegative matrix factorization would be equivalent to an unconstrained matrix factorization after log transform. But that's not the case as the latter would be a matrix decomposition into multiplicative rather than additive components. Plus we need the orthogonality constraints, so that would be nonnegative independent component analysis... $\endgroup$ Commented Jun 17, 2023 at 17:41
  • $\begingroup$ OK I see, but that's really not the question here... The question is how to deparate a number of independent sources for the cases where the observed signal is a nonnegative linear (convex) combination of the sources subject to the sources being independent of each other. Nonnegative independent component analysis would solve that problem. $\endgroup$ Commented Jun 17, 2023 at 18:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.