33

Chemical concentration data often have zeros, but these do not represent zero values: they are codes that variously (and confusingly) represent both nondetects (the measurement indicated, with a high degree of likelihood, that the analyte was not present) and "unquantified" values (the measurement detected the analyte but could not produce a reliable numeric ...


27

As the zeros merely indicate concentrations below the detection limit, maybe setting them to (detection limit)/2 would be appropriate I was just typing that the thing that comes to my mind where log does (frequently) make sense and 0 may occur are concentrations when you did the 2nd edit. As you say, for measured concentrations the 0 just means "I couldn't ...


6

Given a $n$ by $p$ matrix $\pmb X$ the SVD decomposition of $\pmb X$ is: $$\text{svd}((\pmb X-\bar{x})/\sqrt{n-1})=\pmb{UDV}'$$ (I will denote $\pmb V_k$ the matrix formed of the first $k$ columns of $\pmb V$ and $\pmb D_k$ the diagonal matrix formed of the first $k$ rows and columns of $\pmb D$) The SVD decomposition divides the total variance of $\pmb X$...


6

OP asks about values below "limit of detection" in a quantitation (regression) context. Limit of detection, however is a method performance characteristic that refers to qualitative rather than quantitative tasks. This is why my original answer refers to the lower limit of qantitation (quantification) instead. See e.g. Currie, Pure&Appl. Chem., 67, ...


5

I'm mostly using the papers Paul Geladi and Bruce R. Kowalski: Partial least-squares regression: a tutorial, Analytica Chimica Acta, 185, 1-17 (1986). DOI: 10.1016/0003-2670(86)80028-9 and Mevik, B.-H. & Wehrens, R.: The pls Package: Principal Component and Partial Least Squares Regression in R, Journal of Statistical Software, 18, 1 - 24 (2007). DOI: ...


5

@miura I came across this article by Bill Gould on the Stata blog (I think he actually founded Stata) which I think could provide help with your analysis. Near the end of the article he cautions against the use of arbitrary numbers that are close to zero, such as 0.01, 0.0001, 0.0000001, and 0 since in logs they are -4.61, -9.21, -16.12, and $-\infty$. In ...


5

There are mainly two algorithms for PLSR namely NIPALS and SIMPLS. SIMPLS algorithm is generally faster yet numerically less stable(in most cases the difference is very small). The original article of SIMPLS provides the steps which starts with mean centering both X and Y. The maintainer of the package probably relies on these steps. However, directly ...


4

You are doing a (chemical) calibration, and the search phrase you are looking for is method validation in analytical chemistry. There actually exist norms how to validate methods in analytical chemistry, and certain measures of performance like limit of detection (LOD), limit of quantitation (LOQ, probably more relevant for you), recovery rate, etc. If you ...


4

To clarify how to deal with the log of zero in regression models, we have written a pedagogical paper explaining the best solution and the common mistakes people make in practice. We also came out with a new solution to tackle this issue. You can find the paper by clicking here: https://ssrn.com/abstract=3444996 First, we think that ones should wonder why ...


4

A few people will think this question is rather broad but I think it is answerable. It does however require you to get a bit more info from your friend about what he's after because statistics intersects chemistry in a multitude of places. If it's physical chemistry type stuff he's after, I'd start with Statistical Thermodynamics (wikipedia). Also G.S. ...


3

You can set the zeros of the $i^{th}$ variable to the ${\rm mean}(x_i) - n\times{\rm stddev}(x_i)$ where $n$ is large enough to distinguish these cases from the rest (e.g., 6 or 10). Note that any such artificial setup will affect your analyses so you should be careful with your interpretation and in some cases discard these cases to avoid artifacts. ...


3

It is a matter of generating features or variables that describe the SMILE representation of a chemical compound. Computational chemistry has proposed good definitions of different chemical descriptors that can be trasnformed into fingerprints. These fingerprints, are vectors of numbers (binary or real) that gives a description of the chemical compound based ...


3

I did not yet understand what the task at hand is. But I'll try to start towards an answer. the original input is a defined quantity of material and the final output is a number and what I would like to know is how accurately the output reflects the input. This is part of the validation of the analytical method. This feels like some kind of ...


3

Missing values As @gung says, a first step would be to go and find out why there are NAs: concentrations are below LOD/LOQ/signal below critical value: ask for the uncensored values. At the very least make sure that in future data is not censored and instead you are provided with the critical values/LOD/LOQ in addition to the data. Btw: known LOD and LOQ ...


2

Ignore region External knowledge such as the fact that your instrumental set-up cannot measure reliably outside certain spectral ranges, or you're working on a substrate/in a solvent that renders certain regions unsusable, or that for the type of sample you have, no bands at all can appear in certain regions is very valuable knowledge. PLS is fairly good ...


2

First of all, deterministic (i.e. the same computation always yields the same result) does not imply error-free (not even free of random error!). That being said, both ways of modeling, $y = f (x)$ as well as $x = f (y)$ are used in chemometrics. The former is known as ordinary or classical regression or calibration, the latter is referred to as inverse ...


2

(Hi and welcome to cross validated) You raise very good questions: the outcome of the experiments may depend on a large number of factors. Doing really independent replicates of the experiment is often not feasible from a practical point of view (ordering multiple times the same cell line, etc.). However, here are some things you can do: Report ...


2

Welcome to cross validated. First of all, "below limit of quantitation" does not mean the quantity is unmeasurable, it just means that the relative error at those quantities/concentrations is > 10 % (unless another relative error was specified - but that should then be marked very clearly). Typically, quantities below the limit of quantitation are ...


2

You really found a bug, which was immediately fixed by the maintainer (version 1.4.1 on CRAN - available already as .tar.gz, win binaries will take a bit longer): library (chemometrics) library (pls) train <- yarn [yarn$train,] res <- pls1_nipals(train$NIR, train$density, a = 2) x.centered <- scale (test$NIR, center = colMeans (train$NIR), scale =...


2

First of all, the methods you are looking for are called multiway methods in chemometrics, and there is a whole lot of literature on this. Geladi: Analysis of multi-way (multi-mode) data, Chemometrics and Intelligent Laboratory Systems, 7, (1989) 11-30. may be a good starting point to learn about these methods. As @theGD says, you can unfold the N-way ...


2

As @whuber already pointed out, the only way here lies in chemical assumptions. You'd need two assumptions: Can you sensibly assume that all the lipid signals have the same sensitivity (note that sensitivity implies a linear response in general). As you normalize your total counts over the whole spectrum, you also need to be sure that the matrix is either ...


2

I'm analytical chemist specialised in spectroscopy and chemometrics. Chemometrics is statistics for chemical questions/tasks/problems (similar to psychometrics, biometrics, etc.) and definitively a search term your friend should check out. works on the confluence of statistics and chemistry, of which he could not find many articles online. This may be ...


2

As @theGD already pointed out in the comment, scaling is often not needed for spectroscopic data as the features already have a common intensity axis. Here's my guess what's happening when you scale: You have spectra with very nice zero baselines. In other words, all those features outside your analyte signal are constant mean + some noise. If you scale ...


1

As you say, after mean-centering no further intercept term is needed, as the data is guaranteed to go through the origin. In that sense the center is the intercept term of the model and the standard PLS models do have $x$ and $y$ intercept terms. That being said, there are reasons for centering the data on some point that is not the mean of $\mathbf X$ (and $...


1

Any conentration value expressed as ppm means some micrograms of any component divided by one gram of the sample. So, if the analysis were complete the addtion of all concentrations wii be one gram in all samples. Then, you have compositional data


1

Possibly this article (which is online readable) is also helpful. In the following the abstract: Abstract The maximum likelihood PCA (MLPCA) method has been devised in chemometrics as a generalization of the well-known PCA method in order to derive consistent estimators in the presence of errors with known error distribution. For similar reasons, ...


1

The RMSE should give a good indication of how well the two simulations match each other, and if you want to compare the results from different categories you could somehow normalize the RMSE value. A simple way to do this might be by dividing each RMSE by the mean of the given category. Wikipedia also has an entry on this. I have often used this to compare ...


1

It is called a "finite mixture model". Here is the first one, by Pearson. http://ms.mcmaster.ca/peter/mix/demex/excrabs.html http://blog.mrtz.org/2014/04/22/pearsons-polynomial.html Some general references https://en.wikipedia.org/wiki/Mixture_model http://repec.org/snasug08/deb_fmm_slides.pdf Here are libraries for handling it: https://cran.r-...


1

Repeatability expresses how similar (or variable) measurements are when you try to keep conditions constant. Expressing this in absolute numbers makes only sense if these absolute numbers can be interpreted. I suspect that a "repeatability of ± 700 ion counts at m/z 1000" or "average Euklidean distance between repeated measurements is 100 counts" doesn't ...


1

Remember that PCA makes two more or less arbitrary assumptions that ICA does not make: the loadings are defined to be orthonormal, and are ordered by decreasing variance So the order is arbitrary but sometimes convenient. Orthonormal loadings mean that the scores are just rotated and/or flipped, but not stretched. This is sometimes important, but often ...


Only top voted, non community-wiki answers of a minimum length are eligible