2
$\begingroup$

I wish to calculate the correlation between two different types of variables, namely a continuous variable and a variable containing a series/vector/list of floating numbers. I'm unsure about the terminology here, so vector might be the wrong name for the concept.

In my data, I want to calculate the correlation between something called word translation entropy (WTE) and any other continuous variable, let's say translation errors (error count; EC).

Assume that the dataset consists of a number of sentences that have been translated. Every translated word in the source text has some value called word translation entropy. This leads to a vector per sentence, the size of the number of words in a sentence. For instance: I like bananas: <114.9, 0.36, 12.98>. These numbers are not arbitrary but how they are calculated exceeds the scope of this question. You can find more on WTE in the CRITT TPR-DB.1 It is important, though, that the length of the vector differs across sentences as it's determined by the sentence length. Every sentence-pair has been annotated with the error count that were made in the translation (e.g. 0, 2, 4).

Example data

\begin{array} {|l|l|} \hline WTE & EC\\ \hline <45, 1.32> & 0 \\ \hline <8.2, 459.5, 45.3, 12.7, 102.1, 4.0> & 2 \\ \hline <9.5, 47.8, 72.6, 1.23, 36.45> & 1 \\ \hline <0.36, 14.7, 19.6> & 0 \\ \hline <27.2, 69.3, 34.4, 45.89, 96.36, 36.78, 154.7, 123.45> & 3 \\ \hline \end{array}

One way to deal with this kind of data would be to take the mean of the series (i.e. average word translation entropy) and turn it into a continuous variable. This is how the aforementioned CRITT TPR-DB does it. I am not convinced that this is the right approach. As you can see the values can be very far apart so that an average is not the right tool for the job. Perhaps taking the median is a better solution, but here I fear to downsize the large differences, e.g. the median for <8.2, 459.5, 45.3, 12.7, 102.1, 4.0> is 29 in which the high WTE of 459.5 is not well-presented. What this means is that the words with relatively very high or low WTE values really don't have that much influence.

Therefore, I wonder whether there is either another widely accepted way of turning a series of numbers into a single value that is actually representative of the full spectrum of the vector (taking into account great differences), or - better yet - whether there is a way to use the series/vector as a 'single value' when calculating correlations. Note the importance of the variable length of the vector!


1Carl, M., Schaeffer, M. J., & Bangalore, S. (2016). The CRITT Translation Process Research Database. In M. Carl, S. Bangalore, & M. J. Schaeffer (Eds.), New Directions in Empirical Translation Process Research (pp. 13–54). Cham, Switzerland: Springer International Publishing.

$\endgroup$
1
+50
$\begingroup$

How to represent your data (Feature Engineering).

It depends on your assumptions (and domain knowledge):

  • more words result in higher EC
  • difference of 3rd and 4th word in sentence are inversely proportional to EC
  • the actual first 5 entries in vector matter (if less than 5 entries fill with 0's)

they all need different representations of your data.

The position of a WTE value can be as important as the length of a vector/sentence or the mean and standard deviation (or some combination).

If WTE depends on single words only, not word-combinations, then

\begin{array} {|l|l|} \hline WTE & EC\\ \hline <45, 1.32> & 0 \\ \hline <8.2, 459.5, 45.3, 12.7, 102.1, 4.0> & 2 \\ \hline \end{array}

could be transformed to

\begin{array} {|l|l|} \hline WTE & EC\\ \hline 45 & 0 \\ 1.32 & 0 \\ 8.2 & 2 \\ 459.5 & 2 \\ 45.3 & 2 \\ 12.7 & 2 \\ 102.1 & 2 \\ 4.0 & 2 \\ \hline \end{array}

to start with.

$\endgroup$
  • $\begingroup$ I'm very confused by this - due to my own ignorance. Currently WTE is a per-word value, indeed. I do not understand, though, how you can viably transform the data as you did. Splitting the data like that, how is that still representative of the data? A WTE of 45 with EC 0 in contrast with WTE 45.3 -> EC 2. Can you give any sources about this kind of data manipulation? I'd like to read up on a motivation for such processing. $\endgroup$ – Bram Vanroy Oct 29 '18 at 12:27
  • $\begingroup$ Have a look at Feature Engineering in the context of Natural Langugage Processing and Text Analysis. Representing data sometimes comes with a loss of information. In my example words are not (really) considered as part of a sentence. But maybe you can do so by designing features which are suited to your specific problem. $\endgroup$ – lnathan Oct 29 '18 at 17:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.