BACKGROUND: I have huge (>10000x10000px) pathology slides images with different sizes. Here is an example:

enter image description here

You can find this specific example here. This images are pieces of tissue samples (biopsies) that are scanned with a high resolution device. Pathologists use them to look at the morphology of the cells and tell if a cell is malignant (cancer) or bening, and identify other types of structures (blood vessels, muscular cells, etc).

This images have a very high resolution but, as you may see in the image, not all the regions have information in it (there are regions without tissue here and there).

Hence, there are several tools to process these large images and most of them use "tiling" which consist on create patches of regions and process them individually. Like:

enter image description here From this paper

One solution to process this kind of images is the one used in this git. This tool performed feature extraction of every non-blank patches and create a vector of 1024 features for every patch. Finally all patches are concatenated in a matrix for every image with shape 1024xN (being N the number of patches).

OBJECTIVE: I would like to use this feature matrices together with some clinical data associated to each image and build and ML classifier that uses both, the clinical data and the feature matrix as input to make a prediction. Hence, I am looking for a way to merge this two datasets.

However, since the number of patches is different between images, I would like to find a method that reduce the shape of the matrix (1024xN) to the same shape for all the features extracted (1024xC, being C a common dimension for all the output matrices).

PRELIMINARY TRY 1: Here, I am tempted to select some N patches randomly and use them but I would like to know if there is any method to select top-K N patches.

PRELIMINARY TRY 2:I was looking at this answer but I think JLT could not be the solution since it reduce the number of features (here the 1024) and not the number of observations (here N). However, I see that this method also returns a projection matrix but I am unsure if this recapitulates the information in the initial one.

PRELIMINARY TRY 3: This git solve the problem but I would like to compare this method to a ML model (random forest etc.)

TLDR: Hence, I am looking for a method that could select the top-K most informative (¿? not sure if is the correct word here) observations. Here "informative" would mean the ones with less correlation with the rest of the patches, but it could be also any method that summarises the information differently. Any ideas how could this be done?

EDIT: Add more context on what means "informative" in this case

EDIT2: Add some background on the type of source of data I am using and structured the post for better understanding

EDIT 3: Added a bounty for the task and add a possible solution using Deep Learning. However, I would like to use ML tabular models for comparison.

  • 2
    $\begingroup$ meta.stackexchange.com/questions/66377/what-is-the-xy-problem $\endgroup$ Commented Mar 21, 2023 at 13:58
  • 3
    $\begingroup$ Welcome to CV. Could you explain to us what you mean by "informative" in this context? $\endgroup$
    – whuber
    Commented Mar 21, 2023 at 13:59
  • $\begingroup$ Hi! Yes, I understand that "informative" is quite vague. "Informative" in the first part only select the patches that are non-blank. In the second part, I was looking for some method for selecting patches. The first thing came to my mind was to discard linearly correlated patches, but I am unsure if there is any better method. Hope now is clearer $\endgroup$ Commented Mar 21, 2023 at 14:10
  • 1
    $\begingroup$ Thanks for the explanation, but it's completely opaque to me, because the very meaning and structures of these "patches" are vague and the objective of your analysis is not evident. It would help to understand this contextual information. $\endgroup$
    – whuber
    Commented Mar 21, 2023 at 16:13
  • 1
    $\begingroup$ Thanks for the feedback. I performed some edits and I hope now the post has more context and the objective is more evident. $\endgroup$ Commented Mar 21, 2023 at 18:08

1 Answer 1


Interesting question. From what I understand, you wish to combine the features you extract from the image, which have different shapes, 1042xN along with clinical information which is "constant" over the image, that includes, let us assume, some tabular data $X_i$ (patient $i$'s age, marital status, for example). If I am correct and that is the case, then I my suggestion is actually to try to convert your signals from being 1042xN into a scalar and use it in regression at the second stage.
In other words, my suggestion is to use a stacking ensemble. It also connected usεr11852's suggestion but allows for more agile solutions. In general, this approach involves multiple steps:

  1. Convert your image from 1042xN to Cx1, where C can be either an embedding dimension, or a score for the image (see elaboration on that bellow).
  2. Train a model using $[X_i,C_i]$ to predict $y_i$, which, if I got it correctly, gets the value of 1 if patient $i$ has at least $K$ of their cells being malignant (frankly, this is a generic framework, you can define your $y_i$ as you go, but there are some caveats, see below).

1st stage
At this stage, we look at the different tiles as a series of images. We aim to transform this series of images into one score in $[0,1]$. You can choose between multiple approaches here. One, for example, could be, an average of the classifications of the different tiles. In your description, you said that this data is often used to build a per-tile score. So you can do the same, and calculate the aggregated score to be $g(T)=\dfrac{1}{n}\sum_{j=1}^N f(t)$, where $f(t)$ is the per tile score given by model $f$, and $j$ is the tile index running up to $N$. Using the average is just intuitive but you can define any function you'd like, depending on your domain knowledge.
Another approach to dealing with such data, is to treat the per-tile images as a time series. Under this approach you'd have to change the way you look at y. You can define it as the proportion of malignant cells per person, or a binary variable that gets the value of 1 if patient $i$ has more malignant cells than a certain threshold. This new $y_i$ is constant across tiles, and changes from only on the subject's level, in contrast with $y_{ij}$ which was the target for each tile. This is actually the way vision transformers analyze visual data today, and it has proven to be beneficial in terms of predictive capabilities.
If you chose to use any of the above methods, now you have one number for each image, and this number reflects your belief regarding person i's risk, that rises from their image. We can continue to the second stage.

Image transformer example, taken from here

2nd stage At this second stage, we are combining the diagnosis from the image, with the patient's metadata, just as in real life. How we would do that? Using your favorite model.

A note about usεr11852's suggestion If you want to increase the weight of the diagnosis of the image mechanically, instead of returning a scalar value as suggested, you can try to return a vector. This requires a bit different approach, because under this approach we won't use the predicted value of the sequence of the tiles, but rather their representation on some latent space instead. But since you want to change both of the axis (you don't want to add $1042$ scalars nor you don't want to add $N$ scalar instead, so the question that is rising is which dimension we want to reduce, and if both of them, then how?). Sorry, I don't have a quick answer for that.

If you use stacking ensemble (that is the process that was described below, where you train a model on part of the data, and another model that one of its inputs is your first model's output), then you MUST NOT USE the same observations for that. That is, if you choose to train a tile-level classifier or a time-series model on those tiles, that will return a single score for an image, then you should exclude those patients for the second stage. Otherwise, the score's importance will be strongly upward biased, as the score is expected to be correlated with the dependent variable just from construction (it is because models ALWAYS have some degree of overfitting, but that is a discussion for another time).

Good luck!

  • $\begingroup$ Hi David. Thanks for the answer and for the effort to try to give me some hints on this. Your first assumption is correct however there is no clear relation between the cells that are in a tile and the prediction (just remarking this if it makes your answer different, idk). Regarding the rest, there are several public models that could be used to get an image-level prediction and I can stick to them for now. So, here, the point would be to find another dataset to train for the same $y_i$, right? Or $y_i$ could be different? How similar the dataset should be? Difficult questions I guess $\endgroup$ Commented Mar 25, 2023 at 16:41
  • $\begingroup$ Good questions though. Regarding the so-called 1st stage model, the closer the tasks you'd train the two models to predict, the better and more relevant the score of the 1st stage will be in the 2nd stage (where it will be inserted as one of the inputs). That is, if in the first stage, you are training a model to predict some covariate that is expected to be correlated with your final dependent variable, then you should expect a good correlation in the second stage. $\endgroup$ Commented Mar 25, 2023 at 18:15
  • $\begingroup$ Regarding your other comment, "there is no clear relation between the cells that are in a tile and the prediction", if on the "macro-level" there is a relations between the structure of the tissue and the dependent variable, whether it is in the level of the tile or the level of the image itself, then given enough images the model should learn to discriminate between the classes. $\endgroup$ Commented Mar 25, 2023 at 18:19
  • 1
    $\begingroup$ BTW, sorry for the multiple messages, I just wanted to note. Diverging from the world of statistical learning we often want also to have a descriptive model that says how much each factor denotes to the prediction (here, for example). If you're interested in the effect of the image diagnosis on your $y$ then using the same patients for two of the stages will surely bias your coefficient (upwards). $\endgroup$ Commented Mar 28, 2023 at 13:52
  • 2
    $\begingroup$ +1 from me. This is obviously useful even if it doesn't necessarily solve everything. (which is unlikely given the question's breath) $\endgroup$
    – usεr11852
    Commented Mar 30, 2023 at 0:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.