This work presents a supervised subtype differentiation learning of lung cancer features in a latent space constructed with a variational autoencoder. In such space, complicated patterns are quantified by estimating a differentiation grade of typical encoded features of lung cancer subtypes. Specifically, selected tissue samples of non-small cell lung cancer are mapped to a latent space and a logistic regression model assigns differentiation cancer subtype grade to the embedded tissue samples. The latent representation captures the invariant features of the most representative tissue samples for both well-differentiated adenocarcinoma and squamous cell, and confusing cases of poorly differentiated complex mixtures of tissue patterns subtypes. This approach builds up a subtype differentiation grade of non-small cell lung cancer among complex structures which are fully interpretable and integrable with a pathology workflow. Typical tissue samples of well-differentiated lung cancer subtypes are grouped close in the latent space with high confidence of the differentiation grade, while poorly differentiated tissue samples, with lower confidence of the differentiation grade, are located at other latent space regions. A variational autoencoder (VAE) was trained to learn the latent space representation with training data of representative tissue samples picked from well-differentiated adenocarcinoma (five cases) and squamous cell (five cases) lung cancer subtypes. Validation was performed by selecting six cases for training and evaluating the location in the latent space of tissue samples from four different cases. Two different metrics, MAE and RMSE, estimated the location of these patches with respect to the patches belonging to the six cases. The best model, under a cross validation, achieves an average performance of MAE = (0.072 ± 0.0004) and RMSE = (0.2654±0.0019). In addition, for ten different cases (five adenocarcinoma and five of squamous cell), performance was MAE = 0.2275 and RMSE = 0.477. These results demonstrate this type of representation may capture a reduced set of histopathological invariants, use them to quantify complex patterns and improve the reproducibility of certain estimations.

Keywords: Lung Cancer, Latent Space, Variational Autoencoder, Metric Learning, Digital Pathology