Current advances in Pathomics tissue analytics are focused on predicting the biological behavior of human disease and cancer for precision medicine applications. One of the main challenges in Pathomics is to identify the types of data products that can be useful to identify patterns, relationships, or biomarkers to provide clinically actionable information [42]. Current Pathomics applications are incredibly powerful and useful for understanding how and why the dynamic microscopic landscapes of tissues change during the initiation and progression of disease. In order for Pathomics to be clinically useful, quantitative measurements of categorical observations of various phenotypic features of biological phenomena must support the development of computational models that predict changes associated with treatment and outcomes. When Pathomics tissue analytics become more routinely available, advanced understanding of human pathobiology will result from fully integrating Pathomics with clinical data from electronic health records, imaging, and clinical laboratory testing.

Each tissue sample from every patient is unique. Even though clinical history and other countless sources of variability exist in human biology, microscopic evaluation is used to identify morphologic nuances in tissues and cells during the course and progression of various human diseases. Pathomics tissue analytics offers an exciting and useful opportunity to augment traditional histopathologic evaluation on a per tissue sample basis to coincide with the clinical adoption of precision medicine. Even though tumor immune interactions in breast, pancreatic, and lung cancer have been carefully studied and actively explored by pathologists and biologists for many decades, we narrowly focus on the role of Pathomics to capture and quantify tumor immune interactions in H&E WSIs of these kinds of cancer tissue samples as practical examples that can meaningfully impact the role of precision medicine in immunotherapy.

We show straightforward examples of recently developed first-generation specialized deep learning Pathomics applications, referred to as Tumor-TILs analyses, that construct detailed spatial maps of tumor immune interactions in WSIs. Tumor-TIL maps are generated by combining automated detection of cancer regions with automated detection of lymphocytes by using deep learning algorithms in computer vision (see refs [43•, 44, 45] for technical descriptions). Briefly, Tumor-TILs analyses partition WSIs into tiled patches that are analyzed with two distinct deep learning computer vision algorithms, where image patches are designated as either (1) tumor or non-tumor or (2) lymphocytes or no lymphocytes and the remaining patches are considered as background (non-tumor and non-lymphocyte). The algorithmic predictions from each deep learning computer vision model are spatially mapped as probabilities with continuous values from 0 to 1. After evaluating the algorithmic performance of each model individually, the results from each model are combined to generate Tumor-TIL maps that can be used to explore cancer immunopathology.

Tumor-TIL maps represent massive amounts of computational analytics and Pathomics data in a very simple manner in order to help facilitate rapid understanding about cancer immunopathology. The spatial distribution and relative magnitude of lymphocytes are shown within the appropriate histologic context of the tumor microenvironment (TME) by overlaying deep learning algorithmic results on WSIs alongside corresponding H&E WSIs in order to complement traditional microscopic examination. In this manner, Tumor-TILs maps readily complement the routine visual assessment of tissue samples by pathologists with built-in quality assurance and control (QA/QC) for each step of the analyses. This is particularly important if deep learning Pathomics analyses are going to be used to predict treatment response and clinical outcomes for patient care. Since current Pathomics tools can clearly provide interpretable and clinically meaningful data from automated deep learning analyses of WSIs, we want to focus on how these tools can be used to study the functional immunologic status of the TME in WSIs of breast, pancreatic, and lung cancer.

Tumor-TILs analyses help identify and quantify the role of TILs in cancer. TILs represent an intrinsic prognostic biomarker in H&E tissue samples that can be readily evaluated in a wide variety of solid tumors with the potential to refine diagnostic characterization, patient stratification, and treatment selection. Marrying Pathomics with quantifying an intrinsic H&E biomarker in WSIs during the age of precision medicine and immunotherapy is an opportunity that cannot be missed since Tumor-TIL maps provide a simple and easily interpretable global view tumor immune interactions in the TME. Moreover, Tumor-TILs represents the capacity to perform automated, systematic, and uniform analyses in a scalable manner that can be compared to utilizing immunohistochemistry (IHC) to characterize the expression of biomarkers in tumors. Pathologists and biologists profile immune subtypes of cancer with a variety of methods in anatomic pathology, where inflammatory responses are characterized in terms of cellular components, spatial distribution, and relative magnitude. Pathologists also use terminology like “hot” and “cold” to describe the overall functional immune status of the TME in cancer tissue samples.

Inflammatory responses are typically evaluated with diverse terminology to describe immunohistopathology in tissue samples (e.g., inflamed, reactive, and necrotic), cellular components (e.g., “acute” for neutrophils, “chronic” for lymphoplasmacytic cells, and “histiocytic” and “granulomatous” for macrophages), magnitude (e.g., minimal, mild, moderate, and severe), and distribution (e.g., sparse, focal, and diffuse). Tumor-TILs analyses were developed to augment qualitative and semi-quantitative assessment by addressing the tangible demand to develop a quantitative method to capture and evaluate tumor immune interactions in solid tumors with standardized, cost-effective, and scalable methodology (see relevant literature and reviews from the International Immuno-Oncology Biomarker Working Group for TIL assessment guidelines in breast cancer and other solid tumors [22••, 43•, 44,45,46,47,48,49,50,51]).

Since the prognostic and predictive value of TILs as a biomarker in breast cancer is slowly gaining acceptance as a reportable biomarker that needs to be evaluated during routine histopathologic evaluation, Tumor-TILs may be very practical and useful example of Pathomics to help pathologists score TILs and further evaluate cancer pathobiology (in a manner that is similar to scoring and evaluating the expression of IHC biomarkers in different histologic types of cancer). However, the implementation of TILs as a routine and clinically important H&E biomarker has yet to be adopted, similar to the way that Pathomics have yet to be clinically implemented to quantitatively characterize aforementioned histopathologic features of tumors. Therefore, we present Fig. 1 to provide examples of how Tumor-TILs analyses in breast cancer can be immediately helpful to pathologists and provide useful insight about the functional immune status in breast cancer cases. Figure 1a–d and e–h represent two separate cases of breast cancer, where the panels are assembled in a manner to help ascertain the potential clinical value of TILs as a biomarker. This example also demonstrates the value of automated methods like Tumor-TILs as a practical and useful form of Pathomics to study and evaluate clinically meaningful tumor immune interactions in cancer by characterizing the spatial distribution of TILs in WSIs.

Tumor-TILs analyses in breast cancer. Deep learning Pathomics pipelines perform Tumor-TILs analyses in breast cancer, where panels a–d show analyses for specimen BRCA TCGA-AR-A24U-01Z-00-DX1 and panels e–h for specimen BRCA TCGA-AR-A250-01Z-00-DX1. Tumor-TIL maps depicted as follows: top left panels (a, e) show H&E WSIs at low magnification; top right panels (b, f) show automated tumor detection presented as a spatial probability distribution from 0 to 1 to evaluate algorithmic performance (non-tumor tissue colored blue); bottom left panels (c, g) show automated lymphocyte detection presented as a spatial probability distribution from 0 to 1 (non-lymphocyte tissue colored blue); and bottom left panels (d, h) show Tumor-TILs by combining tumor and lymphocyte detection (tumor colored yellow, lymphocytes colored red, and background non-tumor/non-lymphocyte tissue colored gray). Panels d and h show the spatial distribution of lymphocytes within the histologic context of cancer in order to identify peritumoral and intratumoral TILs and facilitate qualitative and quantitative interpretation of tumor immune interactions (to perform TILs scoring in the overall tumor region and spatially distinct areas). In the first case (d), spatial Tumor-TILs analyses show an abundance of peritumoral and intratumoral tumor immune interactions, whereas the second case (h) shows predominantly peritumoral TILs in breast cancer. The peritumoral distributions of TILs in (h) suggest that the tumor may be exerting immunosuppressive effects on TILs , which can help guide the selection of checkpoint inhibitor immunotherapy to activate the cytotoxicity of these lymphocytes. Tumor-TILs analyses have been performed in large collections of WSIs to evaluate both algorithmic performance and functional immune responses to support precision medicine in cancer

The 4-way panels show spatially distributed tumor immune interactions in WSIs of cancer tissue samples. These examples of Tumor-TIL maps present a global perspective of tumor immune interactions at low magnification. We can also overlay high resolution heat maps on H&E WSIs of cancer tissue sample to evaluate tumor and lymphocyte detection in order to perform quality assurance and control (QA/QC) of algorithmic performance. These examples combine a low magnification view of a H&E WSIs with separate panels showing overlaid probabilities of tumor and lymphocyte detection that are subsequently combined to generate Tumor-TIL maps. Performing Tumor-TILs analyses in large datasets of WSIs creates the ability to view the exquisite variability in the magnitude and spatial distribution of immune responses in the TME to support clinical research about TILs per the recommendations from the International Immuno-Oncology Biomarker Working Group [22••, 46, 47, 52,53,54,55]. Figure 1 also demonstrates how straightforward Pathomics analyses like tumor and lymphocyte detection in WSIs fundamentally complements traditional histopathologic examination to reliably characterize the functional immunologic state of the TME. The simplicity of this approach with incorporated QA/QC allows both pathologists and non-pathologists alike to seriously consider the value and potential of Pathomics tissue analytics to enhance visual examination and further our collective understanding of human pathobiology.

Tumor-TILs analyses augment the ability of pathologists to observe immune and non-immune mediated cancer pathobiology in a way that is not humanly possible across tens of thousands of samples. By predicting which patches of tissue contain tumor, lymphocytes, or both, we are effectively observing the ground cover of TILs in these types of cancer in an effort to accurately describe the percent of a cancer tissue sample that contains peritumoral and intratumoral TILs, lymphoid aggregates, and non-tumor associated lymphocytes. Tumor-TILs analyses have been implemented to measure the variability of these aspects of cancer immunopathology in publicly available datasets like the Cancer Genome Atlas (TCGA), as well as WSI datasets from other institutions in collaborative research projects. Automated Tumor-TILs analyses are also being expanded to catalog tumor immune interactions in several other types of cancer.

The overall goal of automated Tumor-TILs analyses is to introduce how clinical research and hospital laboratories can realistically use Pathomics to analyze H&E WSIs of cancer tissue samples from every patient. Automated Pathomics tissue analytic pipelines like Tumor-TILs show how massive amounts of data can be harvested from WSIs to generate clinically useful data products. If we can reliably correlate Tumor-TILs analyses with clinical outcomes, treatment responses, and survival, clinicians may gain the ability to predict prognosis, stratify patients, and identify candidates for immunotherapy and combinations with classical chemotherapy regimens and other treatment modalities.

Due to the growing acceptance of the role of host immunity in cancer and immunotherapy, systematically mapping tumor immune interactions with readily available Pathomics methods appear very necessary to comprehensively evaluate this aspect of tumor pathobiology. Similarly, the incorporation of TILs into the conventional classification of cancer also appears very likely, so methods like Tumor-TILs that can be readily incorporated will be hard to ignore when digital pathology becomes more widely adopted by clinical laboratories. Tumor-TIL maps are already being used to guide the microscopic examination of tissue samples on glass slides and/or WSIs to examine salient tumor immune interactions, but we need to explore how to use these analyses to develop more advanced types of Pathomics tissue analytics to investigate multidimensional relationships between other Pathomics features and histologic growth patterns, cytonuclear features of cancer cells, cancer staging, disease progression, metastasis, recurrence, treatment response, and survival.

In summary for this section, Tumor-TILs can be readily implemented to support various kinds of research to study cancer initiation, development, dissemination, and treatment response in the neoadjuvant and adjuvant setting by harvesting and cataloging massive amounts of important clinical research data. Tumor-TILs also provides numerous opportunities to explore and study the relationships between cancer immunopathology within the context of histologic growth patterns, cellular differentiation, cytonuclear features, cytogenetics, gene expression, and other types of omics to characterize the molecular profiles of various kinds of cancer. Since Tumor-TILs analyses advocate a solid starting point to promote capturing additional immune-related histopathologic parameters to better understand cancer pathobiology [22••], we present Fig. 2 as a proof of concept to demonstrate how Pathomics appears quite useful for studying tumor immune interactions in a highly complex disease like pancreatic cancer.

Tumor-TILs analyses in pancreatic cancer. Deep learning Pathomics pipelines perform Tumor-TILs analyses in pancreatic cancer ductal adenocarcinoma (PDAC), where panels a–d show analyses for specimen PAAD TCGA-IB-A5SS-01Z-00-DX1 and panels e–h for specimen PAAD TCGA-IB-7890-01Z-00-DX1 (Tumor-TIL maps are depicted in the same manner as Fig. 1). In both cases (panels a, h), Tumor-TILs analyses show predominantly peritumoral tumor immune interactions in pancreatic cancer with subtle differences in the magnitude of the responses. Beyond providing the ability to qualitatively score the relative strength of the immune responses, these maps can guide pathologists to areas of interest in glass slides and WSIs to further review other histopathologic features, such as using the abundance of tertiary lymphoid aggregates in the periphery beyond the invasive boundary of the tumor to guide the search for lymphovascular invasion (LVI). Tumor-TIL maps may also provide clues about tumor heterogeneity in terms of differential tumor immune interactions in the main tumor mass that can be compared to the functional role of TILs with infiltrative clusters of tumor cells. Tumor-TIL maps can also serve as a screening tool to help pathologists consider a particular diagnosis since these algorithms are trained with numerous PDAC samples to perform tumor detection

Tumor-TILs have been presented as a proof of concept to demonstrate the power of Pathomics to capture and analyze complex data from pathology images in a uniform manner. We hope that we have shown how these readily available methods can be deployed to analyze hundreds to thousands of WSIs to investigate the role of TILs in cancer immunopathology so far. We also hope to have demonstrated how Pathomics can quantitatively score the number of TILs and characterize their spatial distribution within the appropriate histologic context of cancer, which can be incredibly powerful for cohort discovery and real-time tissue analytics in surgical pathology. Beyond these very important immediate uses, we also want to show how the automated reporting of an overall score of the percent of TILs in a tumor region with a description of their spatial distribution can transform the identification and stratification of patients who might benefit from readily available immunotherapeutic options. To that end, we hope that the selected figures have substantiated the concept of computationally characterizing and scoring TILs as a biomarker with Pathomics, which can be readily utilized as simplistic standalone screening tests to identify hot and cold immune subtypes of cancer and other types of nuances in immunopathologic responses on a case by case basis.

In this respect, Tumor-TIL maps were developed as a visual tool and prototype for a potential Pathomics TILs screening test to help researchers and pathologists quickly evaluate the functional immune status of tumors in order to further guide the evaluation of TILs at higher magnification in relevant ROIs (e.g., TILs with respect to variable morphologic growth patterns and cytonuclear features associated with worse clinical outcomes and poor survival). Therefore, we discuss how Tumor-TILs can be leveraged to drive the development of more powerful ensemble Pathomics by incorporating computational image analysis methodology in this section [12, 13, 56,57,58]. For example, ensemble models can support scientific inquiry by using Tumor-TILs in conjunction with engineered and hand-crafted features to identify particular tumor niches for further laser capture microdissection and molecular testing. Beyond the spatial distributions of TILs, the TILs themselves can be further characterized by size, shape, color, texture, and chromatin density to guide the development of multiplex IHC panels and molecular tests to identify different subsets of immune cells and the expression of immunotherapy drug targets in a non-destructive and cost-effective manner. Other interesting applications include downstream analyses that leverage Tumor-TILs Pathomics data with nearest neighbor distances, clustering, topography, and fractal occupancy analyses to determine the prognostic and predictive significance of the spatial distributions of TILs [22••, 45, 50, 59,60,61,62,63,64,65,66].

As mentioned previously, we can report Tumor-TILs analyses in manner that parallels how the expression of IHC biomarkers in a histologic tissue sections are routinely reported. Similarly, we can envision how Tumor-TILs may be used to suggest further reflex testing during diagnostic workup while being integrated with laboratory data from chemistry, hematology, flow cytometry, human leukocyte antigen (HLA), microbiology, molecular laboratory testing, and clinicopathologic patient data in order to stratify patients and formulate treatment strategies based on computational phenotype. Potential scenarios could involve using first-generation Tumor-TILs to help overcome the problems associated with PD-1/PD-L1 testing assays for immune checkpoint inhibitors [64•, 67, 68] by defining salient tumor immune niches for PD-1/PD-L1 scoring and/or reporting PD-1/PD-L1 IHC scores in conjunction with Pathomics TILs scores integrated with electronic healthcare data to devise an appropriate treatment strategy. Relatively straightforward applications of readily available Pathomics applications like Tumor-TILs can be easily implemented to perform an additional level of time-, cost-, and resource-efficient data collection and analyses in daily practice, clinical trials, and translational clinical research. In the current cycle of development, first-generation Tumor-TILs Pathomics are being used as value added tools in a wide range of clinical research studies to explore and discover various diagnostic, prognostic, and predictive biomarkers.

Up to this point, we presented Tumor-TILs within a narrow focus to help convey realistic avenues for research and clinical adoption with readily available first-generation examples of Pathomics tissue analytics. However, guiding the actual design and selection of immunotherapeutic regimens on a per patient basis with a computational Pathomics marker is a challenging and complex matter when we consider how there are more than 100 different immunophenotypes in human blood [60, 69,70,71,72]. This clinical scenario gets even more complicated by a wide level of variability across individuals during functional physiologic states related to normal homeostasis and clinically detectable adaptations to disease. Thus, it is unsurprising that immunopathology is just as variable in the TME of different types and subtypes of cancer [46, 47, 52, 53, 73,74,75] and a partial explanation of why immune checkpoint treatments have improved overall survival in a subset of patients when compared to standard therapy, which limits our ability to determine the patients who will actually benefit from specific types of immunotherapy.

In this scenario, Tumor-TILs may be useful as first-order analyses that capture the diversity and heterogeneity of immune responses in a scalable and cost-effective manner in WSIs. Tumor-TIL maps can be readily utilized as base layers and incorporated in more advanced Pathomics tissue analytics to help further explore and define the complex relationships between cancer and immune cells in the TME. As stated in the end of the previous section, several types and directions of scientific investigation can be performed by correlating these analyses with a wide spectrum of clinicopathologic, molecular, radiologic, and treatment data to develop more advanced Pathomics applications for precision medicine. As increasingly large and complex datasets from clinical trials become available, Pathomics biomarkers may become as important as specific histopathologic features of cancer and IHC biomarkers.

There has never been an opportunity to study tumor immune interactions by using automated Pathomics image analysis methods to interpret the role of complex immune responses in cancer without employing other laboratory methods like IHC, flow cytometry, genomic sequencing, and gene expression. In this section, we envision practical ensemble Pathomics methods to further support and potentially accelerate scientific discovery and clinical translation by leveraging Tumor-TILs analyses in WSIs. We envision how ensemble Pathomics tissue analytics can conceivably parallel the impact of IHC and molecular subtyping that fundamentally transformed the classification and subclassification of tumors in anatomic pathology.

We believe that advanced Pathomics applications can be feasibly developed and implemented to further unravel the role of TILs along a trajectory that mirrors how IHC biomarkers that were once rigorously tested for use during diagnostic classification are being actively revisited to explore prognostic value and utility for serving as therapeutic drug targets. Since current versions of Tumor-TILs analyses and mapping provide a global view of tumor immune interactions, we present recently developed prototypes that are geared towards improving the characterization of tumor immune interactions in WSIs of cancer tissue samples. Beyond the examples that we present to study tumor immune interactions and TILs, first-generation Tumor-TILs models are being integrated with classical image analysis to also calculate tumor area, perimeter, greatest dimension, and irregularity of the invasive boundary in order to interpret TILs in various types of histologic contexts. We can use these analyses to determine whether the spatial distribution of lymphocytes is intramural, peritumoral, organized as tertiary lymphoid aggregates, loosely scattered, focal, or scant in WSIs to complement how pathologists evaluate and report the assessment of inflammatory responses and TILs with descriptions of spatial location, strength, and tumor immune interactions in relevant diagnostic ROIs (e.g., central tumor, invasive edge, diffusively infiltrative clusters, and tumor deposits).

Therefore, we present Fig. 3 as a prototype that combines Tumor-TILs analyses with mapping microarchitectural growth patterns of adenocarcinoma in lung cancer to study tumor immune interactions within the context of histologic subtype. Using Fig. 3, we can study the spatial distribution of TILs with respect to the spatial distribution of growth patterns of lung adenocarcinoma to quantitatively measure the variability of tumor immune interactions in a more refined histologic context of tumor landscape. Even though this tool needs further refinement and QA/QC, our intention is to show how automated Tumor-TILs plus the classification of histologic subtype can feasibly support current and future clinical trials in lung cancer with immunotherapy by identifying and quantifying treatment responses with checkpoint inhibitor therapy. Figure 3 also demonstrates a proof of concept of a scalable Pathomics method can be enhanced to potentially discover novel biomarkers with ensemble Pathomics for precision medicine applications, immunotherapy, and beyond.

Tumor-TILs analyses combined with histologic subclassification in lung cancer. Working prototype of a deep learning Pathomics pipeline to perform Tumor-TILs analyses and histologic subclassification in lung adenocarcinoma, where Panels a–d show analyses for specimen LUAD TCGA-69-8255-01Z-00-DX1 and panels e–h for specimen LUAD TCGA-69-A59K-01Z-00-DX1 (Tumor-TIL maps are depicted in the same manner as Fig. 1). In both cases (panels a, h), spatial Tumor-TILs analyses show a robust presence of intratumoral TILs in lung cancer but show differences in the presence of peritumoral lymphocytes due to other histopathologic features. The histologic subclassification of these two specimens is primarily lung adenocarcinoma acinar type, where a shows the presence of necrosis associated with very strong lymphocytic responses in comparison to the absence of necrosis in e. Tumor-TILs analyses in d show small nests of viable tumor adjacent to the area of necrosis as well as lymphoid aggregates that can presumably send reinforcements. This image demonstrates the value of pursuing further investigation into characterizing tumor immune interactions and scoring the exquisite variability of TILs with respect to growth pattern and tumor differentiation

Current deep learning models analyze tiled patches due to limited computational resources. However, access to supercomputing and cloud computing resources has led to new opportunities to leverage the advances in digital pathology from the past two decades. Beyond the opportunities to ensemble and automate existing Pathomics methods to extract, quantitatively characterize, and catalog various properties of tissues in vast collections of WSIs, previous computational limitations are being surpassed to analyze and map the features of every single cell in WSIs at significantly higher resolution. Tumor-TILs and other types of ensembled Pathomics can analyze hundreds of thousands to millions of cells and objects in WSIs without a conceivable limit as technology advances even further. We envision using advanced ensembled Pathomics to compute tumor cellularity based on calculations of the number of segmented nuclei in a tumor region; quantitatively characterize tumor differentiation by computing the percent of cells in specific types of architectural configurations; and generate high resolution maps based on the classification of all of the different types of cells in human tissues with accompanying engineered and hand-crafted features that measure color, size, shape, texture, orientation, and chromatin density.

We also propose incorporating Tumor-TILs into advanced ensemble Pathomics methods to capture as many salient histopathologic features as possible for every WSI at every academic institution, healthcare system, and tumor registry. We imagine being able to utilize Pathomics pipelines to analyze WSIs with corresponding data from tumor registries to study tumor heterogeneity, metastatic cancer, therapeutic efficacy, and drug resistance in order to develop models to help select treatment based on predicted survival [22••, 50, 61, 64•, 76, 77]. However, laying the foundation for the deployment of Pathomics for clinical decision support systems will still require a fair amount of time considering that the Food and Drug Administration (FDA) recently aproved the clinical use of a commercial slide scanner and digital pathology system just a few years ago [30, 31, 36]. Before Pathomics data can be used to provide meaningful clinical information to physicians and patients to help predict patient survival, likelihood of disease recurrence, and progression in the very near future, Pathomics methodology that are still early in the development cycle have to be extensively tested and validated in appropriate clinical settings. As shown in Figs. 1, 2, and 3, current Tumor-TILs Pathomics methods extract and present information that cannot be performed at scale by pathologists using microscopes, IHC, sequencing, and gene expression to characterize tumor immune interactions in every tissue sample.

However, currently accepted methods to evaluate tumor immune interactions in tissue samples focus on the manual assessment of stromal TILs (sTILs), which have been shown to be important prognostic and predictive biomarkers in triple-negative (TNBC) and HER2-positive breast cancer. One of the main reasons that sTILs have become so important is due to consistent reproducibility of scoring by pathologists with the use of standardized guidelines developed by the clinical research community in three large studies [40•, 50, 64•, 65, 78]. Despite the acceptance of sTILs as prognostic and predictive biomarkers, the assessment of sTILs is challenging within the complex landscape of cancer, which results in scoring discordance due to the heterogeneity of the distribution of lymphocytes. Discrepancies also arise from the lack of consensus about the precise boundary of tumors, variable presence of tumor-associated stroma, associations of lymphocytes with other microanatomic structures, presence of other kinds of immune cells [50, 65]. Even though these kinds of variations during the evaluation of sTILs have mitigable effects in estimating risk in early TNBC, multiple areas are scored and averaged during the evaluation of sTILs to improve consistency and minimize the effects of observer variability, scoring discrepancies, and cutoffs that could affect treatment selection [50, 65].

Due to the specific interest in sTILs, we wanted to demonstrate the ability to integrate Tumor-TILs analyses with another open source Pathomics method, Hover-Net [79]. We combined the capabilities of Hover-Net, which was developed to perform simultaneous segmentation and classification of all of the nuclei in colorectal cancer, by labeling breast cancer cells, lymphocytes, and stromal cells (e.g., connective tissues, endothelium, and nerves) in a highly curated and quality controlled set of images with segmented nuclei [80] in order to calculate the populations of these cells in Tumor-TIL maps. Figure 4 shows the concept of an integrated Pathomics tissue analytics workflow that starts with deep learning analyses to perform automated tumor and lymphocyte detection in breast cancer to generate Tumor-TIL maps combined with Hover-Net as proof of concept to further support the fundamental value of mapping and quantifying tumor immune interactions in cancer. Our primary goal is to show how we can enhance the capabilities of existing Pathomics methods and analyses like Tumor-TIL maps to gain even further insight at the cellular level, which opens the door to numerous downstream research opportunities. A secondary reason for including our implementation of Hover-Net was to show the generalizability of Pathomics methods that can be customized in a practical manner to answer a wide spectrum of research questions.

Tumor-TILs analytics combined with nuclear segmentation and classification with Hover-Net in H&E WSIs. Working prototype of an integrated workflow that starts with deep learning analyses to perform automated tumor and lymphocyte detection in breast cancer to generate Tumor-TIL maps, as shown in panels a–d for specimen BRCA TCGA-E2-A9RU-01Z-00-DX1 based on the format in Fig. 1. Implementation of Hover-Net is shown in a representative area in e–g to demonstrate simultaneous segmentation and classification of nuclei. Panel e shows a × 200 representative area of the H&E WSI with panel f showing the segmented and classified nuclei overlaid on the H&E and g showing the masks of the nuclei (lymphocytes denoted in red, stroma and endothelium denoted in blue, and cells with tumor-like features in green). We present this application of Hover-Net as a proof of concept to demonstrate one of many opportunities to integrate Tumor-TILs with other open source Pathomics methods to analyze H&E WSIs at higher resolution. This example of ensemble multiscale Pathomics methods can support scoring the number of TILs in distinct areas (e.g., stromal TILs in central tumor and invasive margin), calculating tumor cellularity in terms of the numbers of cancer cells, IHC quantification, and classical image analysis (color, size, shape, texture, and orientation). Going from the global view from Tumor-TIL maps to cellular level analyses, Hover-Net calculated 2312 nuclei in -g to show that 89% of the cells were classified as lymphocytes. Hover-Net has also been implemented to calculate the number of tumor and stromal cells (not shown since we are focusing on TILs). The green cells in this image are reactive macrophages which have features that can be confused with cancer cells (e.g., increased nuclear size, nucleoli, and irregularities in shape and chromatin texture)

Tumor-TILs provides a global overview of tumor immune interactions, whereas Hover-Net focuses on nuclear segmentation and cell classification at much higher magnification and resolution. This kind of ensemble Pathomics methods supports scoring the number of TILs in distinct areas (e.g., stromal TILs in central tumor and invasive margin), calculating tumor cellularity in terms of the numbers of cancer cells, and classical image analysis to further characterize each classified cell in terms of color, size, shape, texture, and orientation. For example, Hover-Net analyses calculated 2312 nuclei in the representative area shown in Fig. 4e, where 89% of the cells were classified as lymphocytes (Hover-Net has also been implemented to characterize the number of tumor and stromal cells, not shown since we are focusing on TILs).

Even though these methods still require considerable refinement and QA/QC, both Figs. 3 and 4 show how the overall spatial patterns of distribution of TILs can be further interpreted in various histologic contexts at various scales of magnification and resolution with respect to histologic subtype, tumor differentiation, and actual numbers of cancer and immune cells. By utilizing deep learning based nuclear segmentation [80] with classification of different types of nuclei with Hover-Net [79], we can further refine our characterization of the spatial relationships between tumor and immune cells in the TME with a level of precision that has been dreamt about for decades by cancer researchers and pathologists. Ensemble Pathomics can be integrated to systematically capture histopathologic features from multiple scales of magnification en route to developing more comprehensive Pathomics pipelines to catalog quantitative measurements of various phenotypic properties of tissues, cells, and nuclei to support better TILs scoring, IHC quantification, and integrated analyses with other types of data.

Our goals were to introduce Pathomics methods that can be readily implemented to study tumor immune interactions on a large scale, but we have to responsibly address that clinical adoption will require rigorous testing and evaluation. Pathomics algorithms will need to be classified as virtual laboratory tests, which requires evaluating sensitivity, specificity, processing speed (e.g., turnaround time), costs, and limitations. In ways that are no different than any other kind of laboratory test, these kinds of deep learning computer vision Pathomics algorithms must be extensively validated in multiple independent clinical studies before deployment. Currently, the FDA Office of Science and Engineering Laboratories (OSEL) is designing a Medical Device Development Tool (MDDT) to evaluate Pathomics methods to evaluate TILs for clinical use in breast cancer as a necessary mechanism to evaluate both simple and advanced ensemble Pathomics applications due to the challenges associated with so many unknowns within such a complex clinicopathologic scenario (please see refs. [50, 65]). We hope that we have shown how Tumor-TILs Pathomics may serve as a useful tool to view the microscopic world of tumor immune interactions at the cellular and molecular level to support the effort to truly characterize functional immunopathology in every cancer patient for future precision medicine applications.