A large body of publicly accessible literature has become the driving force for research innovation. New, meaningful implicit relations between scientific terms can be mined from the published literature to construct hypotheses for further lab experimental studies [1], [2]. However, the time–cost and difficulty of manually surveying such large amount of literature is a bottleneck for extracting undiscovered knowledge [3]. For accelerating knowledge discovery, there is an urgent need for automatic Hypothesis Generation (HG) that aims to use text mining techniques to uncover hidden relations between non-interacting terms in literature [4].

HG was pioneered by Swanson [5] who creatively discovered implicit connection between Fish oil (A) and Raynaud’s Disease (C) because of their co-occurrence with high Blood Viscosity (B) in different sets of publications. This initial work is now called ABC model, which depends purely on the frequency of co-occurrence in publications [4], [5], [6]. The main issue of these approaches is that the term-pairs with high co-occurrence frequencies are not necessarily meaningful associations.

To extract explicit associations from publications, various machine learning strategies are exploited for HG [7], [8]. Machine learning-based approaches have made significant advances. However, they assume that the domain is static, which severely limits the ability of HG models because the biomedical knowledge is rapidly evolving with new associations being added all the time. The dynamic evolution of term-pair meaning provides key information on inferring the possible associations between them in the near future.

Recently, a variety of efforts are made to learn evolutional embeddings of terms [9], [10] or term-pairs [11], [12], [13] from diachronic literature for inferring future term-pair relations. RNN has been used to capture the dynamic evolution of term-pair relations [11], [12], [13]. Akujuobi et al., [11] use RNN to incrementally learn term-pair embeddings based on the inherent smoothness between adjacent time-steps. Zhou et al., [13] employ RNN to model the temporal differences of term-pair relations between two adjacent time-steps for accurately capturing dynamic evolution of term-pair relations.

However, inherent recurrent structures could not learn direct dependencies between any two time-steps in a temporal sequence. By contrast, attention mechanisms can model the direct dependencies no matter what their distance is in a sequence [14]. It is challenging to use attention mechanisms to explicitly capture the evolution of the temporal features over time. The dynamic evolution of term-pair relations has both continuity and difference. However, effectively leveraging attention mechanisms to capture both the continuity and difference in the dynamic evolution of term-pair relations remains a difficult task. In this paper, we simply rely on attention mechanisms to model complex spatiotemporal dependencies of term-pairs for explicitly capturing the time-evolving relations. Our goal is to learn spatiotemporal node-pair embeddings that can represent those natural properties of biomedical knowledge progresses, including smooth evolution and temporal differences. A novel Temporal Attention Networks (TAN) is proposed to produce powerful spatiotemporal embeddings for HG. Specifically, our TAN develops a Temporal Spatial Attention Module (TSAM) to establish temporal dependencies of spatial node-pair embeddings in consecutive time-steps for characterizing their temporal correlations. Meanwhile, a Temporal Difference Attention Module (TDAM) is proposed to sharpen temporal differences for highlighting historical changes of node-pair relations. As such, TAN can accurately capture the temporal evolution of term-pair relations by considering both the relevance and difference of term-pair embeddings.

Inspired by Akujuobi et al., [11], we decompose a graph G= into a sequence of attributed graphlets G=. Gt= is a temporal graphlet observed at time t, where Vt⊂V, Et⊂E and Xt denotes the set of nodes (terms), the term-pair co-occurrence and the node attribute of Vt, respectively. A node-pair label yt is tagged as positive if they are mentioned together in a paper at time t+1, otherwise negative. HG task aims to infer which nodes unlinked will be linked at time T with the temporal graphlets until T-1.

TAN focuses on both the continuity and difference of temporal term-pair embeddings to accurately learn evolutional embeddings of node-pairs for inferring new future connections (hypotheses). Automatically generating high quality and novel hypotheses could stimulate biomedical experts to conduct further lab experiments for accelerating scientific discovery. TAN is only evaluated on HG task here. It can also be extended to other time series forecasting tasks, such as the common temporal reasoning task, the traffic flow prediction task, etc.

Major contributions of our work are three-fold:

A general approach is proposed to model direct dependencies of spatial features while attending to the temporal relevance based on their inherent smoothness among consecutive time-steps.

The proposed approach tracks the semantic evolution of term-pairs by considering both the continuity and difference of temporal term-pair embeddings, with TSAM and TDAM, respectively for HG.

We use purely attention-based networks for learning spatiotemporal embeddings, which can be employed not only to hypothesis generation, but also to other time series problems.