Modeling unobserved geothermal structures using a physics-informed neural network with transfer learning of prior knowledge

Abstract

Deep learning has gained attention as a potentially powerful technique for modeling natural-state geothermal systems; however, its physical validity and prediction inaccuracy at extrapolation ranges are limiting. This study proposes the use of transfer learning in physics-informed neural networks to leverage prior expert knowledge at the target site and satisfy conservation laws for predicting natural-state quantities such as temperature, pressure, and permeability. A neural network pre-trained with multiple numerical datasets of natural-state geothermal systems was generated using numerical reservoir simulations based on uncertainties of the permeabilities, sizes, and locations of geological units. Observed well logs were then used for tuning by transfer learning of the network. Two synthetic datasets were examined using the proposed framework. Our results demonstrate that the use of transfer learning significantly improves the prediction accuracy in extrapolation regions with no observed wells.

Introduction

Numerical geothermal reservoir models are used to simulate coupled heat and fluid transport underground by solving partial differential equations for mass and energy conservation (Pruess et al. 1985; Ingebritsen et al. 2010; O’Sullivan and O’Sullivan 2016). The numerical model represents the spatial extents of subsurface fluid paths and the locations and sizes of heat sources. Expanding the use of the numerical model is essential for understanding the geothermal characteristics of target sites. In particular, numerical modeling has been widely used to understand natural-state geothermal systems, assess resource potential, and strategically manage geothermal reservoirs during development (Franco and Vaccaro 2014).

Numerical modeling of target geothermal fields requires matching a model with observed well logs and incorporating conceptual models based on expert interpretations of geological, geochemical, and geophysical data. Owing to the nonlinearity and complexity of geothermal reservoir simulations, significant effort has been put into efficient inverse modeling of geothermal systems. Some approaches (e.g., gradient-based or Bayesian) have been proposed and used to calibrate geothermal natural-state models (Finsterle 2000; Bjarkason et al. 2016; Bjarkason et al. 2018; Cui et al. 2018; Bjarkason et al. 2019; Maclaren et al. 2020). However, calibrating numerical simulations is extremely time-consuming because of frequent run failures, and gradient-based optimization sometimes falls into local minima. Consequently, modeling geothermal systems still relies on manual calibration (O’Sullivan and O’Sullivan 2016), and current approaches are far from optimal (Maclaren et al. 2020). Further development is required to improve the computational efficiency and robustness of natural-state estimation methods, and new methods with different perspectives are warranted.

Recently, modeling methods for geothermal systems using machine learning including deep neural networks (DNN) have been proposed and applied to estimate underground thermal (Koike et al. 2001; Spichak et al. 2006; Ishitsuka et al. 2018; Ishitsuka et al. 2021; Shahdi et al. 2021) and permeable structures (Suzuki et al. 2024). In a typical approach for these methods, DNNs use temperature logs at wells as training data and predict the temperatures at locations other than the well measurement points (Koike et al. 2001; Spichak et al. 2006; Ishitsuka et al. 2018; Ishitsuka et al. 2021). Although such approaches have shown promise for predictions around regions where well logs are available, prediction performance in the extrapolation region remains challenging. To address this, alternative approaches, including DNN variants incorporating electrical resistivity as training data, have been proposed (Spichak and Zakharova 2009; Spichak and Zakharova 2012; Spichak et al. 2015; Ishitsuka et al. 2018; Ishitsuka et al. 2021). However, these approaches do not completely solve the problem of inaccuracy when extrapolating beyond the well range (Ishitsuka et al. 2021).

Physics-informed neural networks (PINNs) incorporate physics laws described as partial differential equations into the loss function to consider the physical plausibility of the predicted quantities (Raissi 2018; Raissi et al. 2019). The PINN framework is effective for modeling various physical phenomena in different research domains (Liu et al. 2020; Cai et al. 2021; Haghighat et al. 2021; Amini et al. 2022; Sharma et al. 2023). In energy engineering and science, PINNs have been applied for wind power systems (Zhang and Zhao 2021; Li and Zhang 2022; Tian et al. 2024; Wang et al. 2024), solar power systems (Liu et al. 2021; Pombo et al. 2022), thermal modeling of buildings and construction materials (Gokhale et al. 2022; Liu et al. 2024), and petroleum development and production (Lu et al. 2022; Du et al. 2023). In geoscience and geoengineering, PINNs have been utilized to discern hydrological processes (Tartakovsky et al. 2020; He et al. 2020; Yeung et al. 2022; Bhasme et al. 2022; Frame et al. 2023; Meray et al. 2024), seismic wave propagation (Karimpouli and Tahmasebi 2020; Song et al. 2021; Rasht-Behesht et al. 2022; Zhang et al. 2023), crustal deformation (Okazaki et al. 2022), and rock properties (Chen and Zhang 2020; Chakravarty et al. 2021). A PINN for geothermal modeling has recently been developed to enhance the physical plausibility of temperature, pressure, and permeability predictions at depth (Ishitsuka and Lin 2023). Although subsurface heat and fluid migration are coupled phenomena, the predicted temperatures and pressures returned by a conventional DNN do not guarantee physical laws. The PINN framework was also shown to be useful for learning energy production of time-series production data from natural geothermal reservoirs (Qin et al. 2024) as well as predicting system behaviors in enhanced geothermal systems (Yan et al. 2024a, 2024b) and closed-loop geothermal systems (Zhang and Li 2023). Recent reviews of PINNs were conducted by Karniadakis et al. (2021), Cuomo et al. (2022), Muther et al. (2022), and Faroughi et al. (2024). Modeling physical phenomena using DNNs can be furthered with PINNs. However, reduced accuracy in the extrapolation domain, as observed in DNN modeling, remains a concern.

To address the challenges of extrapolation inaccuracy and leverage the effectiveness of PINNs for geothermal modeling based on Ishitsuka and Lin (2023), this study proposes a framework to use transfer learning with PINN (PINN–TL). It utilizes prior expert knowledge (including uncertainties) as multiple numerical models for the target area. Transfer learning is a technique used to improve the prediction performance of a new task by re-training a pre-trained neural network model with new data in the domain of the new task (Pang and Yang 2010; Weiss et al. 2016). When only a small amount of data is available for a new task, transfer learning can be employed to improve predictive performance. One of the recent advancements in PINN applications is to use numerical models to supplement the limitations of observations (Du et al. 2023; Xu et al. 2023; Meray et al. 2024). Geothermal energy development is a typical target that suffers from a lack of observations because deep thermal structures largely control geothermal reservoir conditions, whereas well measurements are often limited to shallow depths. However, to our knowledge, no previous study leveraged prior knowledge to enhance the effectiveness of PINNs for geothermal utilization.

The proposed method is expected to enhance prediction accuracy and capture the characteristics of deep geothermal systems, especially in regions lacking well observations. The scientific advancement of the proposed method is its flexible incorporation of prior information into geothermal modeling combined with deep learning. The effectiveness of the proposed PINN–TL method was verified using 2D numerical simulation models that resemble typical geothermal systems.

Methodology

PINN

PINNs have been proposed as methods for predicting quantities consistent with physical laws (Raissi 2018; Raissi et al. 2019). Traditionally, DNNs are trained based only on observations; however, guaranteeing the physical plausibility of these predictions is often difficult. In PINNs, physical equations and boundary conditions are included as terms in the loss function, $Loss$, of the DNNs to obtain predicted quantities that approximately satisfy the physical laws described by partial differential equations and boundary conditions.

$$Loss=DataMisfit+Physics+Bounds$$

(1)

Here, $DataMisfit$, $Physics$, and $Bounds$ represent the data misfit between predictions and observations, errors in physical constraints, and errors in boundary conditions, respectively. In addition, automatic differentiation (AD) is used to calculate the governing equations described by partial differential equations. AD calculates the differentiation of a target by combining the differentiation of elementary functions using the chain rule, which is less computationally intensive and more accurate than numerical differentiation. Applying AD with DNNs is efficient and reasonably straightforward because DNN libraries commonly include AD implementations to obtain loss derivatives using backpropagation.

Problems characterized by a limited amount of available data and partially known physical laws are challenging for both traditional data-driven and physics-based modeling approaches (Tartakovsky et al. 2020). The PINN approach is advantageous over traditional deep learning when the amount of available training data is limited (Tartakovsky et al. 2020). Incorporating physical laws as additional learning constraints provides additional valuable information and PINNs can thus potentially provide more generalizable predictions (Raissi et al. 2019).

PINN for hydrothermal systems

In this study, we used the PINN method proposed by Ishitsuka and Lin (2023) for hydrothermal systems as a reference PINN. The method approximately satisfies the mass and energy conservation laws (Eqs. 3 and 4) to describe a hydrothermal system with a single-phase pore fluid (Ingebritsen et al. 2006). The loss function of the physics-informed neural network used in this study was as follows:

$$Loss=\frac{1}{M}{\sum }_{m=1}^{M}{\left|{\widehat{T}}_{obs,m}-{\widehat{T}}_{pred,m}\right|}^{2}+\frac{1}{M}{\sum }_{m=1}^{M}{\left|{\widehat{P}}_{obs,m}-{\widehat{P}}_{pred,m}\right|}^{2}+\frac{1}{M}{\sum }_{m=1}^{M}{\left|{\text{log}}_{10}\widehat{K}_{obs,m}-{\text{log}}_{10}\widehat{K}_{pred,m}\right|}^{2}+Physic{s}_{MS}+Physic{s}_{EN}+Bounds$$

(2)

$$Physic{s}_{MS}=\frac{{w}_{MS}}{{M}_{col}}{\sum }_{m=1}^{{M}_{col}}{\left|\nabla \cdot \left(\frac{{K}_{pred,m}{\rho }_{w}}{{\mu }_{w}}\left(\nabla {P}_{pred,m}-{\rho }_{w}g\nabla z\right)\right)\right|}^{2}$$

(3)

$$Physic{s}_{EN}=\frac{{w}_{EN}}{{M}_{col}}{\sum }_{m=1}^{{M}_{col}}{\left|\nabla \cdot \left(\frac{{K}_{pred,m}{\rho }_{w}{c}_{w}{T}_{pred,m}}{{\mu }_{w}}\left(\nabla {P}_{pred,m}-{\rho }_{w}g\nabla z\right)\right)+\nabla \cdot {\lambda }_{r}\nabla {T}_{pred,m}\right|}^{2}$$

(4)

$$Bounds=\frac{{w}_{BN}}{{M}_{side}}{\sum }_{m=1}^{{M}_{side}}{\left|\frac{{\partial P}_{obs,m}}{\partial {\varvec{n}}}-\frac{{\partial P}_{pred,m}}{\partial {\varvec{n}}}\right|}^{2}+\frac{1}{{M}_{top}}{\sum }_{m=1}^{{M}_{top}}{\left|{\widehat{P}}_{obs,m}-{\widehat{P}}_{pred,m}\right|}^{2}+\frac{1}{{M}_{top}}{\sum }_{m=1}^{{M}_{top}}{\left|{\widehat{T}}_{obs,m}-{\widehat{T}}_{pred,m}\right|}^{2}$$

(5)

where $T$, $P$, and $K$ denote the temperature, pressure (pore water pressure), and permeability, respectively, and ${\widehat{T}}_{m}$, ${\widehat{P}}_{m}$, ${\text{log}}_{10}\widehat{K}_{m}$ are the normalized values of the physical parameters at the $m$ th observation location. In this study, we assumed that the temperatures, pressures, and permeabilities used to calculate the data misfit were provided at all data points during pre-training and measured along the wells when the pre-trained network was tuned during transfer learning. Variables $x$ and $z$ denote the horizontal and vertical coordinates, respectively, and $g$ is the acceleration of gravity. Physical constraints are divided into $Physic{s}_{MS}$ and $Physic{s}_{EN}$, which represent the mass and energy conservation laws. The symbols ${\rho }_{w}$, ${c}_{w}$, ${\mu }_{w}$, and ${\lambda }_{r}$ denote the density, specific heat, viscosity of the fluid, and the thermal conductivity of the rock, respectively. For ${\rho }_{w}$, ${c}_{w}$, and ${\mu }_{w}$, the pore fluid was assumed to be pure water, and these thermodynamic properties were calculated based on IAPWS-95 (Wagner and Pruß 2002; Huber et al. 2009). $M$, ${M}_{side}$, and ${M}_{top}$ refer to the number of observation points at the wells and the number of positions satisfying the lateral and upper boundary conditions, respectively, where n denotes a normal vector. ${M}_{col}$ is the number of collocation points at which the loss of the physics constraints, as outlined above, is calculated. We used ${M}_{col}=1000$ in this study. The collocation points were randomly selected throughout the system. When computing the loss function to include the Physics and Boundary terms, we introduced multiplication coefficients (${w}_{MS}$, ${w}_{EN}$, and ${w}_{BN}$ in Eqs. 3–5) to balance their magnitudes, according to Ishitsuka and Lin (2023). The boundary conditions at the bottom were not incorporated in Eq. 5, in contrast with Ishitsuka and Lin (2023), because the bottom boundary condition is often difficult to determine before PINN analysis. In addition to the physical laws, a Dirichlet boundary at the top surface and Neumann boundaries on the sides were set as boundary conditions.

The neural network architecture used in this study comprised three feedforward neural networks with four hidden layers and 50 nodes in each layer. Each feedforward network predicted either the temperature, pressure, or the logarithm of permeability, and each network was linked when calculating the physical constraints and boundary conditions for these physical parameters (Fig. 1). The observations considered the temperatures, pressures, and permeabilities retrieved from well locations. The inputs of the proposed PINN were the horizontal and vertical coordinates ($x$ and $z$) as the fundamental inputs, whereas additional compressed values of the temperature observations at wells (compressed by principal component analysis (PCA)) were used for another approach (Fig. 1). The PCA inputs are described in Sect. “PINN–TL”. For the activation function, we used the hyperbolic tangent function because a twice-differentiable activation function with respect to $x$ and $z$ is required, considering that the physical constraints contain second-order differential equations.

Fig. 1

Basic architecture of the PINN used in this study. Inputs used in Approaches 1 and 2 are listed in (a, b), respectively. Physics and Boundary indicate constraints for physics laws and boundary conditions, respectively

PINN–TL

Transfer learning leverages a pre-trained network to effectively solve a related problem (Pan and Yang 2010; Weiss et al. 2016). The weights and biases, called network parameters, of the pre-trained network are used as the initial values, and only some of the network parameters are updated by transfer learning with newly added data. Using this framework, PINN–TL incorporates prior knowledge into the pre-trained network, and the pre-trained network is then tuned by the observed logs (Fig. 2). An additional advantage of using transfer learning is that it improves the convergence of learning and representation of pre-trained features because the number of parameters to be updated is reduced. Transfer learning further reduces the overfitting of the PINN to the observations.

Fig. 2

Schematic of the physics-informed neural network with transfer learning (PINN–TL). The reference PINN framework used in this study is also shown

We incorporated transfer learning into the PINN framework in three steps. (i) First, multiple numerical models were generated for pre-training. The numerical models were then used to generate temperature, pressure, and permeability datasets. As the prior information contained uncertainties, the model parameters of the simulations were randomly sampled from uniform parameter distributions. (ii) Then, the network was pre-trained using the temperature, pressure, and permeability datasets of the numerical models. During training, data from the entire region were used. (iii) Subsequently, transfer learning was performed with the pre-trained network by training it on the observed temperatures, pressures, and permeabilities at the wells. In the transfer learning step, the network parameters were updated only in the last hidden layer and the output layer, whereas the network parameters in the other layers were fixed. The temperature, pressure, and permeability were predicted using the trained PINN. TensorFlow library version 2.9.1 (Abadi et al. 2016) was used to implement the proposed PINN–TL.

In the pre-training step, all grid data from the pre-training datasets were used. The training set used 90% of the pre-training datasets, and the remaining 10% were used as the validation set. The training set was used to train the network parameters, whereas the validation set was used to monitor the overfitting of the training set. For each analysis, the data used in the training and validation sets were randomly selected. In the transfer learning step, training was performed using well observations, with 90% of the well data near the surface as the training set and the remaining 10% of the deeper well data as the validation set. The optimization algorithm used was adaptive moment optimization (Kingma and Ba 2017), with 50,000 steps for pre-training and 20,000 steps for transfer learning.

We examined two PINN–TL approaches using different inputs. Approach 1 used coordinates ($x$ and $z$) as inputs, whereas Approach 2 used additional inputs of compressed temperature log values at wells (Fig. 1). In Approach 2, PCA was used to derive the compressed values of the temperature logs. PCA is a statistical procedure that determines the orthogonal axes, termed principal components, that maximize the variance of multidimensional observations. By selecting principal components with significant contributions, PCA can reduce the number of observations (i.e., dimensionality reduction) while minimizing the loss of information underlying the observations. This study used PCA to compress the dimensions of temperature observations, because the temperature logs of multiple numerical models can be regarded as multidimensional observations with the dimension being the number of observation points depending on the number of wells, depth, and spacing of observations. By using the compressed values as inputs, we expected that the features of multiple numerical data could be better trained. That is, the use of compressed values in addition to coordinates was expected to increase the number of explanatory variables linked to the outputs. In this study, all the temperature log data from the pre-training simulation models and the reference observations were compressed into eight dimensions using PCA. More specifically, a single numerical model of simplified (five well cases) and Ungaran settings with 250 and 150 temperature measurement points, respectively. Considering 201 numerical models (200 pre-training models and one reference model), the 201-by-${N}_{\text{point}}$ (${N}_{\text{point}}$ = 250 or 150) matrix was compressed into a 201-by-8 matrix using PCA. The compressed data was then used as input for Approach 2. Eight dimensions were chosen because the cumulative contribution ratio exceeded 99% in all cases and was considered sufficient to express the features of each model. As an alternative to PCA, other sophisticated methods, such as multidimensional scaling (Park et al. 2013; Hermans et al. 2015) could be used to select simulation datasets that resemble observed logs.

The neural network architecture generally influences the suitable number of training datasets. Therefore, different numbers of datasets were examined for the pre-training step. In the case of a limited number of pre-training datasets, 1–5 datasets were selected for use based on the similarity of temperature patterns with observed temperature logs. In a large dataset case, 200 datasets were used for pre-training, regardless of their similarity to the observed temperature logs. As default test cases, three and 200 pre-training datasets were used for Approaches 1 and 2, respectively.

Different numbers of pre-training datasets reflect the uncertainties of the prior information. A few datasets are created based on prior information with higher confidence, whereas many pre-training datasets are based on prior knowledge with higher uncertainty. In this study, when selecting 1–5 simulation datasets, we calculated the mean squared error (MSE) between the temperature logs of the target area and pseudo-observed temperatures of the numerical simulation models and adopted the simulation models with the smallest MSE as those with high similarity with the observed logs. Two geothermal settings were used to assess the proposed PINN–TL model, and an examination of the number of pre-training datasets was conducted using one of the geothermal settings. For another geothermal setting, the default numbers of the pre-training datasets were chosen based on the examination of the former setting.

The different inputs of the two approaches are similar to the framework of PINNs that deal with multi-fidelity data (Meng and Karniadakis 2020; Chakraborty 2021). Chakraborty et al. (2021) proposed pre-training a PINN with low-fidelity data and then updating the network with high-fidelity data by transfer learning. This approach has the same flow as Approach 1 in this study. Meng and Karniadakis (2020) used two different networks for low- and high-fidelity data. The outputs of the network for low-fidelity data were used as the inputs of a PINN for high-fidelity data, in addition to the inputs of the coordinates. The addition of inputs is similar to that in Approach 2. Although the additional inputs of Approach 2 are the compressed values after dimensional reduction, both Approach 2 and the approach by Meng and Karniadakis (2020) use additional inputs for high-fidelity outputs.

Prediction performance

As a measure of the PINN–TL prediction performance, we used the Mean Absolute Percentage Error (MAPE), which is expressed by Eq. (6).

$$\text{MAPE}=\frac{100}{N}\sum_{i=1}^{N}\left|\frac{{y}_{pred,i}-{y}_{ref,i}}{{y}_{ref,i}}\right|$$

(6)

Here, N is the number of prediction locations, ${y}_{ref,i}$ is the true value of the parameter (temperature, pressure, or log-transformed permeability), and ${y}_{pred,i}$ is the predicted value. To quantitatively evaluate the prediction error in each analysis, we randomly varied the initial values of the DNN parameters and positions satisfying the physical laws and performed training and estimation multiple times using the same data. For each method, we calculated the mean and standard deviation of the MAPE for 10 PINN analyses that converged by randomly varying the initial network parameters and locations at which the losses of the physics constraints are calculated (i.e., collocation points) and used them to evaluate the prediction accuracy of the method.

Numerical datasets

This study used two natural-state geothermal settings for the reference temperature, pressure, and permeability datasets and created synthetic logs from the reference data. The reference geothermal setting was based on Bjarkason et al. (2019), which has a simplified and regular permeability structure with typical features of fault-driven geothermal systems. Another setting was based on the Ungaran geothermal system in Central Java, Indonesia (Jatmiko et al. 2022). These natural-state geothermal models were simulated using TOUGH2 (Pruess 2004). For both settings, we calculated the natural-state temperature and pressure for approximately 110,000 years from the initial state.

Simplified setting

The target area of the first simplified setting was a 2D space with a depth of 1.6 km and a horizontal width of 2 km, assuming a geothermal system with a fault-controlled permeable region extending vertically (Fig. 3a). The hydrothermal fluid flowed through the vertical permeable region, representing fractures and faults, from the bottom of the analyzed area. It is impeded by an impermeable caprock above the permeable zone, which is assumed to consist of clay minerals (Fig. 3a). The impermeable body in the deepest part of the analyzed area assumes a basement structure (Fig. 3a). The permeable structure is typical of fault-driven geothermal systems in volcanic regions (NEDO 2000). This geothermal setting comprises five geological units with different permeabilities.

Fig. 3

Geological units of the geothermal system under a simplified and b Ungaran settings

The simulation model consisted of 8000 elements with a grid size of 20 m. We assumed that heat propagates from the bottom of the basement with a constant heat flux (0.13 W/m²) as the boundary condition. For the hot water flowing out from the bottom of the fracture zone, we set a mass flow rate [9.03 × 10^–6 kg/(m² s)] and a specific enthalpy (1.09 × 10⁶ J/kg) as the boundary conditions of the reference model. We set a constant temperature and pressure (15 °C, 1013.0 hPa) at the surface as the upper boundary conditions, and the lateral boundaries were closed. All analyzed areas were saturated with pure water. The reference permeability, natural-state temperature, and pressure distributions obtained from the simulation are shown in Fig. 4. For simplicity, we assigned the same values for all rock physical properties in each geological unit [density: 2500 kg/m³; porosity: 0.1; thermal conductivity: 2.5 W/(m·K); and specific heat: 1000 J/(kg·K)] (Bjarkason et al. 2019).

Fig. 4

Reference distributions for a and d permeability, b and e pressure, and c and f temperature used in this study. a–c used the simplified setting, and d–f used the Ungaran setting. Black lines represent the locations and depths of pseudo-wells

For synthetic observations, we considered several vertical wells drilled from the surface, and the temperatures, pressures, and permeabilities were acquired every 20 m. These synthetic wells were drilled to a depth of 1000 m and were located with constant horizontal spacing. The main analysis was performed assuming the case of five wells (Fig. 4); three and nine synthetic wells were considered for comparison (Fig. S1). In all cases, the wells did not reach either the basement or the vertical permeable zone representing fractures, as shown in Figs. 4 and S1.

Ungaran setting

We constructed the permeability structure of the Ungaran geothermal system based on Jatmiko et al. [48]. In the Ungaran geothermal system, the hydrothermal upflow zone is around the summit of Mt. Ungaran, where a fumarole appears on the surface. The heat source is presumably a young magmatic or active volcanic body just below the upflow zone. Magnetotellurics survey results suggest the existence of a caprock and a reservoir with an up-domed shape (Assiddiqy et al. 2021). The Ungaran setting covers a 2D space with a horizontal width of 6 km and an elevation of − 3000 to 2000 m (Fig. 3b). As shown in Fig. 3b, the boundaries of the surface layer, caprock, and reservoir were curvilinear, with the surface layer decreasing in thickness near the mountaintop and the reservoir exhibiting an up-domed shape. At the bottom of the model, we placed a deep fracture zone, which shows a high permeability, instead of a basement. The boundary between the deep fracture zone and overlying geological units was set as a linear horizontal line.

In the simulation, impermeable units were placed on both sides of the model to represent the lateral boundary conditions. A 10 km long and 5 km deep spatial region was divided into 5000 elements at 100 m intervals for analysis. Heat-flux boundary conditions were set at the bottom of the deep fracture and heat source. A heat flux was assigned to the bottoms of the heat source (2.1 W/m²) and deep fracture (0.12 W/m²). A constant surface temperature and pressure (30.0 ℃, 1013.0 hPa) were assigned for the analysis, and the groundwater level was assumed to be equal to the surface level. The permeabilities of the Ungaran setting and natural-state temperatures and pressures obtained from the simulation are shown in Fig. 4. The values of each physical property are listed in Table 1. At this location, we assumed that five vertical wells were drilled from the surface with temperature, pressure, and permeability profiles. These wells extended from the surface to a depth of 3000 m, and data was collected every 100 m at each well. The positions of the wells concerning each distribution are indicated by black lines in Fig. 4.

Table 1 Reference physical properties for each geological formation used in creating the Ungaran setting

Pre-training datasets

Multiple pre-training datasets were generated based on the models described in Sects. “Simplified setting” and “Ungaran setting”. The parameters for which the uncertainties were considered in the simplified and Ungaran geothermal settings are listed in Tables 1 and 2, respectively. Uniform distributions were used to represent uncertainties. Permeability structures and boundary conditions were provided, and hydrothermal simulations using TOUGH2 were conducted to obtain the natural-state temperatures and pressures corresponding to different conditions. Other physical properties of the simulation were held constant. The fundamental assumptions in creating permeability structures and boundary conditions were that (a) permeabilities at the shallow depth range and boundary conditions at the top and sides were accurately determined from existing well logs and geological and geophysical knowledge, and (b) those at depths deeper than the maximum depth of the wells were derived from expert interpretation of geological, geochemical, and geophysical data but possessed uncertainties. Geological knowledge may indicate the existence of basement and/or fault-controlled fractures, whereas geophysical data, such as electromagnetic and gravity observations, may indicate approximate locations, sizes, and physical properties at depths. Thus, multiple numerical models were constructed based on the approximate quantities of permeabilities, their structures, and boundary conditions, and the range of the approximate quantities reflected the uncertainties. None of the datasets of these pre-training numerical models were identical to those of the reference model.

Table 2 Numerical model parameters and uncertainties in the simplified setting

Examples of the pre-training datasets for the simplified and Ungaran settings are shown in Fig. 5. For the simplified setting, we assumed that the spatial extent and permeability of the surface layer, caprock, and reservoir were accurately known. Furthermore, uncertainties were considered regarding the locations and permeabilities of the fracture zone and basement in the simplified setting (Table 2). In the Ungaran setting, the surface layer, caprock, and upper boundary of the reservoir were known, whereas the permeabilities of the vertical and deep fracture zones, reservoir, and depth of the deep fracture zone were considered uncertain (Table 3). The strengths of the boundary heat sources were treated as uncertain in both scenarios. Fig. S2a–e shows the selected datasets with high similarities to the reference temperature logs for the simplified setting used to examine the influence of the number of pre-training datasets. For the Ungaran setting, three selected pre-training datasets (Fig. S2f–h) were used for Approach 1, reflecting the examination of the simplified setting.

Fig. 5

Examples of numerical datasets created for pre-training based on prior information by considering uncertainties: a–e simplified and f–j Ungaran settings. Ka–j, Pa–j, and Ta–j show permeability, pressure, and temperature distributions, respectively. Side boundaries of the Ungaran setting were not used in PINN and PINN–TL analyses and are not shown

Table 3 Numerical model parameters and uncertainties in the Ungaran setting

PINN–TL predictions

Simplified setting

The predicted permeabilities, pressures, and temperatures returned by the reference PINN framework and PINN–TL (Approaches 1 and 2) are shown in Fig. 6. In addition, Fig. 6 shows the difference between the predictions and the reference values. Loss function convergence for the training and validation datasets is shown in Fig. S3. As the validation loss function was almost constant after about 2000–10,000 epochs, the predictions in Fig. 6 were obtained from the trained network at 20,000 epochs. The influence of transfer learning is particularly apparent in the predicted permeabilities (Fig. 6Ka–Kc). The reference PINN successfully predicted the permeability structures of the surface, caprock, and reservoir, but not the high and low permeabilities of the fracture zone and basement (Fig. 6Ka and Ka-diff). The results indicated that the PINN could not predict the permeability structures where training wells were absent. In contrast, PINN–TL successfully predicted the permeability structures of the fracture zone and basement, along with the surface layer, caprock, and reservoir. This comparison demonstrates that the transfer learning of numerical datasets complements information at locations where well logs do not exist, especially at deeper and localized geologic units (Fig. 6Ka, Ka-diff, Kb, Kb-diff, Kc, and Kc-diff). Furthermore, high temperatures and pressures were predicted accurately by PINN–TL unlike the reference PINN (Fig. 6Pa, Ta, Pb, Tb, Pc, and Tc), especially at deep locations (Fig. 6Pa-diff, Ta-diff, Pb-diff, Tb-diff, Pc-diff, and Tc-diff).

Fig. 6

Distributions of predicted parameters for the simplified setting. (Ka), (Kb), and (Kc) are predicted permeability; (Pa), (Pb), and (Pc) are pressure; and (Ka), (Kb), and (Kc) are temperature. (Ka), (Pa), and (Ta) are predictions by the reference PINN. (Kb), (Pb), and (Tb) are predictions by Approach 1 of PINN–TL. (Kc), (Pc), and (Tc) are predictions by Approach 2 of PINN–TL. (Ka-diff–Tc-diff) are the differences between predictions and reference values (prediction—reference value). For Approach 1, the three selected pre-training datasets were used

To quantify the accuracy of the predictions, we calculated the MAPEs for the predicted temperatures, pressures, and logarithms of the permeabilities. Figures 7a, S4a, and S4d show the MAPEs in the interpolation depth region above 1000 m, in which the reference PINN exhibited lower or comparable predicted temperature and permeability MAPEs than those returned by PINN–TL (Fig. 7a). These trends in prediction performance were also observed when the number of wells was three or nine (Fig. S4a and S4d). The lower error of the reference PINN can be explained by the number of network parameters trained using the observed logs. In transfer learning, because most network parameters, except those near the output nodes, were fixed, the number of tunable network parameters became smaller than that of the reference PINN. The MAPE of the pressures returned by the reference PINN exhibited a larger or comparable value to those returned by PINN–TL (Fig. 7a). This may be because the pressure of the setting used in this study was nearly hydrostatic, and the spatial patterns of the pressure of the reference data were similar to those of the pre-trained models.

Fig. 7

Predicted errors (MAPEs) for each method, considering five wells for the simplified setting. MAPEs for a interpolated region shallower than 1000 m, b extrapolated region deeper than 1000 m, and c logarithm of permeability by geological formation. Vertical and error bars represent the means and standard deviations of the MAPEs of 10 individual runs of the reference PINN and PINN–TL (Approaches 1 and 2, respectively)

The prediction accuracies below 1000 m (extrapolation depth range) are shown in Fig. 7b, S4b, and S4e. Prominent decreases in the MAPEs were found in the predicted pressures and permeabilities, which were also confirmed by the spatial distribution of these predicted quantities at depth (Fig. 6). MAPE decreases by PINN–TL in the extrapolation depth range were also found when using three and nine wells (Fig. S4b and S4e). The MAPEs of the temperatures predicted by the PINN–TL were similar to those predicted by the PINN within the extrapolation depth range (Fig. 7b, S4b, and S4e). Comparing Approaches 1 and 2, Approach 2 provided the best predictions for permeabilities, whereas the MAPE of the predicted pressures was the lowest when Approach 1 was used.

Comparing the permeability prediction accuracy of the surface layer, caprock, and reservoir, the reference PINN showed the lowest MAPEs (< 0.1% in all cases), whereas the MAPEs of the predicted permeabilities at the basement were lower when PINN–TL was used (Fig. 7c, S4c, and S4f). These results also demonstrated that the PINN–TL approach was beneficial in the extrapolation zone where well logs were not present. Despite the use of the PINN–TL, MAPEs at the fracture zone did not decrease significantly compared with those at the basement. This limited improvement in the MAPEs at the fracture zone was caused by a small inaccuracy in the horizontal location of the fracture zone. A slight difference in the horizontal locations of the fracture zone significantly increased the MAPE. The MAPEs returned by Approach 2 of the PINN–TL were slightly smaller than those of Approach 1.

Ungaran setting

Examples of the predictions for the Ungaran setting and the prediction error are shown in Fig. 8. Fig. S5 shows the loss function progression over epochs. The predictions by the reference PINN in Fig. 8 were obtained after 20,000 epochs. The predictions for Approaches 1 and 2 in Fig. 8 were from 1900 and 600 epochs, respectively, to minimize the loss function of the validation dataset (Fig. S5). Similar to the simplified setting prediction, the predicted permeability structure using the reference PINN indicated the presence of a surface layer, caprock, and reservoir, but not the vertical fracture zone, deep fracture zone, or heat source (Fig. 8Ka). Unlike the simplified setting, the permeability structures of the shallow geologic units (surface layers, caprock, and reservoir) in which the wells are located were not accurately predicted in the Ungaran model (Fig. 8Ka). The pressures predicted at depth by the reference PINN were underpredicted, and the detailed distribution of the temperatures predicted by the reference PINN did not reflect the geothermal features of the high temperatures extending along the fracture zone (Fig. 8Pa, Pa-diff, Ta, and Ta-diff). In contrast, the predictions by PINN–TL (Approaches 1 and 2) were more consistent with the reference temperatures, pressures, and permeabilities (Figs. 8Kb–Tc and Ka-diff–Tc-diff). In particular, the permeabilities at the vertical fracture zone, deep fracture zone, and heat source were predicted by both PINN–TL approaches (Fig. 8Kb and Kc). The elevation of high temperatures towards the shallow units by the upflow fracture zone was also more clearly predicted by PINN–TL (Fig. 8Tb and Tc).

Fig. 8

Distributions of predicted parameters for the Ungaran setting by each method. (Ka), (Kb), and (Kc) are predicted permeability; (Pa), (Pb), and (Pc) are pressure; and (Ta), (Tb), and (Tc) are temperature. Predictions by PINN without pre-training (Ka, Pa, Ta), PINN–TL Approach 1 (Kb, Pb, Tb), and PINN–TL Approach 2 (Kc, Pc, Tc). The differences between predictions and reference values are labeled as (Ka-diff–Tc-diff). Three numerical models were used for pre-training Approach 1

Figure 9 shows the MAPEs for the shallow (elevation $\ge$ − 1000 m), deep (elevation < − 1000 m), the entire region, and each geological unit of the Ungaran model. Unlike the predictions in the simplified setting, the MAPEs of the temperatures and permeabilities predicted by the reference PINN in the shallow region were larger than those predicted by PINN–TL (Fig. 9a). In fact, the MAPEs at the surface layer, caprock, and reservoir of the predicted permeabilities had smaller errors when PINN–TL was used (Fig. 9d). The MAPEs of the pressure predicted in the shallow region by the reference PINN and PINN–TL were similar, as observed in the simplified setting, but those of the reference PINN had a larger standard deviation than those of PINN–TL (Fig. 9a). We interpret that the more complex permeability structure of the Ungaran setting and the limited number of wells hinder accurate and robust predictions of permeability, temperature, and pressure, even in shallow regions.

Fig. 9

Predicted errors of the Ungaran setting. MAPEs for the a region with ≧ − 1000 m elevation, b region with < − 1000 m elevation, c entire analyzed region, and d logarithm of permeability for each geological unit. Vertical and error bars represent the means and standard deviation of MAPEs of 10 individual runs of the reference PINN and PINN–TL (Approaches 1 and 2), respectively

The MAPEs of the pressures and permeabilities of the deep region using PINN–TL were smaller than those of the reference PINN (Fig. 9b). The MAPEs of permeabilities also showed that PINN–TL reduced the errors in the vertical fracture zone, deep fracture zone, and heat source (Fig. 9d). Compared with the predictions of the simplified setting (Fig. 7), the MAPE for the predicted permeabilities at the fracture zone was significantly reduced using PINN–TL (Fig. 9d), likely because of the lower uncertainty in the location of the fracture zone in the pre-training datasets of the Ungaran setting. Only temperatures in the deep region predicted by PINN–TL showed a similar or slightly higher MAPE than those predicted by the reference PINN (Fig. 9b). This indicates that the reference PINN exhibits good accuracy when limited to temperature predictions at depth. Over the entire analysis range, the temperatures, pressures, and permeabilities predicted by PINN–TL exhibited a smaller MAPE than those of the reference PINN, indicating that PINN–TL was effective in improving the prediction accuracy (Fig. 9c).

The two PINN–TL approaches did not exhibit any significant differences in the MAPEs in the shallow region (Fig. 9a). In the deep and entire regions, Approach 1 had a smaller MAPE in predicting permeabilities, but the prediction by Approach 2 had a smaller error for temperature and pressure (Fig. 9b and c). Nevertheless, considering the predictions in both settings, we conclude that the difference in accuracy between the approaches was insignificant.

Characteristics of the proposed approaches

Influence of the pre-training data

Figure 10a–d show examples of the predicted permeabilities of the simplified setting using Approach 1 when one, three, five, and 200 pre-training datasets were used. When the numbers of pre-training datasets were one, three, and five, the high and low permeabilities at the fracture zone and basement were well identified (Fig. 10a–c). In contrast, high permeabilities representing the fracture zone were not predicted when the number of pre-training datasets was 200, despite predicting the presence of low permeability in the deep basement (Fig. 10d). Hence, the permeabilities of the fracture zone and basement were more accurately predicted by Approach 1 when the number of pre-training datasets was small.

Fig. 10

Examples of predicted permeability distribution in the simplified setting for different numbers of pre-training datasets. Predictions with a one, b three, c five, and d 200 pre-training datasets in Approach 1, and predictions with e three, f five, and g 200 pre-training datasets in Approach 2

The uncertainty in the permeability structures of the pre-training datasets could have affected the predicted permeability structures. In this case of Approach 1 with one, three, or five pre-training datasets, the prior information had low uncertainty, and the predicted permeabilities of the fracture zone and basement were predicted based on well logs and a limited pre-training dataset (Fig. 10a–c). With 200 datasets in Approach 1, the high permeability in the fracture zone was obscured because of uncertain prior information regarding its horizontal locations (Fig. 10d). Notably, even though one dataset was used for pre-training, the predicted quantities were not identical to those of the pre-training dataset, especially for shallow layers with well logs (surface layer, caprock, and reservoir), which were tuned by the observation well logs through transfer learning.

Predictions by Approach 2 with three, five, and 200 pre-training datasets are shown in Fig. 10e–g. The prediction could not be obtained with one pre-trained dataset. In contrast with Approach 1, Approach 2 successfully predicted the permeability of each geological unit when 200 pre-training datasets were used, whereas the predicted permeability of each geological unit was unclear when a smaller number of pre-training datasets were used (Fig. 10e–g). Because Approach 2 uses additional inputs from the compressed values of the observed logs, many pre-training datasets are prerequisites to train the relationship between the inputs and outputs.

Mismatch reduction

The prediction errors of Approaches 1 and 2 were compared with the mismatches of temperature, pressure, and permeability datasets of the pre-training and the reference models. The average MAPEs between the pre-training datasets in the entire analyzed region with the simplified and Ungaran settings are listed in Table 4. For comparison, the MAPEs in the entire region of the simplified setting were 2.3% (temperature), 1.6% (pressure), and 2.5% (logarithm of permeability) in Approach 1, and 3.1% (temperature), 2.9% (pressure), and 1.6% (logarithm of permeability) in Approach 2. For the Ungaran setting, the MAPEs in the entire region were 4.9% (temperature), 5.8% (pressure), and 1.1% (logarithm of permeability) in Approach 1, and 3.4% (temperature), 5.3% (pressure), and 1.7% (logarithm of permeability) in Approach 2.

Table 4 Average MAPEs of the pre-training datasets

In comparison, most of the MAPEs of the predictions by both PINN–TL approaches were larger than those of the average MAPE of the three selected models with the highest similarity of temperature logs to the reference model. As these selected models were used as default pre-training models for Approach 1, it may not be effective to constrain the temperatures, pressures, and permeabilities of the pre-training datasets. Compared with the average MAPE of the 200 pre-training datasets, the MAPE of the predicted temperatures returned by Approach 2 was smaller; thus, the temperatures were better constrained by Approach 2. In contrast, the MAPEs of the predicted pressures by both PINN–TL approaches were larger than the average of the 200 pre-training datasets. This may be because the pressures of the pre-training models in the settings used in this study were almost hydrostatic and had few mismatches. The MAPEs of the logarithm of the predicted permeabilities were similar to the averages of the 200 pre-training datasets. Despite the insignificant difference in MAPEs, the permeability structures were better predicted by PINN–TL. For example, the location of the fracture zone had high uncertainty in the 200 pre-training datasets of the simplified setting, whereas the predicted permeabilities by Approach 2 constrained the location well (Figs. 5Ka–Ke and 6Kc). Such locations are important for interpreting the geological background of the geothermal model; thus, we conclude that Approach 2 is also effective in constraining the permeabilities.

Conclusions

Machine learning including PINNs has shown effectiveness for modeling geothermal systems, however modeling inaccuracy, especially at extrapolation depths beyond well ranges has been a major concern. To lessen this drawback, this study proposed using physics-informed neural networks along with transfer learning (PINN–TL). Expert knowledge was incorporated as multiple numerical models that accounted for uncertainties. The neural networks were pre-trained on the temperatures, pressures, and permeabilities from the numerical models, followed by transfer learning using well observations. Two approaches were considered with different types of inputs and a suitable number of pre-training datasets.

The main outcomes of this study are as follows:

• The applications of both proposed transfer learning (PINN-TL) approaches to geothermal settings showed effectiveness in terms of predicting the characteristics of the target geothermal systems that were not revealed by observations but were informed by prior knowledge with given uncertainties.

• Although the proposed approaches showed similar prediction accuracy, Approach 2 is more practical and effective than Approach 1 because it can better constrain temperatures and permeabilities from prior information on the subsurface structure and physical properties with uncertainties.

Well observations are often limited to a certain depth, and spatial density is insufficient to predict localized and deep geological units; thus, the proposed approaches can complement well observations with expert knowledge, especially in unexplored depths and horizontal ranges.

One of the limitations of the presented method is that permeability measurements along wells are required; however, most geothermal fields generally do not have dense measurements of permeabilities along wells. The development of more practical methods for permeability estimation along wells would be demanded. Alternatively, incorporating other geophysical observations, such as electromagnetic observations as suggested in Ishitsuka and Lin (2023), could be useful to supplement the limited information on permeabilities.

As PINNs are a developing framework, there are still limitations (Muther et al. 2022). For example, the framework requires high-quality observations, although a smaller number of observations are required compared to conventional data-driven neural networks. A high computational burden is also a concern of PINNs. However, ongoing improvements of PINNs would mitigate the limitations and further enhance the effectiveness of the proposed PINN–TL. For example, the development of methods to handle multi-fidelity data would allow analysis of less accurate data supplemented by more accurate data. In addition, the development of parallelization techniques, such as the use of distributed learning (Shukla et al. 2021), may reduce the computational load.

A conventional method for constructing a natural-state geothermal model involves calibrating a numerical simulation to match observations based on expert interpretations of subsurface structures and physical quantities. This process appears similar to the proposed approaches, which allow the consideration of expert interpretation in the transfer learning framework, followed by matching well observations. This similarity could facilitate widespread application in the modeling of various geothermal fields. As this proposed framework of transfer learning applies to modeling other physical phenomena with limited observations, it is expected to find applications in a variety of research fields.