Simple formula learned via machine learning for creep rupture life prediction of high-temperature titanium alloys

Dataset and data preprocessing

The high-temperature titanium alloy dataset was manually collected from the literature ^[34–43]. It consists of 88 samples, each described by 25 features, including 15 chemical compositions, nine heat treatment processes, and two experimental conditions, with data distribution as shown in Supplementary Figure 1. We apply a logarithmic transformation to the target property, creep rupture life, i.e., taking $$ \log(\text{creep rupture life}) $$ as the output of surrogate models. All the 25 features are normalized using the min-max method, as given by

where $$ x $$ represents a specific feature value for a given sample, and $$ x_{max} $$ and $$ x_{min} $$ are the maximum and minimum values of the same feature across all the samples, respectively.

The extrapolation ability is crucial for the surrogate models and is examined by dividing the 88 samples into training and testing sets. That is, models are trained with alloys of short-term creep rupture life and tested on those with long-term creep rupture life. To ensure the examination of extrapolation, the division includes seven categories considering the value of creep rupture life. The first division has 35 samples possessing creep rupture life (RT) $$ > $$ 100 h as the testing data and the left 53 as training data. Similarly, another six divisions are determined with different RT values, i.e., RT $$ > $$ 150 h (29 testing samples), RT $$ > $$ 200 h (22), RT $$ > $$ 300 h (14), RT $$ > $$ 400 h (11), RT $$ > $$ 700 h (8), and RT $$ > $$ 1, 000 h (6).

ML models

The MLR and SISSO are both employed to learn formulas that relate composition, processing, and testing conditions to creep rupture life. In SISSO, arithmetic operations are first applied to the primary features to construct a feature library containing a large number of candidate features. In total, 12 operations are used: Addition ($$ + $$), Subtraction ($$ - $$), Multiplication ($$ \times $$), Division ($$ \div $$), Exponentiation ($$ e^x $$), Squaring ($$ x^2 $$), Cubing ($$ x^3 $$), Logarithm [$$ \log(x) $$], Square root ($$ \sqrt{x} $$), Cube root ($$ \sqrt[3]{x} $$), Inversion ($$ x^{-1} $$), and Absolute value ($$ |x| $$). Then the most important subset is selected by independently evaluating each candidate for its correlation with the target. A formula of the specified dimension can be obtained by fitting on the sparsified library. The form of the formulae learned by SISSO is given by

where $$ x_i $$ is the new feature generated by combining basic features and arithmetic operations. The number of $$ x_i $$ determines the dimension of formulae; for example, the formula with only one new feature is termed 1D.

As a comparison, five black-box models, the RF, eXtreme Gradient Boosting (XGBoost), SVR, convolution neural networks (CNN), and automated ML (Autogluon), are used to build the surrogate models. The RF model optimizes four hyperparameters: n_estimators, max_depth, min_samples_split, and min_samples_leaf. The ranges for these parameters are 10-100 (step = 10), 10-15 (1), 2-5 (1), and 2-5 (1), respectively. For the XGBoost model, we tune eight hyperparameters, including n_estimators, max_depth, min_child_weight, gamma, subsample, colsample_bytree, reg_lambda, and reg_alpha, with ranges of [160, 170], 5-7 (1), 2-4 (1), 0.001-0.1 (0.01), 0.7-0.9 (0.1), 0.7-0.9 (0.1), [2, 3, 5, 8], and [0, 0.1]. For the SVR model, we optimize two hyperparameters, C and epsilon, with ranges of 2-8 (1) and 0.01-1 (0.01), respectively. All models above utilize grid search. The CNN comprises two convolutional layers with eight and 16 kernels of the same size $$ 2\times2 $$, followed by a flattening layer and a fully connected layer with 128 dimensions. The input data is standardized to a uniform shape of $$ 6\times6 $$, and the final output is directed to a single neuron for regression tasks. Autogluon is a toolkit based on existing ML algorithms, leveraging Bayesian optimization to achieve automated basic model selection and hyperparameter tuning. It utilizes a stacking strategy to train an ensemble model on the predictions of these basic models for final prediction ^[24,44]. We employ least absolute shrinkage and selection pperator (Lasso) regression, a linear model regularized with a penalty term, to select key features.

The performance of the surrogate models is evaluated using the mean absolute error (MAE) and the mean squared error (MSE) metrics, as given by

where $$ y_i $$ and $$ \hat{y}_i $$ are the ground truth and predicted value of the $$ i $$th sample, respectively, and $$ n $$ is the number of samples.

Surrogate models for creep rupture life prediction

The surrogate models based on RF, XGBoost, SVR, CNN, Autogluon and MLR are trained with the training data and tested with the testing data. Figure 1 shows the extrapolation performance of the six models; the division with creep rupture life (RT) $$ > $$ 100 h as testing data is used as an example. Interestingly, the MLR model in Figure 1F performs the best compared to other black-box models, as indicated by the consistent distribution of both training and testing data points and the lowest MAE and MSE for testing data. By contrast, the models in Figure 1A-D suggest severe over-fitting. While the CNN model in Figure 1E seems moderate, it is too complex due to the use of multilayer networks and convolution kernels. Figure 2 shows the extrapolation capability of the MLR model in the seven divisions with different training and testing data. For both MAE and MSE, it is evident that the MLR model and the CNN model outperform the others in almost all the seven divisions. The low errors on testing data indicate that our surrogate models trained on alloys with short-term creep rupture life can predict those with long-term creep rupture life. This is important in practice as merely measuring the creep rupture life of low values is much time- and cost-effective. Considering that the MLP model can provide formulae for predicting creep rupture life and has performance comparable to the CNN model, we will focus on the MLP model in the following sections. It should be noted that the MLP model utilizes all the 25 features as inputs; this will inevitably lead to a quite complex formula, which still shows a black-box-like character. This then motivates us to reduce the features and obtain a more simple MLP model.

Figure 1. Extrapolation performance of the six surrogate models on both training and testing data, the data division with RT $$ > $$ 100 h (35 testing points and 53 training points) is used as an example. (A) RF; (B) XGBoost; (C) SVR; (D) Autogluon; (E) CNN; and (F) MLR. RT: Creep rupture life; RF: random forest; XGBoost: eXtreme Gradient Boosting; SVR: support vector regression; CNN: convolution neural networks; MLR: multiple linear regression.

Figure 2. Comparison of model (based on 25 features) performances on different testing datasets. (A) MSE and (B) MAE. MSE: Mean squared error; MAE: mean absolute error.

Reducing features for multiple-linear model

The Lasso regression is used to determine the key features. Lasso is an integration of a linear model with a regularized term. A key parameter is the coefficient ($$ \lambda $$) of the regularized term and its increase will decrease the coefficient of each independent variable. All the coefficients are limited to zero once $$ \lambda $$ exceeds a critical value. The more rapid the coefficient becomes to zero, the less important the variable is. Taking the testing set with RT $$ > $$ 100 h as an example, the Lasso model is trained using the training data of 53 samples. Figure 3A shows the change of each feature coefficient with $$ \lambda $$. The coefficient of the feature, 2nd_aging_temperature, is reduced to zero very quickly while those for V and Fe become zero the most slowly.

Figure 3. Feature selection via Lasso and model performance with reduced features, the data division with RT $$ > $$ 100 h is presented as an example. (A) Coefficient change of each of the 25 features with $$ \lambda $$; (B) Model bias (binomial deviance) as a function of $$ \lambda $$, the red points represent the mean value and the associated error bars are the standard deviation from multiple cross-validations; (C) The performance of the Lasso model with reduced features (11). Lasso: Least absolute shrinkage and selection pperator; RT: creep rupture life.

Figure 3B depicts the relationship between $$ \lambda $$ and binomial deviance, which is interpreted as the deviation between predictions and actual values under a specific $$ \lambda $$. The left dashed vertical line around 0.001 corresponds to the Lasso model with the lowest error for the training data of 53 samples. The right dashed line means that at the corresponding $$ \lambda $$, all the coefficients are zero. Based on the results from Lasso, in total, 11 features give rise to the optimal prediction model. Figure 3C shows the performance of the Lasso model on the training and testing data, both closely aligned along the diagonal line. The errors slightly increase compared to Figure 1F, indicating that the removal of 14 features by Lasso has little effect on the performance of the linear model. The formula resulting from Lasso is as follows:

A similar Lasso-based feature selection procedure is applied to the other six datasets with different divisions. For each dataset, we determine about ten features for an optimal linear model with the lowest error. To ensure the surrogate models and formulae can be generalized to different data division scenarios, we use the intersection of the features from the seven data sets for further modeling. The intersection has nine features: solution_time, C, Ni, Nb+Ta, V, Fe, Sn, test_temperature, and test_stress. Using these nine features, we retrain the MLR models and examine their performance for the seven data sets, as shown in Figure 4. All the models still have superior performance in both training and testing data, confirming the robustness of the nine features for extrapolation.

Figure 4. Extrapolation performance of the MLR models with nine features on the seven data divisions. (A) RT $$ > $$ 100 h; (B) RT $$ > $$ 150 h; (C) RT $$ > $$ 200 h; (D) RT $$ > $$ 300 h; (E) RT $$ > $$ 400 h; (F) RT $$ > $$ 700 h; and (G) RT $$ > $$ 1, 000 h. MLR: Multiple linear regression; RT: creep rupture life.

The feature, solution_temperature, is removed due to the very small coefficient in all seven data divisions. In addition, it is noted that of the nine features, six are the contents of elements. To further simplify the MLR models, we transform the six elemental contents to a composition-based feature, the Molybdenum equivalent ($$ [Mo]_{eq} $$), a well-known attribute used for the design of titanium alloys ^[45]. We reconstruct the MLR models using the three features: i.e., $$ [Mo]_{eq} $$, test_temperature, and test_stress. Using the dataset with RT $$ > $$ 100 h as an example, the calculation of $$ [Mo]_{eq} $$ and the learned formula are given by

Figure 5A-G shows the performance of the new MLR models. Compared to Figure 4, there is a slight degradation in model performance. This suggests that even with three features, we can obtain accurate formulae for extrapolated prediction of creep rupture life of high-temperature titanium alloys. To demonstrate the superiority of the MLR models, we also retrain the five black-box models for comparison, as shown in Figure 6. It is found that in all seven data division scenarios, MLR models always exhibit the lowest MSE and MAE metrics. An interesting observation is that the performance of MLR consistently outperforms CNN in Figure 6, whereas this is not the case in Figure 2. This can be attributed to the fact that the CNN model can generate information-rich features by the use of convolutional kernels on high-dimensional feature space; however, the only three retained features herein may not be adequate for training a robust CNN model.

Figure 5. Extrapolation performance of the MLR models with three features on the seven data divisions. (A) RT $$ > $$ 100 h; (B) RT $$ > $$ 150 h; (C) RT $$ > $$ 200 h; (D) RT $$ > $$ 300 h; (E) RT $$ > $$ 400 h; (F) RT $$ > $$ 700 h; and (G) RT $$ > $$ 1, 000 h. MLR: Multiple linear regression; RT: creep rupture life.

Figure 6. Comparison of model (based on three features) performances on different testing data sets. (A) MSE and (B) MAE. MSE: Mean squared error; MAE: mean absolute error.

Simple formulae for creep rupture life prediction

The MLR algorithm assumes a linear relationship between independent and dependent variables, and the resulting models (formulae) with three features demonstrate optimal performance in extrapolated predictions. However, there may exist nonlinearity in the high-temperature titanium alloy creep data, which deserves further exploration for improved formulae. The SISSO algorithm is employed to search for better formulae, which is realized by combining the three selected basic features with arithmetic operations. Such combinations may capture the possible nonlinear patterns behind the data and provide highly accurate formulae ^[46].

We apply SISSO to the seven datasets and identify the corresponding optimal 2D (two variables, $$ x_1 $$ and $$ x_2 $$) and 3D (three variables, $$ x_1 $$, $$ x_2 $$ and $$ x_3 $$) formulae for testing data, as listed in Table 1. The dimension can be further expanded for more accurate formulae, but we herein only consider the 2D and 3D ones that have relatively simple forms. It is seen that for different datasets, the 3D formulae do not necessarily outperform the 2D ones. The optimal formulae for each data set are highlighted in bold. All the formulae from SISSO have lower MSE and MAE than those from MLR models. The model improvement is particularly prominent for the data set with RT $$ > $$ 400 h (RT400-Test), where the MSE and MAE are reduced by approximately 44% and 27%, respectively.

Note that although nonlinear arithmetic operations are involved in generating new features, the optimal 2D formulae we obtain contain only addition and the 3D formulae include only addition and exponentiation. Moreover, for all the seven datasets, the 2D formulae have the same $$ x $$ and the only differences lie in the coefficients. The three 3D formulae in bold have errors merely about 0.01 lower than those of the 2D formulae. To obtain a unified formula for all these seven datasets, we average the coefficients of the 2D formulae for the seven datasets and the corresponding formula is as below

We examine the efficacy of the above formula on the seven datasets, and the model performance is compared to optimal 2D and 3D formulae (bolds in Table 1) and the MLR models [Figure 7]. Figure 7 shows that the unified formula performs similarly to those from SISSO and better than those from MLR. However, it should be pointed out that the formulae from SISSO and MLR change with different training and testing data. By contrast, the final formula in Equation (8) has a fixed form and can be preferred by the materials community.

Figure 7. Comparing the SISSO and MLR-based formulae to the "mean" formula (based on three features) on different testing datasets. (A) MSE and (B) MAE. SISSO: Sure independence screening and sparsifying operator; MLR: multiple linear regression; MSE: mean squared error; MAE: mean absolute error.

Traditionally, the TTP extrapolation methods have been used for predicting the long-term creep rupture life by fitting the measured short-term rupture life. These methods usually only consider the test_stress and test_temperature as inputs; thus, they are limited to a small subset of alloys where adequate experimental data are available. Here, in addition to test conditions, the learned formula also involves the composition-based $$ [Mo]_{eq} $$ that often indicates the phase transition and stability of titanium alloys. As a result, it allows us to apply the unified formula to titanium alloys of various chemistries. Additionally, the learned formula considering $$ [Mo]_{eq} $$ can be combined with optimization algorithms to not only further facilitate the design of new alloys with improved creep properties ^[47], but also aid in the clarification of the deformation mechanisms. It is intriguing to see that the learned simple formula outperforms the five black-box surrogate models, including the advanced automated ML and convolutional neural networks. The reason may lie in the linear-like trend between test conditions and creep rupture life for a given alloy ^[16,48], as generally the black-box algorithms are more flexible and often provide more accurate prediction models. For different data, various algorithms should be tried and then the proper one can be selected. That is, there is no universal algorithm for all kinds of data. Previous studies have also shown a similar phenomenon that the linear models perform better than black box models in extrapolation tasks ^[49,50].

Authors' contributions

Made substantial contributions to conception and design of the study and performed data analysis and interpretation: Wang P, Zhao S, Zhou C, Yuan R

Performed data acquisition and provided administrative, technical, and material support: Fan J, Tang B, Li J, Yuan R

Availability of data and materials

The data and codes are uploaded to the author's github repository: https://github.com/nwpuai4msegroup/creep-rupture-life-prediction.

Financial support and sponsorship

This work was financially supported by the National Key Research and Development Program of China (2021YFB3702604).

Conflicts of interest

All authors declared that there are no conflicts of interest.

Ethical approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Copyright

© The Author(s) 2024.