Multi-step policy evaluation for adaptive-critic-based tracking control towards nonlinear systems

1. INTRODUCTION

In practical engineering applications, the controller design should not only meet the basic performance requirements but also need to further improve the control effect and reduce costs ^[1–4]. However, real industrial systems are often complex nonlinear systems ^{[5, 6]}. Solving the Hamilton-Jacobi-Bellman (HJB) equations is an unavoidable obstacle when designing optimal control policies for nonlinear systems ^[7]. Considering that the analytical solution of the HJB equation is difficult to obtain, adaptive dynamic programming (ADP) based on the actor-critic framework approximates the solution of the HJB equation by the iteration^{[8, 9]}. The ADP algorithm exhibits strong adaptive and optimization capabilities by incorporating the advantages of reinforcement learning, neural networks, and dynamic programming ^{[10, 11]}. Therefore, the ADP algorithm has a satisfactory performance in solving the HJB equation. Through continuous development, the ADP algorithm has become one of the key methods for solving optimal control problems of nonlinear systems ^[12–14]. In terms of the iterative form, the ADP algorithms can be categorized into value iteration (Ⅵ) and policy iteration (PI) ^{[15, 16]}. PI has faster convergence but requires an initial admissible control policy ^[17]. However, it is difficult to obtain an initial admissible control policy for nonlinear systems. VI does not require an initial admissible control policy but has a slower convergence rate ^[18].

In the industry, we often need to solve more complex tracking problems for nonlinear systems ^[19–24], including the consensus tracking problem for multi-agent systems ^[22] and the output tracking control for single-agent systems ^[25]. In fact, the regulation problem can be seen as a simplified special case of the tracking problem. To overcome these challenges, some scholars have started to develop optimal learning control algorithms based on the ADP framework for solving tracking problems ^[26]. Currently, tracking control algorithms based on the ADP framework can be categorized into three groups. The first class of tracking control algorithms constructs the augmented system with respect to the original system and the reference trajectory and then builds the cost function based on the state vector and the control input of the augmented system ^{[27, 28]}. Although this type of tracking control algorithm has a well-established theory, it cannot completely eliminate tracking errors. The second class of tracking control algorithms utilizes the square of the tracking error at the next moment to build a novel utility function ^[19]. With the novel utility function, the tracking error can be completely eliminated. However, the second class of tracking control algorithms is currently only applicable to affine nonlinear systems with known system models, which greatly reduces the application value of the algorithm. The third type of tracking control algorithm transforms the tracking problem into the regulation problem by constructing a virtual steady control ^[29]. Thanks to the steady control, the third type of tracking control algorithm also completely eliminates tracking errors and is suitable for nonlinear systems. It should be emphasized that solving the steady control imposes a large computational burden on the algorithm.

Therefore, we expected that a speed-up mechanism could be designed for the tracking control algorithm with steady control to improve the convergence speed. Inspired by the idea of eligibility traces, Al-Dabooni et al. designed an online n-step ADP algorithm with higher learning efficiency^[30]. Ha et al. introduced a relaxation factor in the framework of the offline ADP algorithm^[31]. By adjusting the value of the relaxation factor, the convergence rate of the cost function could also be increased or decreased. Zhao et al. designed an incremental ADP algorithm by constructing a new type of cost function, which showed higher learning efficiency in solving zero-sum game problems for nonlinear systems^[29]. Luo et al. successfully combined the advantages of VI and PI by designing a multi-step policy evaluation mechanism^{[17, 25, 32]}. The convergence speed of the algorithm was greatly improved by increasing the policy evaluation step, which did not require admissible control policies.

In this context, a multi-step adaptive critic tracking control (MsACTC) algorithm is established by introducing a multi-step policy evaluation mechanism into the steady tracking control algorithm. Compared to the general tracking control algorithm with steady control, the MsACTC algorithm exhibits higher learning efficiency as the evaluation step increases. Furthermore, the convergence proof of the MsACTC algorithm is given. The MsACTC algorithm is successfully implemented through neural networks and the least square method. Finally, we verify the optimization ability and learning efficiency of the proposed algorithm through simulations of two nonlinear systems. Table 1 explains the meaning of the mathematical symbols used in this article.

2. PROBLEM DESCRIPTION

Consider such a class of discrete-time nonlinear systems

where $$ {x_k} \in {\mathbb{R}^s} $$ represents the state vector, and $$ {u({x_k})} \in {\mathbb{R}^m} $$ represents the control input. $$ F( \cdot , \cdot ) $$ is the unknown system function. In addition, we propose two assumptions to describe equation (1) more specifically.

Assumption 1$$ x(k) $$ and $$ u({x_k}) $$ are observable, and there exists a continuous control policy on $$ \Omega $$ such that the system (1) is asymptotically stable.

Assumption 2$$ F( \cdot , \cdot ) $$ is smooth differentiable with respect to its arguments.

The reference trajectory can be denoted as

where $$ H( \cdot ) $$ denotes the known trajectory function. The tracking error $$ {e_k} $$ is defined as

We assume that there exists a steady control $$ v({r_k}) $$ satisfying the following equation:

The tracking control policy $$ \mu ({e_k}) $$ is denoted as

According to equations (1)-(5), we can obtain the following error system:

The cost function is defined as

where $$ U\left( {{e_k}, \mu ({e_k})} \right) = x_k^{\mathsf{T}}Q{x_k} + \mu^{\mathsf{T}} {({e_k})}R\mu ({e_k}) $$ is the utility function. $$ Q $$ and $$ R $$ are positive definite matrices. We need to minimize the cost function by continuously adjusting the tracking control policy. The optimal cost function and the optimal control policy are expressed as follows:

and

So far, the tracking problem of the system (1) has been successfully transformed into the regulation problem of the system (6).

3. MULTI-STEP ADAPTIVE CRITIC TRACKING CONTROL ALGORITHM

It should be noted that equation (8) is the HJB equation, and its analytical solution is difficult to obtain directly. In addition, since the system function $$ F(\cdot ) $$ is unknown, the steady control cannot be directly obtained by solving equation (3) either. Therefore, the MsACTC algorithm is designed to overcome these challenges.

4. IMPLEMENTATION OF ALGORITHM

To implement the algorithm, three neural networks are constructed, including a model network (MN), a critic network (CN), and an action network (AN). Figure 1 illustrates the overall structure of the MsACTC algorithm. It should be noted that the training of the MN is carried out independently. The training of the CN and the AN need to cooperate with each other iteratively. It should be noted that the implementation approach used in this paper is similar to that in ^[22].

Figure 1. The structure diagram of the MsACTC algorithm. MsACTC: Multi-step adaptive critic tracking control.

The MN is used to estimate the dynamics of the system (1) and to solve the steady control $$ v({r_k}) $$. Since the MN is required for the training process of the CN and the AN, we should first complete the training of the MN. The output of the MN can be expressed as

where $$ {\bar x_k} = {\left[ {x_k^{\mathsf{T}}, u^{\mathsf{T}}{{({x_k})}}} \right]^{\mathsf{T}}} $$. $$ w_1^{(j)} \in {\mathbb{R}^{(s + m) \times {l_m}}} $$ and $$ w_2^{(j)} \in {\mathbb{R}^{{l_m} \times s}} $$ denote the weight vectors. $$ \beta _1^{(j)} \in {\mathbb{R}^{{l_m}}} $$ and $$ \beta _2^{(j)} \in {\mathbb{R}^{s}} $$ are bias vectors. $$ l_m $$ denotes the number of neurons in the hidden layer for the MN. $$ j $$ denotes the training round index.

In order to make the MN with higher recognition accuracy, we utilize the neural network toolbox in Matlab to train the MN. The training parameter settings are given in Section 5.

After the MN is trained, according to equation (4), we can obtain the steady control $$ {v({r_k})} $$ by solving the following equation:

The CN is used to approximate the cost function, and its output can be expressed as

where $$ {\omega ^{(i)}} \in {\mathbb{R}^{{l_c}}} $$. The activation function $$ \phi ({e_k}) = {[{\phi _1}({e_k}), \cdots , {\phi _{{l_c}}}({e_k})]^{\mathsf{T}}} $$. $$ l_c $$ denotes the number of neurons in the hidden layer for the CN. According to equations (12) and (25), we have

A sample set $$ {\psi _M} = \left\{ {e^{[q]}}|e_k^{[q]} \in {\Omega _e}, q = 1, 2 \cdots , M \right\} $$ is collected on $$ {\Omega} $$, where $$ M $$ denotes the maximum number of samples. Based on the MN and $$ {{\mu ^{(i)}}({e_k})} $$, we can predict the future error vector $$ e_{k + n}^{[q]} $$. Considering the sample set $$ {\psi _M} $$, equation (26) can be rewritten as

where $$ {\zeta ^{[q](i)}} = \sum\limits_{z = k}^{k + n - 1} {U\left( {e_z^{[q]}, {\mu ^{(i)}}\left(e_z^{[q]}\right)} \right)} + {\phi ^{\mathsf{T}}}\left(e_{k + n}^{[q]}\right){\omega ^{(i)}} $$. Using the least-square method, $$ \omega ^{(i + 1)} $$ can be calculated as

where $$ \lambda = {\left[{\phi ^{\mathsf{T}}}\left(e_k^{[1]}\right), {\phi ^{\mathsf{T}}}\left(e_k^{[2]}\right), \cdots , {\phi ^{\mathsf{T}}}\left(e_k^{[M]}\right)\right]^{\mathsf{T}}} $$ and $$ {\zeta ^{(i)}} = {\left[{\zeta ^{[1](i)}}, {\zeta ^{[2](i)}}, \cdots , {\zeta ^{[M](i)}}\right]^{\mathsf{T}}} $$.

The AN is used to approximate the tracking control policy, and its output can be expressed as

where $$ {\varpi ^{(i)}} \in {\mathbb{R}^{{l_a}}} $$. The activation function $$ \varphi ({e_k}) = {[{\varphi _1}({e_k}), \cdots , {\varphi _{{l_a}}}({e_k})]^{\mathsf{T}}} $$. $$ l_a $$ denotes the number of neurons in the hidden layer for the AN. According to equations (11) and (29), we have

Considering the sample set $$ {\psi _M} $$, equation (30) can be rewritten as

where $$ {\theta ^{[q](i)}} = - \frac{1}{2}{R^{ - 1}}{\left( {\frac{{\partial \hat x_{k + 1}^{[q]}}}{{\partial {u^{(i)}}(x_k^{[q]})}}} \right)^{\mathsf{T}}}\frac{{\partial \hat V_{k + 1}^{(i + 1)}}}{{\partial {u^{(i)}}(\hat e_{k + 1}^{[q]})}} $$. Thus, $$ \varpi ^{(i + 1)} $$ can be calculated as

where $$ \rho = {\left[{\varphi ^{\mathsf{T}}}\left(e_k^{[1]}\right), {\varphi ^{\mathsf{T}}}\left(e_k^{[2]}\right), \cdots , {\varphi ^{\mathsf{T}}}\left(e_k^{[M]}\right)\right]^{\mathsf{T}}} $$ and $$ {\theta ^{(i)}} = {\left[{\theta ^{[1](i)}}, {\theta ^{[2](i)}}, \cdots , {\theta ^{[M](i)}}\right]^{\mathsf{T}}} $$.

5. SIMULATION RESULT

In this section, we complete two simulations. Example 1 is to illustrate the effectiveness of the MsACTC algorithm and the correctness of Theorem 1. Example 2 is to show that the condition (14) is not sufficiently necessary for the convergence of the cost function.

5.1. Example 1

Consider the modified Van der Pol's oscillator system

(33) $$ {x_{k + 1}} = \left[ \begin{array}{l} {x_{k1}} + 0.1{x_{k2}}\\ - 0.1{x_{k1}} + 1.1x_{k2}^2 - 0.1x_{k1}^2{x_{k2}} \end{array} \right] + \left[ {\begin{array}{*{20}{c}} {0.35}&0\\ 0&{0.35} \end{array}} \right]u({x_k}). $$

The reference trajectory is defined as

(34) $$ {r_{k + 1}} = \left[ {\begin{array}{*{20}{c}} 1&{0.05}\\ { - 0.05}&1 \end{array}} \right]{r_k}. $$

Before the neural networks are trained, we make a sample set of $$ x_k $$ in the region where $$ - 1 \le {x_{k1}} \le 1 $$ and $$ - 1 \le {x_{k2}} \le 1 $$. In order to obtain the sample set of error vectors $$ e_k $$, the sample set of the reference trajectory $$ r_k $$ is also constructed in the region where $$ - 1 \le {r_{k1}} \le 1 $$ and $$ - 1 \le {r_{k2}} \le 1 $$. Assuming that the system (33) is unknown, we need to construct the MN using the neural network toolbox and the data generated by the system (33). The parameters of the toolbox are set as follows: the activation function is chosen as $$ \sigma (z) = ({e^z} - {e^{ - z}})/({e^z} + {e^{ - z}}) $$, the optimization algorithm is chosen as Levenberg-Marquardt backpropagation, the maximum training number of the MN is set as $$ 1000 $$, and the loss function is chosen as the mean squared normalized error function. The other parameters in the MsACTC algorithm are set as $$ Q = I_2 $$, $$ R = 5I_2 $$, $$ P = 6I_2 $$, $$ \eta = 10^{-7} $$, $$ M = 961 $$, $$ l_m = 20 $$, $$ l_a = 5 $$, and $$ l_c = 7 $$. Figure 2 shows the testing error after the training of the MN, and the result is satisfactory.

Figure 2. Testing errors of the MN (Example 1). MN: Model network.

We construct the activation functions for the CN and the AN as

(35) $$ \left\{ \begin{array}{l} \phi ({e_k}) = {\left[ {e_{k1}^2, e_{k2}^2, {e_{k1}}{e_{k2}}, e_{k1}^2{e_{k2}}, {e_{k1}}e_{k2}^2, e_{k1}^3, e_{k2}^3} \right]^{\mathsf{T}}}\\ \varphi ({e_k}) = {\left[ {{e_{k1}}, {e_{k2}}, {e_{k1}}{e_{k2}}, e_{k1}^2, e_{k2}^2} \right]^{\mathsf{T}}} \end{array} \right. $$

In order to satisfy the condition (14), the weight vector $$ \omega ^{(0)} $$ of the CN is initialized by one pre-training. $$ \omega ^{(0)} $$ satisfies $$ \hat V_k^{(0)} = {\phi ^{\mathsf{T}}}({e_k}){\omega ^{(0)}} = e_k^{\mathsf{T}} P {e_k} $$. The weight matrix of the AN is initialized as zero. Before starting the iteration, we set the maximum iteration number $$ i_{\rm {M}} = 25 $$. We use the different step sizes $$ n $$ of policy evaluation for the simulation to verify the effect of $$ n $$ on the convergence speed of the MsACTC algorithm.

In order to compare the convergence speeds under different values of $$ n $$, Figure 3 utilizes the convergence process of the norm of the weights to reflect the iterative process of the cost function and the tracking control policy. According to Figure 3, we can observe that the larger the value of $$ n $$, the faster the weights converge. Moreover, the weights corresponding to different values of $$ n $$ eventually converge to the same value, which indicates the optimality of the MsACTC algorithm. Figure 4 shows the iterative process of the cost function for $$ n=3 $$ through a mesh plot.

Figure 3. Convergence processes of network weights with different values of $$ n $$ (Example 1). AN: Action network; CN: critic network.

Figure 4. Convergence processes of the cost function for $$ n=3 $$ (Example 1).

We apply the optimal control policy obtained by the MsACTC algorithm for $$ n = 3 $$ to the system (33) for 300 time steps. We set $$ x_0=[0.2, -0.2]^{\mathsf{T}} $$ and $$ r_0=[-0.3, -0.3]^{\mathsf{T}} $$. The changing processes of the system state and tracking error are shown in Figure 5. The variation of control inputs is shown in Figure 6. With the control policy, the system states track the reference trajectory after $$ 50 $$ time steps.

Figure 5. Trajectories of state vectors and tracking errors (Example 1).

Figure 6. Trajectories of control inputs (Example 1).

5.2. Example 2

Consider the modified torsional pendulum system

(36) $$ \begin{align} {x_{k + 1}} = \left[ \begin{array}{l} {x_{k1}} + 0.1{x_{k2}} + 0.12{u_1}({x_k})\\ - 0.225\sin ({x_{k1}}) + 0.975{x_{k2}} + 0.125(\tanh \left( {{u_2}({x_k})} \right) + {u_2}({x_k})) \end{array} \right]. \end{align} $$

Due to the excellent adaptive ability, the MsACTC algorithm does not have strict requirements for the settings of the parameters. Therefore, the parameter values and the setting of activation functions are kept the same as those in Example 1. In Example 2, we set $$ V_k^{(0)} = 0 $$ to show that the condition (14) is not necessary for the convergence of $$ V_k^{(i)} $$. That is, $$ {\varpi ^{(0)}} $$ and $$ {\omega ^{(0)}} $$ are set as zero. In Example 2, we set $$ x_0=[0.5, -0.5]^{\mathsf{T}} $$ and $$ r_0=[-0.5, 0.3]^{\mathsf{T}} $$. After completing the initialization, the procedure of the experiment is the same as that in Example 1. The simulation results are shown in Figures 7-11. It is worth noting the direction of convergence of weights and the cost function in Figures 8 and 9. For Example 1, the direction of convergence is illustrated in Figures 3 and 4 is from the top to the bottom. For Example 2, the direction of convergence in Figures 8 and 9 is from the bottom to the top.

Figure 7. Testing errors of the MN (Example 2). MN: Model network.

Figure 8. Convergence processes of network weights with different values of $$ n $$ (Example 2). AN: Action network; CN: critic network.

Figure 9. Convergence processes of the cost function for $$ n=3 $$ (Example 2).

Figure 10. Trajectories of state vectors and tracking errors (Example 2).

Figure 11. Trajectories of control inputs (Example 2).

Even if the condition (14) does not hold, $$ V_k^{(i)} $$ and $$ {{\mu ^{(i)}}({e_k})} $$ can still converge to $$ V_k^ * $$ and $$ {\mu ^ * }({e_k}) $$ through the iteration. As $$ n $$ increases, the algorithm converges faster. However, it is worth noting that if the condition (14) is not satisfied, too large $$ n $$ will make the cost function diverge during the iteration.

6. CONCLUSIONS AND OUTLOOK

In this paper, the MsACTC algorithm is developed by introducing the multi-step policy evaluation mechanism into the adaptive critic tracking control algorithm. For nonlinear systems with unknown models, the MN is built based on data to estimate the state vector in the future time. By solving the steady control $$ v({r_k}) $$ through the MN, the tracking problem is transformed into the regulation problem. We give the convergence proof of the MsACTC algorithm. The weights of the CN and the AN are updated based on the least-square method. Simulation results verify the effectiveness of the MsACTC algorithm and the correctness of the theoretical results. It is worth exploring how to obtain the most suitable value of $$ n $$ in the MsACTC algorithm. When the condition (14) holds, the control policy is admissible after one policy improvement. Therefore, $$ n $$ can be assigned arbitrarily in the range of positive integers. However, when the value of $$ n $$ is large enough, even if $$ n $$ continues to increase, the convergence speed will not be significantly improved. In this case, increasing the value of $$ n $$ will lead to a waste of computing resources. If the condition (14) is not satisfied, then the algorithm cannot guarantee that the control policy is admissible, which means that the value of $$ n $$ is no longer arbitrary. This is because an unstable control policy will cause the state vector to diverge during multi-step policy evaluation. In general, a satisfactory method for selecting the value of $$ n $$ has not yet been proposed. In the future, we may continue to investigate this issue. In addition, this paper only uses two digital simulation examples to test the MsACTC algorithm. This means that the potential application areas of the algorithm have not been fully explored. Therefore, we will try to verify the performance of the algorithm in more application scenarios in the future.

DECLARATIONS