Evolution of adaptive learning for nonlinear dynamic systems: a systematic survey

2.1. Problem statement

The design of adaptive control laws is always under the assumption that system dynamics are exactly specified by models. Hence, when the true plant dynamics is not perfectly described by any model, as expected from a practice point of view, one can only question the real behavior of the control. The robust stability, required for any adaptive control to achieve practical applicability of the algorithms, can be provided when only the modeling error is sufficiently “small”. Unfortunately, stability alone cannot guarantee robustness, since the modeling error appears as a disturbance and usually causes divergence of the adaptive process.

While one of the fundamental fields of application of adaptive control is in systems with unknown time-varying parameters, the algorithms have been proved robust, in the presence of noise and bounded disturbances, only for systems with constant parameters^[13]. Ideally, when there are no disturbances or noise and when parameters are constant, adaptation shows smooth convergence and stability properties. On the other hand, the adaptive laws are not robust in the presence of bounded disturbances, noise and time-varying parameters.

In order to mathematically state the problem of non-robustness of adaptive control to bounded disturbances, let us start by considering a MIMO system in the form^[14,15],

where ξ(t) ∈ Rⁿ is a bounded time-dependant disturbance, x ∈ Rⁿ is the extended system state vector, y ∈ Rⁿ is the controlled system output, u ∈ R^m is the control input and Ψ ∈ R^N is the known N-dimensional regressor vector. We assume (A_ref, B, B_ref, C_ref) are known and A_ref is Hurwitz. y_cmd∈ R^m in this case is a bounded command for y. Λ ∈ R^m×m is a diagonal positive definite matrix and Κ ∈ R^N×m is a constant matrix, where both matrices represent the matched uncertainties of the system. In addition, we assume that,

and that the disturbance upper bound ξ_max≥ 0 is known and constant.

The control goal is bounded tracking of the reference model dynamics,

driven by a bounded time-dependant command y_cmd∈ R^m.

Based on Equation (1), the control input is selected as,

where K ∈ R^N×m is the matrix of adaptive parameters. If we substitute Equation (4) into Equation (1), we get,

where,

is the matrix of estimation errors. The tracking error is e = x - x_ref. Subtracting the reference model dynamics in Equation (3) from that of Equation (1) yields the tracking error dynamics,

The Lyapunov function candidate is selected,

where Γ_K = Γ_K^T > 0 represents constant rates of adaptation, and P = P^T > 0 is the unique symmetric positive definite solution of the algebraic Lyapunov equation,

with Q = Q^T > 0. The time derivative of V, along the trajectories of Equation (7),

Applying the trace identity,

yields,

Using the following adaptive law yields,

then,

and, consequently, V < 0 outside of the set,

Hence, trajectories [e(t),ΔK(t)], of the error dynamics in Equation (7) coupled with the adaptive law in Equation (13), enter the set E₀ in finite time and stay there for all future times. However, the set E₀ is not compact in the (e,ΔK) space. Moreover, it is unbounded since ΔK is not restricted. Inside the set E₀, V can become positive and consequently, the parameter errors can grow unbounded, even though the tracking error norm remains less then e₀ at all times. This phenomenon is caused by the disturbance term ξ(t). It shows that the adaptive law in Equation (13) is not robust to bounded disturbances, no matter how small the latter is.

In the 1980s, several studies analyzed the stability of adaptive control systems and many of them concentrated on linear disturbance-free systems^[16-20]. The results, however, are not completely satisfactory, since they do not consider the cases where disturbances are present, which could completely change the efficiency of the control system, even when very small, leading to instability. In the years that followed, there have been many attempts to overcome the limitations of adaptive control in the presence of bounded disturbances. In these published papers^[21-25], it is shown that unmodelled dynamics or even very small bounded disturbances can cause instability in most of the adaptive control algorithms.

Many efforts to design robust adaptive controllers in the case of unknown parameters have consistently progressed along two different shapes^[26-31]. In the first, the adaptive law is altered so that the overall system has bounded solutions in the presence of bounded disturbances. The second relies on the persistent excitation of certain relevant signals in the adaptive loop. The next subsections will present some of the main “modifications” proposed to enforce robustness with bounded disturbances.

2.2. Dead-zone modification

In many physical devices, the output is zero until the magnitude of the input exceeds a certain value. Such an input-output relation is called a dead-zone^[32]. In a first approach to prevent instability of the adaptive process in the presence of bounded external disturbances, Egardt^[26] introduced a modification of the law so that adaptation takes place when the identification error exceeds a certain threshold. The term dead-zone was first proposed by Narendra and Peterson^[18] in 1980, where the adaptation process stops when the norm of the state error vector becomes smaller than a prescribed value. In 1980, the study was initiated by Narendra to determine an adaptive control law that ensures the boundedness of all signals in the presence of bounded disturbances, in the case of continuous systems. In the study of Peterson and Narendra^[30], they highlight the cruciality of the proper choice of the dead zone for establishing global stability in the presence of an external disturbance. A larger dead zone implies that adaptation will take place in shorter periods of time, which also means larger parameter errors and larger output. One of the assumptions made in this paper is that a bound of the disturbance can be determined even though the plant parameters are unknown. The adaptive law shall consider that the module of the augmented error is not greater than the bound plus an arbitrary positive constant. Hence, the only knowledge needed to calculate the size of the dead zone is the bound of the disturbance, which can be computed^[30]. It is also worth noting that no prior knowledge of the zeros of the plant’s transfer function is needed to find the bound.

Samson^[28] presented a brief study in 1983 based on all his previous works and the analysis of Egardt in his book. Although his paper was only concerned with the stability analysis and not the convergence of the adaptive control to an optimal state, he was able to efficiently introduce a new attempt to use the possible statistical properties of the bounded disturbances. The three properties P₁-P₃ should be verified by the identification algorithm and are similar to the ones demanded for the disturbance-free cases, but less restrictive. The first property states that the identified vector has to be uniformly bounded, which prevents the system from diverging faster than exponentially. The second property ensures that the prediction error remains relatively small, which indicates that the “adaptive observer” transfer function is very similar to that of the system. Finally, the third property allows the control of the time-varying adaptive observer of the system.

In 1983, a modified dead-zone technique was proposed in Bunich’s research^[33] and was widely used. This modification permits a size reduction of the residual set for the error, hence, simplifying the convergence proof. The drawback is the necessary, yet restrictive, knowledge of a bound on the disturbance in order to appropriately determine the size of the dead-zone.

The work of Peterson and Narendra invigorated a new study by Sastry^[34] where he examined the robustness aspects of MRAC. Sastry used the same approach to show that a suitably chosen dead-zone can also stabilize the adaptive system against the effects of unmodelled dynamics. Though, the error between the plant and the model output does not converge to zero but rather to a magnitude less than the size of the dead-zone. In other terms, no adaptation takes place when the system is unable to distinguish between the error signal and the disturbance.

The issue in the dead-zone modification is that it is not Lipschitz, which may cause high-frequency oscillations and other undesirable effects when the tracking error is at or near the dead-zone boundary. In 1986, Slotine and Coetsee^[35] proposed a “smoother” version of the dead-zone modification. Unfortunately, we were not able to get a hold of a copy of this paper, but the major idea was explained in his book in 1990^[32].

2.3. σ-modification

The dead-zone modification assumes a priori knowledge of an upper bound for the system disturbance. On the other hand, the σ-modification scheme does not require any prior information about bounds for the disturbances. This modification was proposed by Ioannou and Kokotovic^[36] in 1983, which Ioannou referred to later as “fixed σ-modification”. The modification basically adds damping to the ideal adaptive law. They introduced the modification by adding a decay term -σΓθ to the disturbance-free integral adaptive law, where σ is a positive scalar to be chosen by the designer. The stability properties with the modification were established based on the existence of a positive constant p such that, for σ > p, the solutions for the error and adaptive law equations are bounded for any bounded initial condition. A conservative value of σ has to be chosen in order to guarantee σ > p. It was also shown that the modification yields the local stability of an MRAC scheme when the plant is a linear system of relative degree one and has unmodeled parasitics.

However, even though the robustness achievement is done smoothly and in a simpler way with the σ-modification scheme, there is thepotential destruction of some of the convergence properties, since there is no more asymptotic convergence and the fine tracking error is confined within a bounded region. Consequently, many additional modifications have been suggested later, motivated by the aforementioned drawback of the σ-modification. In 1986, Ioannou and Tsakalis^[37] proposed the “switching σ-modification”. In contrast to Ioannou’s earlier work^[38,39], the switching of σ from 0 to σ₀ is modified so that σ is a continuous function of |θ(t)| [Equation (16)], since the previous modification choices forced the adaptive law to be discontinuous, which might not guarantee the existence of a solution and would probably cause oscillations on the switching surface during implementation. Hence, the continuous switching, as shown in Figure 2, replaces the discontinuous one and is defined in Equation (16),

Figure 2. Switching σ-modification (continuous).

where M₀ > 0, σ₀ > 0 are design constants and M₀ is chosen to be large enough so that M₀ > |θ^*|.

In 1992, Tsakalis^[40] employed the σ-modification to target the adaptive control problem of a linear, time-varying SISO plant. The signal boundedness for adaptive laws was guaranteed using the σ-modification, normalization and a sufficient condition. The condition relates the speed and the range of the plant parameter variations with the σ value and simplifies the selection of the design parameters in the adaptive law.

In a more recent study, He et al.^[41] opted to revisit the fundamental σ-modification scheme and propose a qualitative analysis for all the scenarios where this modification can lead to perfect tracking, and where it can allow proper modification of the adaptive laws. The analysis method pre-supposes the existence of a Lyapunov function for an extended system, as shown in the reference^[42]. The efficacy of the proposed analysis was demonstrated in a Robust adaptive control system in order to detect its global asymptotic convergence under the fixed σ-modification scheme. When it comes to simulation results, the system shows asymptotic convergence of its trajectories without the modification; however, it may lose its asymptotic stability when the feedback gain and the modification gain are not well designed when using the modification. The recovery of the global asymptotic convergence is primarily dependant on the proper design of both gains, as shown in their last simulation.

2.4. ϵ-modification

The downside of the σ-modification is that when the tracking error becomes small enough, the adaptive parameters have an inclination to revert to the origin, which undoes the gain values that caused the tracking error to become small in the first place. In order to overcome this undesirable effect, Narendra and Annaswamy^[43] developed the ϵ-modification. The suggested modification was motivated by that given in the work of Ioannou and Kokotovic^[36], which similarly guarantees bounded solutions in the presence of bounded disturbances when the reference input is not persistently exciting, and needs less prior information regarding plant and disturbance. However, the catching point comes when the reference input is persistently exciting and has a sufficiently large amplitude. In this case, as we mentioned earlier, the origin of the error equations is exponentially stable, unlike that in Ioannou’s σ-modification. The new adaptive law replaces the σ with a term proportional to the magnitude of the output error, called ϵ (or e₁in the work of Narendra and Annaswamy^[43]).

Ideally, let’s consider the first order plant described with Equation (17),

where a_pis an unknown constant. The reference model is defined in Equation (18),

where a_m > 0 and r is a bounded piecewise continuous function. Equation (19) shows how the aim of the adaptive control is to choose the control input, u, such that the output of the plant approaches that of the model,

where θ is the control parameter. Therefore, we deduct the error equations in Equation (20),

and hence, based on Equation (21), the proposed adaptive law can be defined,

where ϵ is playing a double role since it attempts to decrease the magnitude of the output error while keeping the parameter θ or the parameter error φ bounded. The choice of the Lyapunov function, in Equation (22), gives the time derivative of V,

If we define a set D,

we then can deduct that V ≤ 0 inside the set D. The modification, which is synthesized by the additional term -|ϵ|θ in the adaptive law, shows that the set D is compact, which allows us to apply LaSalle’s theorem^[44] and prove that all solutions of the error equations are bounded.

If we distinguish between the three possible cases based on the reference input states: null, constant or persistently excited, and as mentioned earlier, the third case’s application highlights the difference between the proposed modification and the aforementioned σ-modification. When the σ-modification is used to adjust the control parameter θ, in the presence of the disturbance v, we can set the error equations as in Equation (23),

where it has been shown that three stable equilibrium states exist in case x_m = 0, none of them is the origin, and has a single equilibrium state whose distance from the origin decreases as the amplitude of x_m increases, and that is in case x_m is a constant. Which clearly highlights the addition of the ϵ-modification.

2.5. Summary

Basically, the approaches discussed above reduce the effects of parameter adaptation when the measure error is not due to an error in the parameter estimates. They contribute to either parameter error, noise error, high-frequency unmodelled dynamics error, or disturbances, which consist of anything undescribed by the three previous groups^[45]. A brief comparison of all the aforementioned modification techniques is shown in Table 1.

Considering the robustness problem, one can see that the disturbance is generated internally, which makes it dependable in the actual plant’s input-output signals. Particularly, the disturbance will grow unboundedly if the adaptive system is unstable and the input-output signals are growing without bound. Videlicet, the stability problem becomes internal and signal dependant. Thus, the boundedness of the disturbance should not be presumed, which proves that, despite the intrinsic results shown in the previous literature, the aforementioned approaches do not necessarily solve the robustness problem in the presence of bounded disturbances^[46].

Over the years, the adaptive controllers have proven themselves effective, especially in the process that can be modeled linearly with slowly time-varying parameters relative to the system’s dynamics. The 1980s were the peak of theoretical research on this case. On the other hand, many practical examples can be found in these research^[8,47-52].

An overview of some practical examples of adaptive control applications in two different fields, thermal exchange and robotics, is given in Table 2^[53-57]. We would also like to refer the readers to a very concise survey written by Åström^[8] in 1983 for more practical examples of the applications of adaptive control. In addition, adaptive controllers are extremely practical and fruitful when it comes to servo systems that have large disturbances, like load changes, or uncertainties, like frictions, and that have measurable states. The number one practical field in that era was robotics^[53-57].

Obviously, adaptive controllers are not the “perfect” solution to all control problems. For instance, they do not provide stability for systems where parameter dynamics are at least the same magnitude as the system’s dynamics. The controller robustness can be improved by employing artificial intelligence (AI) techniques, such as fuzzy logic and neural networks^[58-60]. Essentially, these methods approximate a nonlinear function and provide a good representation of the nonlinear unknown plant^[61], although it is typically used as a model-free controller. The plant is treated as a “black box”, with input and output data gathered and trained on. The AI framework addresses the plant’s model after the training phase, and can handle the plant with practically no need for a mathematical model. It is feasible to build the complete algorithm using AI techniques, or to merge the analytical and AI approaches such that some functions are done analytically and the remainder are performed using AI techniques^[62].

3.1. Neural network and the control of dynamic nonlinear systems

The characteristic of neural networks is that they are quite parallel. They can speed up computations and assist in the solving of issues that need much processing. Since NNs have nonlinear representations and can respond to changes in the environment, they easily reflect physical conditions like industrial processes and their control, whereas precise mathematical models are harder to construct.

One of the few theoretical frameworks for employing NNs for the controllability and stability of dynamical systems has been established by Levin and Narendra^[69]. Their research is limited to feedforward MLNs with dynamic BP and nonlinear systems with full state information access. Figure 3 presents the proposed architecture of the NNs. Equation (24) considers a system at a discrete-time index k,

Figure 3. Architecture of the proposed NNs in the work of Levin and Narendra^[69]. NNs: Neural Networks.

where x(k) ∈ χ ⊂ Rⁿ, u(k) ∈ U ⊂ R^r and f(0,0) = 0 so that x = 0 is an equilibrium. Conditions are given, in Equation (25), under which the two following NNs can be trained to feedback linearize and stabilize the system.

The results are extended to non-feedback linearizable systems. If the controllability matrix around the origin has a full rank, a methodology and conditions for training a single NN to directly stabilize the system around the origin have been devised. Narendra and Parthasarathy^[70] use NNs to create various identification and controller structures. Although the MLNs represent static nonlinear maps and the RNNs represent nonlinear dynamic feedback systems, they suggest that the feedforward MLNs and RNNs are comparable. They describe four network models of varying complexity for identifying and controlling nonlinear dynamical systems using basic examples.

Sontag proposed an article where he tried to explore the capabilities and the ultimate limitations of alternative NN architectures^[71]. He suggests that NNs with two hidden layers may be used to stabilize nonlinear systems in general. Intuitively, the conclusion contradicts NNs approximation theories, which claim that single hidden layer NNs are universal approximators. Sontag’s solutions are based on the description of the control issue as an inverse kinematics problem rather than an approximation problem.

In 1990, Barto^[72] drew an interesting parallel between connectionist learning approaches and those investigated in the well-established field of classical adaptive control. When utilizing NNs to address a parameter estimate problem, the representations are frequently chosen based on how nervous systems represent information. In contrast, in a traditional method, issue representation options are made based on the physics of the problem. As opposed to conventional methods, a connectionist approach is dependent on the structure of the network and the correlation between the connectionist weights. A traditional controller may readily include a priori information; however, in NNs, it is often an input-output connection. In both techniques, performance may be assessed using cost functions such as least mean squared error. All of the training data is available at the same time with off-line approaches. However, with on-line approaches, the required feature is continuous learning, and as a result, the methods must be extremely efficient in order to keep up with the changing events over time.

Adaptive NNs have recently been used by a growing number of academics and researchers to construct acceptable control rules for nonlinear systems. An overview of the primarymost recent literature that implemented adaptive NNs-based techniques is discussed in Table 3^[73-82].

3.2. Inverted pendulum

Many researchers have studied learning control using the inverted pendulum problem. The canonical underactuated system, called the cart-pole system, is illustrated in Figure 4. Because deriving the dynamics is relatively simple, it is considered a basic control issue, yet it still hides some underlying complexity owing to its underactuated character. The multiple obstacles that must be addressed to properly regulate such extremely complex nonlinear unstable systems include severe nonlinearities, variable operating circumstances, structured and unstructured dynamical uncertainties, and external disturbances. The purpose of the control is to balance the pole by moving the cart, which has a restricted range of movements. We distinguish between the position of the cart h and its velocity h, and the angle of the pole θ with its angular velocity θ.

Figure 4. The Cart-pole scheme.

In 1983, Barto et al.^[83] showed how a system consisting of two neuronlike adaptive elements, associative search element (ASE) and adaptive critic element (ACE), can solve a difficult learning control problem such as the cart-pole system. Their work was based on the addition of a single ACE to the ASE developed by Michie and Chambers in the works of Michie and Chambers^[84,85]. They have partitioned the state space into 162 boxes. Their simulations revealed that the ASE/ACE system outperformed the boxes system in terms of run time. The system was more likely to solve the problem before it had 100 failures, but the boxes system was less likely to do so. The ASE/ACE system’s high performance was nearly completely owing to the ACE’s provision of reinforcement throughout the trials. Learning occurs only upon failure with box systems and ASEs without an ACE, which happens less frequently as learning progresses. An ASE can get input on each time step with the ACE in place. The system attempts to access some areas of the state space and avoids others as a result of the learning achieved by this input.

Anderson^[86] built on the work of Barto et al.^[83] by using a variant of the common error BP algorithm to two-layered networks that learn to balance the pendulum given the inverted pendulum’s real state variables as input. Two years later^[87], he summarized both aforementioned works by discussing the neural network structures and learning methods from a functional viewpoint and by presenting the experimental results. He described NN learning techniques, which use two functions to learn how to construct action sequences. The first is an action function, which converts the current state into control actions. The second is an evaluation function, which converts the present state into an assessment of that state. There were two sorts of networks that emerged: “action and evaluation” networks. This is an adaptive critic architecture version

In 1991, Lin and Kim integrated the CMAC into the self-learning control scheme that was based on the work of Lin and Kim^[88]. The CMAC model was originally proposed by Albus^[89-92] and it was based on models of human memory and neuromuscular control. The CMAC-based technique in the work of Lin and Kim^[88] is tested using the inverted pendulum problem, and the results are compared to those of Barto et al.^[83] and Anderson^[87]. The technique has the highest learning speed due to its capability of generalization and good learning behavior. Furthermore, the memory size can be reduced compared to the box-based system. A summarized timeline of the above literature, where NN-based control was implemented to balance the inverted pendulum, is presented in Figure 5.

Figure 5. Timeline scheme of the works that kickstarted the use of NNs to control the inverted pendulum. NNs: Neural Networks.

Many control laws for inverted pendulums have been presented in those research work^[93-95], including classical, robust, and adaptive control laws, but they all take structured parametric uncertainty into account. In 2009, Chaoui et al.^[96] proposed an ANN based adaptive control strategy for inverted pendulums that accomplishes asymptotic motion tracking and posture control with unknown dynamics. Two neural networks ANN_x and ANN_θ are designed to control the motion along the x axis and the pendulum posture with unknown dynamics. Figure 6 shows the block diagram of the proposed system.

Figure 6. Block diagram of the ANN-based adaptive control scheme^[96]. ANN: Artificial Neural Network.

Three experiments are carried out to evaluate the performance of the proposed controller. The velocity and posture of the pendulum progressively decrease to zero in the first experiment. The proposed adaptive control, on the other hand, produces a smooth control signal. The controllers also deal with friction nonlinearities and accomplish quick error convergence and tracking. The second experiment introduces a starting posture position to test the controller’s capacity to correct for a non-zero position error. Posture control takes precedence over motion tracking, as posture is critical for such systems. The purpose of the third experiment is to demonstrate the modularity of the proposed controller in terms of adjusting for external disturbances. The suggested controller’s design does not clearly model the induced external disturbance, which generally has a considerable impact on the positioning system’s accuracy and generates unacceptably high-frequency oscillations. The controller is able to deal with the unexpected force change successfully. Furthermore, the motion and posture errors are kept to a minimum, resulting in a smooth control signal.

3.3. Applications for robotic manipulators

There has been great interest in universal controllers that mimic the functions of human processes to learn about the systems they are controlling on-line so that performance improves automatically. NN-based controllers are derived for robot manipulators in a variety of applications, including position control, force control, link flexibility stabilization and the management of high-frequency joint and motor dynamics. The nature of joint torques must be determined for the end effector to follow the required trajectory as quickly and accurately as feasible, which is a common difficulty for robot manipulators. Both parametric and structural uncertainties necessitate adaptive control. Parametric uncertainties originate from a lack of accurate information about the manipulator’s mass characteristics, unknown loads, and load location uncertainty, among other things. Structural uncertainties are a result of the presence of high-frequency unmodeled dynamics, resonant modes, and other structural reservations.

The late 1980s and early 1990s were booming years for both NNs and robotic manipulators research. In this era, the literature survey concerning the application of NNs in robotic manipulators is very rich. Thus, we direct the readers to some interesting approaches in these studies^[97-102] and the references therein.

From 1987 to 1989, Miller et al.^[103-107] discuss a broad CMAC learning technique and its application to robotic manipulators’ dynamic control. The dynamics do not need to be known in this application. Through input and output measurements, the control scheme learns about the process. The findings show that when compared to fixed-gain controllers, the CMAC learning control performs better. Also, because measured and estimated values must be transformed to discrete form, each variable’s resolution and range must be carefully selected, and the number of memory regions handled by each input state in the CMAC architecture is the most important design parameter. In another popular approach, Miller et al.^[108] used CMAC in the real-time control of an industrial robot and other applications. In their network, they utilize hundreds of thousands of adjustable weights that, in their experience, converge in a few iterations.

Huan et al.^[109] examine the issue of building robot hand controllers that are device-dependent. Their argument for a controller like this is that it would isolate low-level control issues from high-level capabilities. They employ a BP algorithm with a single hidden layer comprised of four neurons to achieve this goal. The inputs are determined by the object’s size, while the outputs are determined by the grab modes. In this way, they have demonstrated how to build a p-g table using simulation. Another BP architecture was used by Wang and Yeh^[110] to control a robot model which simulates PUMA560. A network to simulate the plant and a controller network make up their self-adaptive neural controller (SANC). The plant model is trained either off-line with mathematical model outputs or on-line with plant outputs through excitations. The control network is modified by working in series with the plant network during the “controlling and adapting” phase. The control network is also trained off-line in a “memorizing phase” with data from the adapting phase in a random way, which is another element of this training. This trait, according to the authors, aids in overcoming the temporal instability that is inherent with BP. Their numerical findings show that the SANC technique produces good trajectory-tracking accuracy.

Up to the early 2000s, the main goal of robotic manipulators designs was to minimize vibration and achieve good position accuracy, which led to maximizing stiffness. This high stiffness is achieved by using heavy material and a bulky design. As a result, it is demonstrated that heavy rigid manipulators are wasteful in terms of power consumption and operational speed. It is necessary to reduce the weight of the arms and increase their speed of action in order to boost industrial output. As a result of their light weight, low cost, bigger work volume, improved mobility, higher operational speed, power economy, and a wider range of applications, flexible-joint manipulators have gotten much attention. Figure 7 shows a representation of a flexible joint manipulator model.

Figure 7. Flexible-joint manipulator model.

Controlling such systems, however, still challenges significant nonlinearities, such as coupling caused by the manipulator’s flexibility, changing operating conditions, structured and unstructured dynamical uncertainties, and external disturbances. Complex dynamics regulate flexible-joint manipulators^[111-114]. This emphasizes the need to examine alternate control techniques for these types of manipulator systems in order to meet their increasingly stringent design criteria. Many control laws for flexible joints have been presented in those studies^[115-118] to solely address (structured) parametric uncertainties. The proposed controllers need a complete a priori knowledge of the system dynamics. Several adaptive control systems^[119-121] have been proposed to alleviate this necessity. The majority of these control strategies use singular perturbation theory to extend adaptive control theory established for rigid bodies to flexible ones^[122-125].

Based on all the above reasons, computational intelligence techniques, such as ANNs and fuzzy logic controllers, have been credited in a variety of applications as powerful controllers of the types of systems that may be subjected to structured and unstructured uncertainties^[126,127]. As a result, there have been advancements in the field of intelligent control^[128,129]. Various neural network models have been used to operate flexible-joint manipulators, and the results have been adequate^[130]. Chaoui et al.^[131,132] developed a control strategy inspired by sliding mode control that uses a feedforward-NN to learn the system dynamics. Hui et al.^[133] proposed a time-delay neuro-fuzzy network. The joint velocity signals were estimated using a linear observer in this system, which avoided the need to measure them directly. Subudhi and Morris^[134] proposed a hybrid architecture that included a NN for controlling the slow dynamic subsystem and an H_∞ for controlling the rapid dynamic subsystem. Despite its effectiveness, NN-based control systems are still unable to incorporate any humanlike experience already obtained about the dynamics of the system in question, which is regarded as one of the soft computing approaches’ primary flaws.

Chaoui et al.^[135] suggested an ANN-based control technique in 2009, which used ANNs’ learning and approximation skills to estimate the system dynamics. The MRAC is made up of feedforward (ANN_FF) and feedback (ANN_FBK) NN-based adaptive controllers. The reference model is built in the same manner as a sliding hyperplane in variable structure control, and its output, which may be regarded as a filtered error signal, is utilized as an error signal to adjust the ANN_FBK’s weights. It comprises a first-order model that specifies the required dynamics of the error between the desired and real load locations, as well as between the motor and load velocity, in order to maintain internal stability. The ANN_FF offers an approximate inverse model for the positioning system, while the ANN_FBK corrects residual errors, assuring the manipulator’s internal stability and rapid controller response.

The feedback’s learning rate is dependent on the load inertia, which is a flaw in this construction. To improve the stability region of the NN-based controllers, a supervisor is proposed to modify the learning rate of the ANNs. The supervisor also increases the adaptation process’s convergence qualities.

Nowadays, the subject of multiple-arms manipulation highlights some interesting progress in using intelligent control approaches. Hou et al.^[136] used a dual NN to solve a multicriteria optimization problem for coordinated manipulation. Li et al.^[137,138] are representatives who operate on several mobile manipulators with communication delays. Some promising approaches, such as LMI and fuzzy-NN controls, were used in both articles^[137,138], to improve motion/force performances, which were crucial in multilateral teleoperation applications.

In 2017, He et al.^[139] proposed an Adaptive NN-based controller for a robotic manipulator with time-varying output constraints. The adaptive NNs were utilized to adjust for the robotic manipulator system's uncertain dynamics. The disturbance-observer (DO) is designed to compensate for the influence of an unknown disturbance, and asymmetric barrier Lyapunov Functions (BLFs) are used in the control design process to avoid violating time-varying output constraints. The effects of system uncertainties are successfully corrected, and the system's resilience is increased using the adaptive NN-based controller. The NN estimating errors are coupled with the unknown disturbance from people and the environment to form a combined disturbance that is then approximated by a DO.

In a recent interesting paper, He et al.^[140] attempted to control the vibrations of a flexible robotic manipulator in the presence of input dead-zone. The lumped technique is used to discretize the flexible link system^[141,142]. A weightless linear angular spring and a concentrated point mass are used to partition the flexible link into a finite number of spring-mass parts. They design NN controllers with complete state feedback and output feedback based on the constructed model. All state variables must be known to provide state feedback. An observer is presented to approximate the unknown system state variables in the case of control with output feedback. In summary, an overview of the evolution of NNs implementation in robotic manipulation is shown in Table 4. Each of these papers has been categorized based on the nature of its approach.

3.4. From machine learning to deep learning

ML has transformed various disciplines in the previous several decades, starting in the 1950s. NN is a subfield of ML, a subset of AI, and it is this subfield that gave birth to Deep Learning (DL). There are three types of DL approaches: supervised, semi-supervised, and unsupervised. There is also a category of learning strategy known as RL or DRL, which is commonly considered in the context of semi-supervised or unsupervised learning approaches. Figure 8 shows the classification of all the aforementioned categories.

Figure 8. Classification of AI categories.

The common-sense principle behind RL is that if an action is followed by a satisfying state of affairs, or an improvement in the state of affairs, the inclination to produce that action is enhanced, or in other words reinforced. Figure 9 presents a common diagram model of general RL. The origin of RL is well rooted in computer science, though similar methods such as adaptive dynamic programming and neuro-dynamic programming (NDP)^[143] were developed in parallel by researchers and many others from the field of optimal control. NDP was nothing but reliance on both concepts of Dynamic-Programming and NN. For the 1990’s AI community, NDP was called RL. This is what makes RL one of the major NN approaches to learning control^[60].

Figure 9. Universal model of RL. RL: Reinforcement learning.

On the other hand, deep models may be thought of as deep-structured ANNs. ANNs were first proposed in 1947 by Pitts and McCulloch^[144]. Many major milestones in perceptrons, BP algorithm, Rectified Linear Unit, Max-pooling, dropout, batch normalization, and other areas of study were achieved in the years that followed. DL’s current success is due to all of these ongoing algorithmic advancements, as well as the appearance of large-scale training data and the rapid development of high-performance parallel computing platforms, such as Graphics Processing Units^[145]. Figure 10 shows the main types of DL architectures. In 2016, Liu et al.^[146] proposed a detailed survey about DL architectures. Four main deep learning architectures, which are restricted Boltzmann machines (RBMs), deep belief networks (DBNs), autoencoder (AE), and convolutional neural networks (CNNs), are reviewed.

Figure 10. Main DL architectures. (A) Schematic diagram of RBMs; (B) schematic diagram of AEs; (C) schematic diagram of DBNs; (D) conceptual structure of CNNs. RBMs: restricted Boltzmann machines; DBNs: deep belief networks; AE: autoencoder; CNNs: convolutional neural networks.

4.1. Reinforcement learning for robotic control

The reinforcement learning (RL) domain of robotics differs significantly from the majority of well-studied RL benchmark issues. In robotics, assuming that the true state is totally visible and noise-free is typically impractical. The learning system will have no way of knowing which state it is in, and even very dissimilar states may appear to be quite similar. As a result, RL in robots is frequently represented as a partially observed system. Consequently, the learning system must approximate the real state using filters. Experience with an actual physical system is time-consuming, costly, and difficult to duplicate. Because each trial run is expensive, such applications drive us to concentrate on issues that do not surface as frequently in traditional RL benchmark instances. Appropriate approximations of state, policy, value function, and/or system dynamics must be introduced in order to learn within a tolerable time period. While real-world experience is costly, it can typically not be substituted solely by simulation learning. Even little modeling flaws in analytical or learned models of the system might result in significantly divergent behavior, at least for highly dynamic jobs. As a result, algorithms must be resistant to under-modeling and uncertainty.

Another issue that arises frequently in robotic RL is generating appropriate reward functions. To cope with the expense of real-world experience, rewards that steer the learning system fast to success are required. This problem is known as reward shaping, and it requires a significant amount of manual contribution^[148]. In robotics, defining excellent reward functions necessitates a substantial degree of domain expertise and can be difficult in practice.

Not all RL methods are equally appropriate for robotics. Indeed, many of the methods used to solve complex issues thus far have been model-based, and robot learning systems frequently use policy search methods rather than value function-based approaches. Such design decisions are in stark contrast to maybe the majority of early ML research. The papers that follow will discuss several approaches to incorporating RL into robotics and manipulation. Kober et al.^[149] conducted a comprehensive review of RL in robotics in 2013. They provide a reasonably comprehensive overview of “Real” Robotic RL and mention the most innovative studies, which are organized by significant findings.

In the last 15 years or so, the use of RL in robots has continuously risen. An overview of the RL-based implementation in robots’ control is shown in Table 5^[150-172], where each of the undermentioned papers has been categorized based on the nature of their approach.

A stacked Q-learning technique for a robot interacting with its surroundings was introduced by Digney^[150]. In an inverted pole-balancing issue, Schaal^[151] employed RL for robot learning. For compliance tasks, Kuan and Young^[152] developed an RL-based mechanism in conjunction with a robust sliding mode impedance controller, which they evaluated in simulation. To cope with the variation in the different compliance tasks, they apply an RL-based method in their research. Bucak and Zohdy^[153,154] proposed an RL-based control strategy for one and two link robots in 1999 and 2001. Althoefer et al.^[155] used RL to attain motion and avoid obstacles in a Fuzzy rule-based system for a robot manipulator. Q-learning for robot control was investigated by Gaskett^[156]. For a mobile robot navigation challenge, Smart and Kaelbling also opted for an RL-based approach^[157]. For optimal control of a musculoskeletal-type robot arm with two joints and six muscles, Izawa et al.^[158] used an RL actor-critic framework. For an optimum reaching task, they employed the proposed technique. RL approaches in humanoid robots are characterized, by Peters et al.^[159], as greedy methods, “vanilla” policy gradient methods, and natural gradient methods. They highly encourage the adoption of a natural gradient approach to control humanoid robots, because natural-actor-critic (NAC) structures converge fast and are better suited to high-dimensional systems like humanoid robots. They have proposed a number of different ways to design RL-based control systems for humanoid robots. An expansion of this study was given in 2009 by Bhatnagar et al.^[160]. Theodorou et al.^[161] employed RL for optimal control of arm kinematics. NAC applications in robotics were presented by Peters and Schaal^[162]. For the estimate, the NAC employs the natural gradient approach. Other works presented here^[163-165] go into greater depth on actor-critic based RL in robots. Buchli et al.^[166] propose RL for variable impedance management methods based on policy improvement using a route integral approach. Only simulations were used to illustrate the efficiency of the suggested method. Theodorou et al.^[167] used a robot dog to evaluate RL based on policy improvement using path integral^[168]. RL-based control for robot manipulators in uncertain circumstances was given by Shah and Gopal^[169]. Kim et al.^[170,171] applied an RL-based method to determine acceptable compliance for various scenarios by interaction with the environment. The usefulness of Kim et al.^[170,171]’s RL-based impedance learning technique has been demonstrated in simulations.

For a robot goalkeeper and inverted pendulum examples, Adam et al.^[172] proposed a very interesting article on the experimental implementation of experience replay Q-learning and experience replay SARSA approaches. In this form of RL scheme, the data obtained during the online learning process is saved and fed back to the RL system continuously^[172]. The results are encouraging, albeit the implementation method may not be appropriate for all actual systems, as the exploration phase indicates very irregular, nearly unstable behavior, which might harm a more delicate plant.

It is worth noting that several of the RL systems outlined above are conceptually well-developed, with convergence proofs available. However, there is still much work to be done on RL, and real-time implementations of most of these systems are still a great difficulty. Furthermore, adequate benchmark challenges^[173] are required to test newly created or improved RL algorithms.

4.2. Deep reinforcement learning for robotic manipulation control

In 2012, deep learning (DL) achieved its first major breakthrough with a CNN for classification^[174]. It iteratively trains the parameters using loss computation and BP using hundreds of thousands of data-label pairs. Although this approach has developed steadily since its inception and is currently one of the most widely used DL structures, it is not ideal for robotic manipulation control because obtaining a large number of pictures of joint angles with labeled data to train the model is too time-consuming. CNN has been used in several studies to learn the motor torques required to drive a robot using raw RGB video pictures^[175]. However, as we will see later, employing deep reinforcement learning (DRL) is a more promising and fascinating notion.

In the context of robotic manipulation control, the purpose of DRL is to train a deep policy NN, such as the one shown in Figure 10, to discover the best command sequence for completing the job. The present state, as shown in Figure 11, is the input, which can comprise the angles of the manipulator’s joints, the location of the end effector, and their derivative information, such as velocity and acceleration. Furthermore, the current posture of target objects, as well as the status of relevant sensors if any are present in the surroundings, can be tallied in the current state. The policy network’s output is an action that specifies which control instructions, such as torques or velocity commands, should be applied to each actuator. A positive reward will be produced when the robotic manipulator completes a job. The algorithm is supposed to discover the best successful control method for robotic manipulation using these delayed and weak data.

Figure 11. DRL scheme for Robotic manipulation control. DRL: Deep reinforcement learning.

The study of sample efficiency for supervised deep learning determines the scale of the training set required in learning. Consequently, even though it is more challenging than supervised deep learning, the study of sample efficiency for DRL in robotic control provides how much data is needed to build an optimal policy. The first demonstration of using DRL on a robot was in 2015, when Levine et al.^[176] applied trajectory optimization techniques and policy search methods with NNs to accomplish a practical sample efficient learning. They employ a recently developed policy search approach to learn a variety of dynamic manipulation behaviors with very broad policy representations, without requiring known models or example demonstrations in this study. This method uses repeatedly refitted time-varying linear models to train a collection of trajectories for the desired motion skill, and then unifies these trajectories into a single control policy that can generalize to new scenarios. Some modifications are needed in order to lower the sample count and automate parameter selection to enable this technique to run on a real robot. Finally, this approach has proven that the learning of robust controllers for complexity is possible, which did achieve various compound tasks such as stacking tight-fitting Lego blocks and putting together a toy airplane after minutes of interaction time.

The concept of imitation learning became very popular for robotic manipulation, since relying on learning from trial and error required a significant amount of system interaction time if based solely on DRL approaches^[177]. In 2018, an interesting approach was proposed by Vecerik et al.^[178] combining both imitation learning and task-reward-based learning, which improved the agent’s abilities in simulation. The approach was based on an extension of Deep Deterministic Policy Gradient (DDPG) algorithm for tasks with sparse rewards. Unfortunately, in real robot experiments, the location of the object, as well as the explicit states of joints like position and velocity, must be specified, which limits the approach’s applications to high-dimensional data^[179].

In 2017, Andrychowicz et al.^[180] proposed Hindsight Experience Replay as a novel technique that provides for sample-efficient learning from sparse and binary rewards, avoiding the need for complex reward engineering. It may be used in conjunction with any off-policy RL algorithm to create an implicit curriculum.

In October 2021, AI researchers at Stanford University presented a new technique called deep evolutionary reinforcement learning, or DERL^[181]. The new method employs a sophisticated virtual environment as well as RL to develop virtual agents that can change their physical form as well as their learning abilities. The discoveries might have far-reaching ramifications for AI research in general and robotics research in particular in the future. Each agent in the DERL architecture employs DRL to gain the abilities it needs to achieve its objectives throughout the course of its existence. MuJoCo, a virtual environment that enables very accurate rigid-body physics modeling, was employed by the researchers to create their framework. Universal Animal is their design space, and the objective is to construct morphologies that can master locomotion and item manipulation tasks in a range of terrains. The developed agents were put through their paces in eight various tasks, including patrolling, fleeing, manipulating items, and exploring. Their findings reveal that AI agents who have developed in different terrains learn and perform better than AI agents who have only seen flat terrain.

An overview of the connection of the above-mentioned work is presented in Table 6. Some basic problems are listed in the table, and each paper’s approach is presented and categorized based on observation and action space, reward shaping and algorithm types.

Although DRL-based robotic manipulation control algorithms have proliferated in recent years, the issues of acquiring robust and diverse manipulation abilities for robots using DRL have yet to be properly overcome for real-world applications.

4.3. Summary

Over the last several years, the robotics community has been progressively using RL and DRL-based algorithms to manage complicated robots or multi-robot systems, as well as to give end-to-end policies from perception to control. Since both algorithms base their knowledge acquisition on trial-and-error, they naturally require a large number of episodes, which limits the learning in terms of time and experience variability in real-world scenarios. In addition, the real-world experience must consider the potential dangers or unexpected behaviors of the considered robot, especially when it comes to safety-critical applications. Even though there are some successful real applications to DRL in robotics, especially with tasks involving object manipulations^[182,183], the success of its algorithms beyond the simulated worlds is fairly limited. Transferring DRL policies from simulation environments to reality, referred to as “sim-to-real”, is a necessary step toward more complex robotic systems that have DL-defined controllers. This has led to an increase in research in “sim-to-real” transfer, which resulted in many publications over the past few years.

Another angle that we see crucial for robotics applications is local vs. global learning. For instance, when humans learn a new task, like walking, they automatically build upon the previously learned skill in order to learn a new one, like running, which becomes significantly easier. It is essential to reuse other locally learned information from past data sets. When it comes to robot RL/DRL, the publicity of the making of such data sets with many skills should be available and accessible to everyone in robotic research, which would be considered a huge asset. When it comes to reward shaping, RL approaches have significantly benefited from it by using rewards that convey closeness and are not only based on binary success or failure. For robotics, it is challenging to shape such a reward design, hence, it would be optimal if the reward-shaping is physically motivated, like for instance, minimizing the torques while achieving a task.