Enhancing unmanned aerial vehicle communication through distributed ledger and multi-agent deep reinforcement learning for fairness and scalability

1. INTRODUCTION

Unmanned Aerial Vehicles (UAVs) have significantly revolutionized communication in different areas, including wireless sensor networks (WSN), cellular networks, the Internet of Things (IoT), and Space-Air-Ground Integrated Networks (SAGIN). These flexible platforms facilitate the rapid implementation of communication services to users on the ground in situations when terrestrial infrastructure is not accessible or has been compromised, including in disaster-impacted areas or conflict-ridden battlefields. UAVs are versatile and can be used for various purposes such as improving communication coverage, serving as mobile relays, enabling edge computing, and performing data collection. These applications have advanced UAV-assisted technology to the forefront of research in wireless communications and network sectors ^[1–4]. The benefits of UAVs are distinctive and can be credited to several significant advancements. Advancements in industrial technology have enabled the downsizing of electronic equipment, enhancing their capacity and facilitating the integration of more advanced modules on UAVs at a lower cost. Furthermore, the great mobility of UAVs allows them to be deployed in difficult terrains such as mountains and rivers, where setting up ground-based infrastructure is not feasible. This guarantees thorough and smooth communication capacities. Thirdly, UAVs provide excellent visibility for ground communication systems, minimizing signal route loss caused by barriers and improving Line-of-Sight (LoS) connections. In recent years, substantial research has been dedicated to enhancing the control systems of UAVs to address the challenges associated with their deployment. Notable among these advancements is the development of distributed adaptive fuzzy formation control for multiple UAVs, which can handle uncertainties and actuator faults while operating under switching topologies. This method utilizes fuzzy logic to adaptively manage the formation of UAVs, ensuring robust performance despite the presence of system uncertainties and potential faults^[5]. Additionally, neural adaptive distributed formation control has emerged as a significant approach for managing nonlinear multi-UAV systems with unmodeled dynamics. By leveraging neural networks, this control strategy can adapt to complex, nonlinear interactions within the UAV network, ensuring stable formation control even when the system dynamics are not fully known or are subject to change. These neural adaptive methods provide a high degree of flexibility and robustness, making them suitable for dynamic and uncertain environments^[6]. Furthermore, other advanced control techniques, such as Model Predictive Control (MPC) and Reinforcement Learning (RL), have been applied to UAV systems to enhance their autonomous capabilities. MPC allows UAVs to predict and optimize their trajectories based on future states, ensuring efficient navigation and collision avoidance. RL, on the other hand, enables UAVs to learn optimal control policies through interaction with their environment, adapting to new scenarios and improving performance over time ^{[7, 8]}. These advancements in UAV control systems are critical for enabling UAVs to operate autonomously and efficiently in various applications, from disaster response to commercial delivery services. The integration of these sophisticated control techniques ensures that UAVs can maintain stable formations, handle dynamic changes, and operate reliably even in the presence of uncertainties and external disturbances.

2. SYSTEM MODEL

The framework illustrated in Figure 1 presents an advanced system for communication that consists of a group of UAVs, each equipped with DLT and MADRL capabilities. The purpose of this design is to overcome the limitations of traditional communication networks, particularly in areas where conventional infrastructure is insufficient or degraded. In this innovative network, UAVs act as independent aerial relay stations, forming a robust mesh network to provide connectivity to GUs across different terrains. Each UAV $$ m $$ from the fleet $$ M $$ has a position $$ \mathbf {w}_{u_m}(t) $$ at time $$ t $$, which includes altitude $$ H $$. The movement of UAVs between time slots is limited by their maximum speed $$ V_{max} $$ and the time slot duration $$ \delta $$^{[20, 21]}.

Figure 1. Proposed system model using integrated MADRL and distributed ledger techniques.

The UAVs manage communication links with GUs using a Time Division Multiple Access (TDMA) scheme. The binary indicator variables $$ \alpha_{src_{m, k}}(t) $$ and $$ \beta_{dst_{m, k}}(t) $$ denote whether a source $$ k_{src} $$ or destination $$ k_{dst} $$ GU is connected to UAV $$ m $$ during time slot $$ t $$^{[22, 23]}.

The constraints C1 and C2 for access control are expressed as

The integration of DLT ensures a secure and unchangeable exchange of data, while MADRL enables UAVs to make informed decisions for enhancing the network's performance. In MADRL, the Q-function is used to measure the effectiveness of a particular action taken in a specific state, represented as $$ \Theta(s, a) $$, where $$ s $$ denotes the state and $$ a $$ represents the action. The primary goal is to optimize the policy parameters $$ \Theta $$ such that the total future rewards are maximized. This optimization process involves using an update function $$ U $$, which modifies $$ \Theta $$ iteratively according to a specific update rule.

The change $$ \Delta \Theta $$ in the policy parameters is defined as

In this particular formulation, the learning rate $$\alpha$$ represents the extent to which new information overrides old information. The function $$\mathcal{R}(s, a)$$ provides the immediate reward after taking an action $$a$$ in a particular state $$s$$. The discount factor $$\gamma$$ balances the significance of immediate and future rewards. The policy $$\pi(\cdot|s')$$, which can potentially be an $$\epsilon$$-greedy strategy, guides the selection of the subsequent action $$a'$$ in the next state $$s'$$, and the operation $$\max_{a' \sim \pi(\cdot|s')}$$ identifies the action that maximizes the expected Q-value given the policy $$\pi$$ and the new state $$s'$$. By means of this mechanism, the algorithm gradually approaches an optimal policy that prescribes the best action to take in any given state to maximize long-term rewards. Dual transmitters on UAVs and Orthogonal Frequency-Division Multiple Access (OFDMA) can reduce interference with a factor $$ \eta $$ that mitigates its impact in UAV-to-UAV communication.

where $$ I_{total} $$ represents the overall interference experienced by a GU. The value of $$ N $$ refers to the total number of interfering signals, and the value of $$ I_i $$ is the interference generated by the $$ i-th $$ signal. The term $$ \eta $$ (where $$ 0 < \eta < 1 $$) indicates the effectiveness of interference reduction techniques.

3. PROBLEM FORMULATION AND OBJECTIVE FUNCTION

We define the accumulative throughput for the k-th user pair at time slot $$ t $$ as

Here, $$ r_k(r) $$ represents the throughput for user pair $$ k $$ at time slot $$ r $$.

Subsequently, the sum accumulative throughput for all user pairs is given by

To ensure fairness, we utilize Jain's index ^[26] to measure fair throughput

The parameter $$ e $$ represents the energy consumed by each UAV $$ m $$ at each time slot $$ t $$. where the fairness index $$ f(t) $$ ranges between $$ \frac{1}{K} $$ and 1.

The access policies of GUs and the locations of UAVs at time slot t are represented by $$ A = \{ \alpha_{m, k}^{src}(t), \beta_{m, k}^{dst}(t) \}_{m \in M, k \in K, t \in T} $$ and $$ W = \{ w_{u}^{m}(t) \}_{m \in M, t \in T} $$, respectively.

Our objective function aims to maximize

Subject to constraints (1)–(4), where $$ C1 $$ enforces a safe distance $$ d_{safe} $$ between any two UAVs to avoid collisions, for any m, n in M and t in T, and $$ C2 $$ secures the network's algebraic connectedness with a positive measure $$ \lambda_2 $$. The target function aims to enhance equity and efficiency, as well as decrease the energy usage of all UAVs, in accordance with system needs. Constraint $$ C1 $$ controls the spatial mobility of UAVs, while constraints (2)–(4) manage the access control for GUs. Constraint $$ C2 $$ is based on the second smallest eigenvalue of the Laplacian matrix to guarantee that the UAVs sustain a linked network topology.

4. COMBINED MADRL AND DISTRIBUTED LEDGER

The IPPO method improves trajectory optimization for many UAVs by employing a decentralized approach. Every UAV functions within a partially observable Markov Decision Process (MDP) framework, making decisions based on its own observations rather than a common global state. The system's state is determined by aggregating all UAV observations, while actions are decided independently in each location.

1. Observation Space Every UAV has a specific observation area that contains crucial environmental data, including the locations of all UAVs and GUs, total throughput, satisfaction levels, and access regulations. This extensive observation space is created to avoid execution problems in simulation and acts as a placeholder because of the fixed input dimensions needed by neural networks. Let $$ \mathcal{O}_i $$ represent the observation space for UAV $$ i $$. The information can be expressed as a vector containing environmental data.

where $$ \mathbf{p}_j $$ is the location of UAV $$ j $$, $$ \mathbf{g}_k $$ is the position of GU $$ k $$, $$ \tau $$ is the accumulative throughput, $$ \sigma $$ is satisfaction levels, and $$ \alpha $$ is access policies.

2. Action Space A UAV's action space is defined by the direction and distance of movement. The collective action space for all UAVs is the combination of individual actions, with a dimensionality equal to twice the number of UAVs. The action space for UAV $$ i $$, represented as $$ \mathcal{A}_i $$, comprises movement direction $$ \theta_i $$ and distance $$ d_i $$. The communal action space $$ \mathcal{A} $$ is determined by

where $$ N $$ is the number of UAVs.

3. Reward Function The incentive function integrates a cooperative element shared by all UAVs and an individual element that encompasses penalties for border breaches, unsafe distances, connectivity problems, and energy usage. The incentive function design ensures that UAVs work together to optimize fair throughput while minimizing energy consumption and penalty infractions. The reward function for UAV $$ i $$ is denoted as $$ R_i $$.

where $$ C $$ is the cooperative component, $$ I $$ is the individual throughput, $$ B $$ represents border violation penalties, $$ D $$ represents unsafe distance penalties, $$ E $$ represents energy consumption, and $$ w_c $$, $$ w_i $$, $$ w_b $$, $$ w_d $$, $$ w_e $$ are the corresponding weights.

4. Actor-Critic Networks This methodology maintains distinct actor-critic networks for each UAV, updating them exclusively based on individual observations, in contrast to classic centralized training and exploration (CTDE) systems that could restrict the diversity of agent policies. This encourages a wider range of actions and decreases policy similarity among actors. Each UAV $$ i $$ has actor $$ \pi_i $$ and critic $$ Q_i $$ networks, which are defined by parameters $$ \theta_i $$ and $$ \phi_i $$ correspondingly.

The networks are updated using gradients from the collected experiences:

where $$ A_i $$ is the advantage function for UAV $$ i $$.

5. Algorithms The training algorithm requires UAVs to observe the state, make judgments using the actor network, execute actions, receive rewards, and update their networks with knowledge stored in memory. IPPO differs from MAPPO in that it utilizes only the buffered information of each agent during the update phase, instead of aggregating information from all agents. The IPPO algorithm is defined by the following iterative phases.

Algorithm

$$ \bullet $$ Observe state $$ \mathcal{O}_i $$ for UAV $$ i $$.

$$ \bullet $$ Choose action $$ a_i $$ using actor network: $$ a_i \sim \pi_i(\cdot|\mathcal{O}_i; \theta_i) $$.

$$ \bullet $$ Execute action $$ a_i $$ and observe reward $$ r_i $$ and new state $$ \mathcal{O}_i' $$.

$$ \bullet $$ Store transition $$ (\mathcal{O}_i, a_i, r_i, \mathcal{O}_i') $$ in memory.

$$ \bullet $$ Sample random mini-batch from memory and update $$ \theta_i $$ and $$ \phi_i $$ using stochastic gradient descent.

The key distinction of IPPO in the update phase is in the use of the experiences:

where $$ \eta $$ is the learning rate.

5. RESULTS AND DISCUSSION

UAV-assisted communication network simulation is a technology that aims to connect locations without ground-based infrastructure. It uses a 10 km x 10 km 3D simulated environment to replicate both rural and urban terrains. UAVs fly between 100 and 300 meters to communicate directly with ground users for optimal transmission. To simulate real-world user dispersal, 50 GUs are randomly dispersed over the simulation territory. The MADRL algorithm is used to balance current and future rewards with a learning rate of 0.01 and a discount factor of 0.95. The communication technicalities include a noise spectral density of -174 dBm/Hz and a transmit power of 20 dBm for UAVs and 23 dBm for GUs. The implementation details of the IPPO algorithm describe the MADRL model: Each UAV agent's policy and value networks are updated depending on observations and rewards to imitate learning. Each UAV contains a blockchain module that imitates DL technology for secure and immutable communication transactions. The networks have densely linked layers with Restricted Linear Unit (ReLU) activation and a softmax layer for action probability. The observation space contains important data such as the UAV's location, battery level, adjacent GUs, and network throughput. Specific actions describe UAV direction and distance motions for the next time slot, defining the action space. The incentive function encourages GUs to improve throughput, energy efficiency, and fairness.

The simulation results depicted in Figure 2 illustrate the comparative performance of four UAV communication strategies across a sequence of time slots. The primary strategy, MADRL with DL, is shown to achieve the highest throughput, a testament to the efficacy of integrating DL technology with advanced RL algorithms. This strategy exhibits a rapid ascent in throughput that stabilizes as it approaches the system's capacity limit, forming an S-shaped logistic curve that represents a realistic growth pattern in network throughput. The second strategy, standalone MADRL, although effective, does not reach the peak performance achieved by its DL-enhanced counterpart, suggesting that while MADRL is a robust approach, its integration with DL technologies offers significant improvements. This curve parallels the top performer at a slightly subdued growth rate, demonstrating substantial but not maximal efficiency. The traditional RL strategy curve rises at a more modest pace, reflecting a lower growth rate and capacity. This indicates that traditional RL, while capable, falls short of the more sophisticated MADRL techniques, particularly in environments that demand dynamic and complex decision-making. Lastly, the Static K-Means strategy lags behind, with the least growth rate and capacity, suggesting its relative inadequacy in adapting to the evolving demands of UAV communication networks. Its performance trajectory, while still positive, is the most gradual and plateaus at the lowest throughput level, underscoring the limitations of less dynamic optimization methods. The results collectively encapsulate the cumulative throughput achievements of the UAV networks over time, measured in Gbps.

Figure 2. Comparison investigation proposed framework based on MADRL with DL, standalone MADRL, traditional RL and static K-means.

Figure 3 presents a comparison of four UAV communication optimization techniques across four metrics, including cumulative rewards, fairness index, cumulative throughput, and energy consumption. After 5000 episodes, the proposed MADRL with DL approach demonstrates the highest rewards in the cumulative rewards graph, indicating its superior performance in accumulating benefits over time. The data analysis shows a steady and significant rise in rewards, suggesting that this strategy is highly effective in achieving the desired outcomes in the simulated scenario. The fairness index graph evaluates the fair distribution of resources among users or agents. The proposed MADRL with DL consistently exhibits good performance, with a fairness index close to one, indicating optimal fairness. This method effectively distributes resources in a fair manner, guaranteeing that no individual or group is given special treatment. The cumulative throughput graph demonstrates the amount of data successfully transmitted over the network. The proposed MADRL with DL approach achieves the highest throughput, indicating efficient network traffic management. This technique shows a consistent and substantial increase in throughput, indicating an effective optimization strategy for maximizing data flow in the network. The energy consumption graph compares the energy needed by each technique to achieve its objectives. The proposed MADRL with DL technique is notable for its minimal energy usage, emphasizing its effectiveness. It is essential to focus on energy efficiency for UAV operations to ensure sustainability and cost-effectiveness. The proposed technique is designed to be high-performing and energy-efficient. The proposed MADRL with DL approach outperforms the standalone MADRL, traditional RL, and static K-Means strategies in all performance metrics. These findings highlight the advantages of incorporating deep learning with DL technology in UAV communication systems, resulting in improved performance, fairness, throughput, and energy efficiency. Utilizing effective analysis rather than relying on a curve can highlight the enhanced benefits of using advanced algorithms to oversee intricate communication networks.

Figure 3. Throughput, rewards, fairness index and energy estimations.

Figure 4 compares the performance of various communication mechanisms using different numbers of UAVs. The methods evaluated are the proposed MADRL with DL, Standalone MADRL, Traditional RL, and Static K-Means. The fairness index is used to assess the equitable allocation of resources among UAVs or network users. The MADRL with DL proposal shows the highest fairness index, indicating its effectiveness in ensuring equitable conditions throughout the network. As the number of UAVs increases, fairness grows, demonstrating the effectiveness of the proposed MADRL with Deep Learning as the UAV fleet size scales up. The Standalone MADRL has a higher fairness index compared to the Traditional RL and Static K-Means, indicating a more equitable distribution of resources. The throughput represents the data transmission rate controlled by each approach. The graph shows that as the number of UAVs increases, all techniques enhance throughput, but the Proposed MADRL with DL performs better than the other methods, showcasing its superior capacity to efficiently manage more intricate connections and larger data transfers. The Standalone MADRL outperforms Traditional RL, while Static K-Means has the lowest throughput, indicating that advanced dynamic techniques can effectively utilize more UAVs to enhance network capacity. Similarly, energy consumption shows the energy usage for each technique. Reduced energy usage is favorable as it indicates a more effective use of resources. The Proposed MADRL with DL consumes more energy compared to previous options, but it offers a trade-off for increased fairness and throughput. The Standalone MADRL has somewhat greater efficiency compared to the Proposed MADRL with DL, while Traditional RL consumes less energy than both MADRL approaches. The Static K-Means algorithm, while highly energy-efficient, may exhibit lower performance compared to other algorithms, as seen by preceding measures. The data indicates that the Proposed MADRL with DL method achieves superior performance in fairness and throughput, albeit with increased energy usage. The Standalone MADRL method is characterized by a balanced combination of moderate energy consumption and performance. Traditional RL and Static K-Means are more energy-efficient but lag in network performance, showing a trade-off between energy efficiency and operational efficacy. This research can help identify the optimal technique for UAV network communication based on the specific needs for fairness, throughput, and energy efficiency.

Figure 4. Number of UAVs evaluation in terms of energy, throughput and fairness index.

6. CONCLUSIONS

This research marks a significant advancement in UAV communication systems, particularly beneficial in environments lacking ground-based infrastructure. By integrating MADRL with DLT, we have significantly enhanced the performance, fairness, and energy efficiency of UAV networks. The proposed system architecture leverages a cluster of UAVs to deliver equitable communication services, striking an optimal balance between network connectivity and resource utilization. The decentralized learning strategy, based on the IPPO technique, enables UAVs to tailor their operational strategies based on individual observations, fostering a dynamic and adaptive communication environment. While our findings demonstrate that the MADRL with DLT approach substantially outperforms traditional methods across various metrics, this study also opens several avenues for future research. Future work could explore the integration of more complex adaptive algorithms to further enhance the system's responsiveness to changing environmental conditions and user demands. Additionally, further research is needed to address potential challenges in scalability and manageability as the size and complexity of UAV networks increase. The complexities involved in real-world implementations, such as regulatory hurdles and varying environmental conditions, also present substantial challenges that require innovative solutions. Moreover, as UAV technologies and the frameworks for their operation evolve, continuous improvements in security measures will be essential to safeguard against increasingly sophisticated cyber threats. The adoption of newer cryptographic techniques and advanced security protocols will be crucial to ensure the integrity and confidentiality of the data transmitted within these networks.

DECLARATIONS