Muscle synergy analysis for gesture recognition based on sEMG images and Shapley value

2.1. sEMG image preprocessing

To convert the raw sEMG signal to a color sEMG image, it is necessary to preprocess the raw signal to obtain the sEMG signal parameter matrix. The parameters of the parameter matrix are mapped to 0-255 to obtain a greyscale image, and then the single-channel greyscale image is merged into a multi-channel color image.

We selected the sEMG signals acquired from sparse electrodes in the Ninapro dataset^[18]. The twelve gesture images corresponding to this data subset are shown in Figure 2 below. The raw signal is first band-pass filtered with a sampling rate of 100 Hz, and a 10-bit A/C conversion is performed^[19]. The resulting values are normalized to a range of minus one to plus one, corresponding to a voltage of minus 2.5 mV to plus 2.5 mV. This normalization is based on the maximum and minimum values of all data. The sEMG data from the ten acquisition modules are packaged in an ARM controller^[20]. Following the time domain sampling, the sampling duration is defined as one second, and the sEMG signal parameter matrix can be obtained, which is transformed into a sEMG grayscale image. A sliding window is taken; a pre-defined sliding distance is slid in the time domain direction (this sliding distance value is small to ensure that the gap between adjacent grayscale image information is negligible, and data enhancement is also performed to expand the data), and the sliding position is used as the starting point. The sampling duration remains defined as one second, and we can obtain another sEMG grayscale image. By repeating the above steps, the sEMG signal is transformed into a series of sEMG grayscale images. These images are then sorted by gesture labels and time domain. Finally, three adjacent grayscale images are taken for RGB three-channel transformation to obtain the sEMG color images we need, and the above steps are repeated to obtain the sEMG color image set. This image is the input of our network and the object of experimental analysis.

Figure 2. Twelve gestures and corresponding labels.

2.2. CNN model for gesture recognition

A CNN is a feed-forward neural network with artificial neurons that respond to a subset of the surrounding units in the coverage area, which is excellent for large image processing^[21]. The computational model based on deep CNNs is trained end-to-end, from the original pixel to the final category, without any additional information or manually designed feature extractors. Therefore, this method can effectively fulfill the requirement for experimental validation. A deep learning framework is used to recognize gestures from sEMG images and computationally elucidate patterns in transient sEMG images. We built a network architecture with four convolutional layers and three fully connected layers. This network is the most basic CNN, but the test results are still very good.

2.3. Critical electrode channel selection

Muscle synergy methods are analyzed by manually extracting specific features from sEMG signals. The Grad-CAM interpretable method can be embedded in a CNN, avoiding the step of hand-selecting features and thus obtaining information about muscle activation during the recognition process. In addition to this, Grad-CAM can explain the basis of gesture recognition by the network from a global perspective and get the contribution of features to the gesture recognition task. The electrode channel location where the contributing feature region is located was selected, and this electrode channel was used as the key electrode channel for muscle synergy analysis. Grad-CAM uses the gradient of any target concept that flows into the final convolution layer to generate a coarse localization mapping that highlights important regions of the predicted concept in the image^[22]. Given a gesture image as input, we propagate the image through the CNN part of the model and then obtain the raw score for that class by task-specific computation. The gradient of all classes is set to zero except for the desired class, which is set to one. This signal is then back-propagated to the corrected convolutional feature map, and we combine the two to compute the coarse Grad-CAM localization, with the result representing the specific decision the model must look for. Finally, we multiply the heat map with the bootstrap back propagation points to obtain a concept-specific Grad-CAM visualization.

The typical positioning map for the gesture category $$ c $$, $$ L_{\text{Grad-CAM}}^{c}\in {{\mathbb{R}}^{u\times v}} $$, can be calculated as the following steps. We first compute the fractional gradient $$ {{y}^{c}} $$ of gesture category $$ c $$ before softmax. $$ {{y}^{c}} $$ is the linear categorical logic score of the gesture category $$ c $$. $$ {{A}^{k}} $$ is a feature map about the convolutional layer, representing each element of the feature map of the $$ k $$th channel. $$ \alpha _{k}^{c} $$ is the neuron importance weights, which are obtained by the global average pooling (GAP) layer. The importance of the $$ k $$th channel for the gesture category $$ c $$, $$ \alpha _{k}^{c} $$, is calculated as

This weight $$ \alpha _{k}^{c} $$ represents a partial linearization of the underlying deep network from $$ A $$ and obtains the importance of the feature mapping $$ k $$ for the target gesture class $$ c $$. It is then obtained by performing a weighted combination of forward activation maps through a ReLU, and the ReLU is able to filter irrelevant features. The $$ L_{\text{Grad-CAM}}^{c}\in {{\mathbb{R}}^{u\times v}} $$ is measured as

However, class activation mapping (CAM) generates the feature maps of the penultimate layer K, $$ {{A}^{k}}\in {{\mathbb{R}}^{u\times v}} $$. These feature maps are then spatially merged using GAP and linearly transformed to generate a score $$ {{S}^{c}} $$ for each gesture class.

To generate the locality map of the modified image classification constructs, as described above, the order of summation should be exchanged to obtain $$ L_{\text{CAM}}^{c} $$, and then the score $$ {{S}^{c}} $$ is calculated as

Grad-CAM can help us understand the process of predicting gestures in CNN models. The high-importance feature region obtained by Grad-CAM allows us to intuitively understand which regions have a greater impact on the network, by which we can invert their corresponding muscles and thus understand which muscles produce more information for that labeled gesture to enable the network to perform the recognition task. On the other hand, this high-importance feature region also helps us to narrow down the channel information interaction and reduces a lot of redundant information for muscle synergy analysis.

2.4. Muscle Synergy Analysis

The total reward value of the gesture recognition process is the output $$ f\left( x \right) $$ obtained with all the input information entered into the network minus the output $$ f\left( \varnothing \right) $$ obtained without any information entered into the network. Here, '$$ \varnothing $$' denotes an empty input. The total contribution value obtained from the ten electrode channel inputs needs to be fairly distributed to each channel, and to fairly calculate the contribution of each channel individually, we use the Shapley value to consider the effect of the participation or non-participation of that channel in the input on the results for different gesture recognition situations^[23]. Therefore, the Shapley value for each input information about the channel is measured as

$$ {{\varnothing }_\text{Shapley}}(i) $$ is the Shapley value for the input $$ i $$. $$ S $$ is the number of electrodes participating in the input, and $$ N $$ is the total number of electrodes. $$ S $$ in formulas is the ten electrode channel information, and $$ f\left( {{x}_{s}} \right) $$ denotes the output of the neural network obtained from the input $$ {{x}_{S}} $$. Previous studies have focused on the interactions between two variables. Given an input $$ I $$ and a total number of participants $$ n $$, the total reward for all channels is calculated as

There is a simple definition of interaction. If the input $$ S $$ contains $$ m $$ channels, and the information of these $$ m $$ channels always functions together as input, then these $$ m $$ channels can be considered to form a coalition. The coalition can receive a reward, denoted by $$ {{\phi }_{S}} $$. It is usually different from when a single channel acts alone to participate in the game. The additional bonus $$ {{\phi }_{S}}-\sum\nolimits_{i\in S}{{{\phi }_{i}}} $$ received by the alliance can be quantified as an interaction. If $$ {{\phi }_{S}}-\sum\nolimits_{i\in S}{{{\phi }_{i}}}>0 $$, we consider that the participants in the coalition have a positive influence. On the contrary, it means that there is a negative or antagonistic effect between the group of variables.

However, this equation for measuring interactions can only be used in a single alliance. Their interactions are either purely positive or purely negative. We first discuss the interaction between two single channels during a certain gesture recognition process. Given two variables $$ i $$ and $$ j $$, $$ \phi \left( i|N \right) $$ and $$ \phi \left( j|N \right) $$ denote their Shapley Values, respectively. If the variables $$ i $$ and $$ j $$ always participate in the process together and are always absent together, then they can be considered to form a coalition. The reward received by the coalition is usually different from the sum of the rewards received by the variables $$ i $$ and $$ j $$ when they participate in the whole process alone. This coalition can be considered as a single variable, denoted by $$ {{S}_{ij}} $$. In this way, we can consider the entire process with only $$ n-1 $$ channels, including the variable $$ {{S}_{ij}} $$ and excluding the variables $$ i $$ and $$ j $$. The interaction between variables $$ i $$ and $$ j $$ is defined as an additional bonus brought by the coalition $$ {{S}_{ij}} $$. The extra contribution $$ B({{S}_{ij}}) $$ made by the cooperation of the variables $$ i $$ and $$ j $$ is calculated as

The variables $$ i $$ and $$ j $$ cooperate to obtain a high reward, and the interaction is positive if $$ B\left( {{S}_{ij}} \right)>0 $$. The interaction between variables $$ i $$ and $$ j $$ leads to a low reward; the interaction is negative if $$ B\left( {{S}_{ij}} \right)<0 $$.

We can extend the definition of interaction to multiple variables. We define the channels that always input information together as the set $$ A $$. $$ N $$ denotes the total number of channels. We consider the channels contained in $$ A $$ as a whole and define it as a single variable and is denoted by $$ \left[ A \right] $$. The interaction in $$ A $$ is $$ B([A]) $$, which is measured as

By using the SVMS method, we first verify quantitatively whether there is a synergistic effect between forearm muscles. Then, we verify the correlation between a single electrode and a small muscle block for which a single electrode is used. Finally, we define the positively correlated muscle blocks as a coalition and explore the synergy between muscle groups.

3.1. sEMG image preprocessing

The Ninapro (Non-Invasive Adaptive Prosthetics) dataset was divided into three sub-databases according to the acquisition procedure and subject characteristics, and we worked on the data from the first database. The first database contains 27 intact subjects (20 males and seven females), 25 of them using the left hand and two using the right hand, and their age was in the range of 28 ± 3.4 years. The acquisition targets sEMG signals from 12 basic finger movements, and the acquisition device contains multiple sensors for recording hand kinetics, kinematics, and corresponding muscle activity. Hand kinetics were measured using the finger force linear sensor FFLS, flexion and extension forces of all fingers, and abduction and adduction of the thumb were detected using strain gauge sensors. Ten MyoBok 13E200-50 electrodes (Otto Bock HealthCare GmbH) provided amplified, band-pass filtered, and root mean square (RMS) corrected versions of the raw sEMG signals corrected version of the original sEMG signal. The amplification gain of the electrodes was set to approximately 14000. The first eight electrodes were placed in an isometric path around the forearm. Electrode nine was placed on the upper-limb flexors and electrode ten on the upper-limb extensors. The labels of the dataset were categorized into action modes and resting modes without force by active segment detection. The data was scanned using analysis windows, which can be categorized as overlapping or non-overlapping. In order to obtain more samples, overlapping analysis windows are often used in practical applications. The detected action segments are temporally ordered. Using an overlapping analysis window with a fixed step size, the ten columns of data with the same action labels continue to be partitioned into a series of equally sized arrays under the temporal order. In overlapping analysis windows, the length of the analysis window is an important parameter. In general, the larger the length of the analysis window, the better the action recognition, but the longer the processing time. The response time of a real-time control system should be less than or equal to 300ms; otherwise, it will bring a sense of delay. However, it is crucial to emphasize that the accuracy of the recognition model itself is a prerequisite for ensuring interpretable analysis. At a sEMG signal sampling frequency of 100 Hz, we use an analysis window of size 100*10 and a sliding step of 1 to extract values from the original signal. During this process, we ensure that the gesture labels corresponding to these 100 frames of data are the same. We slide the window in temporal order until we encounter label differences, resulting in a series of arrays of size 100*10. We map the values of these arrays from 0 to 255 and use the fromarray function in the PIL (Python Imaging Library) to convert the 100*10 arrays into grayscale images. After obtaining the grayscale images, the array of three grayscale images adjacent to the same gesture is dimensionally transformed using the swapaxes function. A new 3D array is formed, and this 3D array is transformed to form the sEMG color image. By performing RGB three-channel color image conversion of the three adjacent grayscale images according to the sliding step down-search, we obtain the sEMG color image dataset for each gesture. In the data preprocessing, we used overlapping analysis windows. In the overlapping analysis window, the size of the analysis window and the sliding step size are two important parameters for practical applications. We choose a large window size of 100*10 and a small sliding step size of 1. This choice helps us to obtain a larger amount of data to ensure a better recognition result, which indirectly ensures the accuracy of the subsequent Shapley value calculation. Figure 3 below shows the sEMG signal images for the 12 gestures. Table 1 shows the number of sEMG color images extracted from each gesture in the dataset.

Figure 3. Color images of sEMG signals of twelve hand gestures. sEMG: Surface electromyography.

3.2. CNN model framework and hyper-parameter settings

The model we build contains four convolutional layers and three fully connected layers, where the size of the convolutional kernels of the convolutional layers is uniformly set to 3x3. These layers have a stride value of 1 and a padding value of 1 and utilize 32, 64, 128, and 128 convolutional kernels, respectively. Each convolutional layer is followed by a normalization layer and a ReLU activation function. Between each convolutional layer, there is a 2x2 pooling layer. The last three fully connected layers have outputs of 1024, 512, and 12, and then the classification results are output. We chose to build a CNN with only four layers of convolution. This was done to demonstrate the stability of the Shapley value for muscle analysis and highlight its usefulness within a normal network. The loss function is a cross-entropy loss function, and the optimizer is stochastic gradient descent. Figure 4 shows the architecture of the CNN network model. We use these image data as input for a CNN to perform multi-classification recognition tasks and obtain gesture recognition accuracy. At the end of the network, we introduce the Grad-CAM method and perform CAM with gradient weighting on the input images to generate heat maps that show the importance attribution of the network. Based on the information obtained from Grad-CAM, which indicates the input data that the network considers important, we analyze muscle synergy by removing redundant information for the gesture action.

Figure 4. CNN network model constructs and associated hyper-parameters. CNN: Convolutional neural network.

3.3. Experimental results

In this section, we present the experimental results. First, we provide an overview of our network training results, including the training loss, testing loss, training accuracy, and testing accuracy. Then, we showcase the results of the heat intensity map of the CAM, which we obtained by applying the Grad-CAM method. For the high-importance feature regions of the heat intensity map, we identified the target objects for SVMS analysis.

3.3.1. Gesture recognition results

We trained the network for 50 epochs, using 70% of the sEMG color images as the training set and 30% of the sEMG color images as the testing set. We used random seeds, thus ensuring that the data in the test set were not propagated by the neural network during the training process. Then, the network training was performed, and we obtained more satisfactory results, and the prediction accuracy reached 94.26%. Figure 5 illustrates the iterative process of training accuracy and testing accuracy. Figure 6 shows the comparison of the recognition accuracy of CNN with sliding windows and traditional machine learning methods.

Figure 5. The change of train accuracy and test accuracy for 50 iterations.

Figure 6. Recognition accuracy of CNN with sliding windows vs. traditional machine learning methods. CNN: Convolutional neural network; SVM: shapley-value-based muscle.

In comparison to previous machine learning methods, such as k-nn, SVM, Random Forests, and LDA on the Ninapro DB1 dataset, the CNN model achieved higher recognition accuracy in the gesture recognition task.

3.3.2. Grad-CAM results

We let the machine randomize some color images of sEMG signals and apply the Grad-CAM algorithm to calculate the predictions of the network for the regions of interest of the input. The gradient information of the last convolutional layer of the CNN model is first computed, and then the gradient information is weighted and averaged with the output of the convolutional layer to obtain a feature map. Then, the feature map is upsampled to obtain a heat map of the same size as the input image.

Figure 7 shows the results of the Grad-CAM; we find that the regions with attention are presented in a column arrangement, which also intuitively helps us to understand how much attention the network pays to the channels. The black area indicates that the electrode channel information in this section contributes 0 to the gesture recognition task, and not all of the electrode channel acquisition information has a positive effect on the gesture recognition task. Tian et al. outlined the significant impact of artificial intelligence technology on the application of sensors^[24]. Acquisition devices for sEMG signals also generate redundant information during gesture recognition tasks. Myoelectric acquisition devices have some connection to the engineering task; they could further explore state-of-the-art methods. The red box shows the regions with a high degree of attention. Among these 12 gesture recognitions, we found that the high attention regions of the network for the input are mainly concentrated in columns 4, 6, 7, 8, 9, and 10 by the Grad-CAM graph. Therefore, we mainly discuss the correlation between the forearm muscle groups corresponding to 4, 6, 7, and 8 and the upper-limb flexors and upper-limb extensors corresponding to 9 and 10.

Figure 7. The Grad-CAM plot of the network output. The highlighted region in the plot indicates the region where the network is of high importance for the input features, and the red box in the plot is our label for that region.

Figure 8. Muscle synergy analysis using non-negative matrix decomposition^[25].

3.4. Muscle Synergy Analysis

The way in which the game interactions are determined determines the Shpeley value calculation process. In order to accurately calculate the contribution value of each electrode channel to the gesture recognition task process, it is necessary to follow a permutation. This ensures that the overall game rounds are guaranteed to cover the interactions between all electrode channels. The combinations function of the itertools toolkit was used to generate all combinations, each of which corresponds to the kinds of game interactions in each round. Input the information of the corresponding electrode channel according to the kind of game interaction in each round. The electrodes appearing in the kind are kept input, and the information in the columns corresponding to the electrode channels not appearing in the kind are all set to zero. The modified image is recognized, and the prediction matrix of that input for the recognition task is taken out before the softmax layer of the network. This matrix contains the scores for that round for different gesture recognition tasks. Use these scores as the base scores for each gesture. After the combination cases and base scores are calculated for all rounds, the contribution of each channel is obtained using the Shapley value formula. In this section, we first verify that the synergy between different muscles is able to influence the network through a single-channel test. We took the minimum value of the prediction matrix parameters for the whole process as the base value of the membership score, representing the fraction of the contribution that the network considers for that part. We then focused on testing channel electrodes 4, 6, 7, and 8 to explore the synergy between the forearm muscle groups corresponding to these four electrodes and also tested the synergy between the upper-limb flexors and upper-limb extensors corresponding to channel electrodes 9 and 10 in 12 gestures. Finally, we took the inputs of the 4, 6, 7, and 8 electrode channels as one coalition member and the inputs of the 9 and 10 electrode channels as another coalition member and then explored the interactions between these two muscle groups and obtained some results.

Authors' contributions

Made substantial contributions to the conception and design of the study and performed data analysis and interpretation: Ao X, Wang F, Wang R

Provided administrative, technical, and material support: She J

Availability of data and materials

Not applicable.

Financial support and sponsorship

This work was supported by the National Natural Science Foundation of China under Grant 62106240; the Natural Science Foundation of Hubei Province, China, under Grant 2020CFA031; China Postdoctoral Science Foundation under Grant 2022M722943; Wuhan Applied Foundational Frontier Project under Grant 2020010601012175; the 111 Project under Grant B17040; and JSPS (Japan Society for the Promotion of Science) KAKENHI under Grants 20H04566 and 22H03998.

Conflicts of interest

All authors declared that there are no conflicts of interest.

Ethical approval and consent to participate

Before the initiation of data acquisition in the Ninapro database, each subject was given a thorough written and oral explanation of the experiment itself, including the associated risks; the subjects would then sign an informed consent form. The Ninapro acquisition experiment was conducted according to the principles expressed in the Declaration of Helsinki (www.wma.net/en/20activities/10ethics/10helsinki) and received approval from the Ethics Commission of the Canton Valais (Switzerland), ensuring the safety of the involved hardware and adherence to the WMA (World Medical Association).

Consent for publication

Not applicable.

Copyright

© The Author(s) 2023.