Spinal cord injury (SCI), especially in the cervical level leads to severe motor disabilities not only in the lower extremities but also in the upper limbs. These, and vegetative restrictions lead to lifelong dependency from other people and a reduction in quality of life. Tetraplegic persons which have a lesion in the cervical level between C6 and C5 have reduced or lost hand and finger function, while people with a lesion higher C5 also have restrictions in the elbow function. Additionally the range of motion of the shoulders is reduced. From C3 upwards should function and breathing can be reduced or lost.
With the help of so-called motor neuroprostheses lost motor functions can be restored to a certain extent. These systems are based on functional electrical stimulation (FES) [ 1 ], a method, where electrical pulses get sent to the motor neurons which then elicit action potentials and then evoke muscle contractions. Consequently, the target muscle groups need to be innervated. With this method, muscles can be 1) trained and then 2) their force be controlled.
To some extent, a means of control for neuroprostheses for the hand can be e.g., a simple switch or a shoulder position sensor [ 2 ]. However, as soon as the individual received restrictions in elbow function and shoulder mobility a brain-computer interface can be a new means of control.
Brain-computer interfaces (BCIs) are systems which generate control signals directly from mentally modulated brain activity [ 3 , 4 ]. In case of non-invasive systems, electroencephalogram (EEG) is recorded and with the help of signal processing methods and machine learning principles transferred into such control signals. Many applications and use scenarios exist, mainly they are to 1) restore, 2) replace and 3) improve parts of the function of the central nervous system. Or in other cases to 4) enhance people or to 5) use such systems in other research fields [ 5 ].
One of the aims of the Graz BCI group is to establish and restore hand and arm function in individuals with high spinal cord injury by using a brain-computer interface to detect and convert movement intentions into control signals (Figure 1(a) and Figure 1(b)).
Today, the research field is about to open a new era, where arm movement trajectories get decoded directly from brain signals. Currently, most advances are done in invasive research by intracortical recordings [ 6 - 10 ].
The goal of this review work is to highlight the development of BCI-controlled motor neuroprostheses from its beginnings to the current state of art. Since the Graz-BCI group is pioneering in this field, mainly works from this group are reviewed. First, we report on the development from mechanical orthotic systems to hybrid neuroprosthesis. Then, we discuss the different control approaches based on BCI. An outlook on the new developments and their practicality will be given in the discussion section.
2. Orthoses and Neuroprostheses for grasp restoration
Controlling the upper extremity, especially restoring the grasp function of a person with a high spinal cord injury in a non-invasive manner is a challenging task. This section shows the different approaches which were done by the Graz BCI group and in collaboration with the spinal cord injury center of Heidelberg University starting with simple orthosis through to complex motor neuroprosthesis.
Electric Hand Orthosis
One of the first attempts building an electric driven hand orthosis was shown by Müller [ 11 ]. The orthosis was developed to support the grasp function of the left hand of a tetraplegic end-user. Such reciprocal wrist orthoses (e.g., [ 12 ]) are normally used in patients who have residual wrist extension but lost finger function. If the wrist extends the fingers automatically get into a grasp position and if the wrist flexes the fingers get into the grasp release position. To make it useful for a tetraplegic user without wrist function it was powered with an electric motor and thus restoring the grasp function. The orthosis was fixed at six different points: 1) the forearm near the wrist joint, 2) the index finger near the first finger joint, 3) the index finger near the second finger joint, 4) the thumb joint for the palm shovel, 5) the back of the hand and 6) the thumb near the second thumb joint (see Figure 1(c)). The index and middle finger needed to be fixed at the orthosis and the other fingers move with them in a passive way. It was also necessary to support the paralyzed hand of the user which was done by using a palm shovel. It was designed in a way that it ended below the finger fold to not limit the grasp function.
Since orthotic systems for grasp restoration never worked satisfactorily and the development of easy to use functional electrical stimulation devices made huge progress, the idea of using FES to restore grasp function was evident [ 1 , 13 ]. The paralyzed muscles were stimulated using surface electrodes placed on the forearm. Those electrodes induced pulses, which elicited physiological action potentials on the efferent nerves. For this grasp, four muscle groups had to be stimulated: the finger and the thumb (M. ext. Digitorum communis EDC and M. ext. Pollicis longus EPL), for opening the hand and flex the fingers (M. flex. Digitorum superficialis FDS and M. flex. Digitorum profundus), for closing the hand and flex the thumb
Figure 1. (a) General idea of bypassing the interrupted pathways in the spinal cord with the help of a BCI and motor neuroprosthetics to restore hand and arm function by decoding of motor intentions. (b) Vision of the Graz BCI group: a non-invasive BCI-controlled motor neuroprosthesis. (c) Electrical hand orthosis controlled with a single switch BCI [ 11 ]. (d) BCI controls non-invasive grasp neuroprosthesis by stepwise activating different grasp phases. (e) BCI in the control of an implanted neuroprosthesis by stepwise activating different grasp phases. (f) Hybrid BCI to select type of grasp and breaks of stimulation, shoulder position sensor controls stimulation strength and therefore grasp forces. (g) Temporal coding of single motor imagery pattern for the control of hand (open, close) and elbow function (flexion, extension) with the help of a hybrid neuroprosthesis.
(M. flex. Pollicis longus FPL) and for stabilizing the hand (M. ext. Carpi radialis longus/brevis ECRL/ECRB). Stimulation frequency was 20 Hz and pulse width 300 µs. The amplitude of the stimulation pulses was adjusted individually and ranged up to 35 mA.
To activate the different muscles by using a simple switch on the wheelchair, the whole grasping pattern was divided into five grasp phases: 1) hand opening, 2) closing of fingers, 3) closing of thumb, 4) opening of hand and 5) idling state (hand is relaxed). The setup which was used consisted of three channels and each of them was adjusted independently. A pair of round electrodes (32 mm) was placed near the motor points of each of the required muscles. To keep the thumb up during the flexion of the fingers the FPL and the EPL were stimulated at the same time. The thumb needed strong activation, while the activation level of the FDS/FPL was set as low as possible to close the fingers sufficient.
By the application of a shoulder position sensor and an analog to stimulation pattern mapping, end-users are able to control the hand function with small movements of the contralateral shoulder.
Beside the development of surface based FES systems, also implantable systems were researched and developed [ 1 ]. One of these is the so called Freehand system® [ 14 ]. With this system the implanted electrodes are placed in the muscles of the forearm and the hand and they are subcutaneously connected to an eight-channel stimulator which is placed in the chest. The electrodes are implanted into the extensor digitorum profundus, extensor pollicis longus, abductor pollicis longus, flexor digitorum profundus, flexor pollicis longus (FPL), adductor pollicis, and oppoenuns pollicis. Each of the muscles were stimulated using short pulses on different positions to find the optimal position for the electrodes by assessing the muscle response. To provide thumb stability for the pinch an FPL split tendon transfer was performed. The power and stimulation parameters are supplied by an external control unit. This system is usually controlled with a shoulder position sensor placed on the contralateral shoulder and provides two grasp patterns as well as grasp fixation function.
The advantage of such a system is clearly, that the hand patterns are very stable over time, no adjusting on a day by day basis is necessary and there are no cables.
Currently, several such systems are used by Ajiboye et al. to restore hand and arm movement in spinal cord injured controlled with a Brain-Machine interface (BMI) using implanted microelectrodes [ 6 ].
The combination of FES with an orthosis leads to a hybrid neuroprosthesis, which can be used to restore hand, finger and elbow function in users with a high level SCI [ 15 ].
An elbow orthosis was developed on one side for a stable elbow position and on the other side to avoid fatigue in the upper arm muscles. It consisted of a self-locking, electrically lockable and unlockable elbow joint with a configurable weight support system. The exoskeletal orthosis is available for the left and right hand and can be extended via a rotational wrist module, a module for ulnar-radial abduction and a thenar wrist-stabilizing orthosis module. The control devices include a 2-axis shoulder position sensor and electromyographic recording hardware for measuring residual muscle activity. Furthermore, the orthosis was equipped with a strong anti-gravity module for supporting elbow flexion. To keep the wrist in a neutral position and enabling proper finger flexion, an additional wrist-stabilizing module was used. To simplify the usability of the FES, a neoprene sleeve based on the user’s right forearm was custom-made which ensures the correct electrode positions and reduces the setup time. By protraction/retraction or elevation/depression of the shoulder the user can control the degree of elbow flexion/tension or of hand opening/closing by stimulating the corresponding muscles.
3. BCI controls Neuroprosthetics: from hand to hand & elbow control
The following section describes the application of non-invasive BCI systems to control the above introduced means for hand/arm function restoration.
From Orthosis to Neuroprosthesis Control
The first step to control a paralyzed upper limb with a non-invasive BCI at the TU Graz was made by Pfurtscheller et al. in 2000 [ 11 ]. The goal was to restore rudimentary grasp function of a tetraplegic end-user. The person involved had a complete motor and sensory lesion below C5, as well as an incomplete lesion below C4. He has no elbow function in the right hand, but residual voluntary movement in his left shoulder and elbow, which lead to controlling the left hand with the electrically driven hand orthosis.
The BCI used for controlling this orthosis was trained with a cue-based MI [ 16 ] approach over a period of five months. Bipolar EEG from C3, Cz and C4 was recorded and analyzed by applying an adaptive autoregressive model (AAR) of order six with a following linear discrimination analysis (LDA).
First, the end-user tried to imagine left vs. right hand movement, as this has shown the best results in healthy participants so far. However, as this did not give the desired classification accuracies (65%), the end-user switched to imagine right hand or both feet movement which yielded better results (95%). This came in also very handy as the residual movement of the left biceps might have interfered with the classification of imagined left hand movement.
Offline analysis of the EEG data revealed that the high classification accuracy was a result of voluntarily induced beta oscillations (event-related-synchronization, ERS) in the foot representation area. Notable here is that those beta oscillations increased in dominance and had an earlier onset over the course of the training period.
After five months of training, the end-user was able to open and close the orthosis with nearly 100%, which enabled him to independently eat an apple.
Following, in 2003 Pfurtscheller et al. tried to restore a more sophisticated grasp with the application of a surface based motor neuroprosthesis based on FES [ 17 ]. The system was tested with the same end-user as in [ 11 ]. Again, the grasp of the left hand was restored by FES, as there is still residual movement in the left shoulder and elbow.
Again, imagination of both feet movement was used to control the BCI, as this has shown to be very stable. Notable here is that even though the end-user has hardly trained in the years between 2000 and 2003, the self induced beta-oscillation ERS could still be found over the foot representation area. The system was trained with 160 cue-based right-hand or foot movement imaginations and classification was done by using LDA with band filtered logarithmized signals of channel Cz and C3 (bipolar derivations) as features.
The FES then controlled the grasp movement in different steps, where every step was triggered by a beta burst induced by voluntary imagination of feet movement. Finally this enabled the end-user to grasp a cylinder with his paralyzed hand (Figure 1(d)).
With that first evidence was given that it is possible to combine FES systems with EEG-based BCIs, which on the one hand introduces high currents and voltages to the user and on the other hand measures EEG in μV-range.
Encouraged by this results, Müller-Putz et al. [ 18 ] showed that it is possible to train an end-user in a relatively short time period to control an implanted neuroprosthesis (Freehand system®) with a BCI. The end-user participating in this study was a 42-year old male with an SCI below C5 suffering from motor and sensory lesion. His volitional muscle activation is restricted to both shoulders and elbows, but there is no active wrist or finger function.
For the BCI to be used as a switch, EEG was derived from three bipolar channels over C3, Cz and C4.
The participant was instructed to execute certain tasks to find out which brain signals fit best as switching request. Therefore the end-user was told to imagine feet and left hand movements. These imageries were used for online training (Basket-paradigm [ 19 ]). During each trial a ball descended from the top to the bottom of a screen within 3 s and the end-user was instructed to “catch” the balls with the belonging baskets using MI. Depending on the executed imagery the basket moved to the left or right hand side.
After three days, several runs (40 trials per run) were performed and the left hand MI was chosen as switch for the freehand system.
To examine the performance of the experiment the BCI system was connected with the Freehand System. For controlling the lateral grasp, the pattern was divided into three phases: 1) finger and thumb extension, 2) finger flexion and thumb extension and 3) finger and thumb flexion. The end-user was instructed to make a grasp-release test [ 20 ] using the lateral grasp.
The test person was instructed to move a paperweight (250 g) from one place to another as often as possible within three minutes. During the interval, the paperweight was moved five times which means that 16 switches were performed.
The end-user was able to trigger a switch with a mean duration of 10.7 s (see Figure 1(e)).
Hybrid BCI Approaches
The goal of Rohm et al. [ 21 ] was to control a hybrid neuroprosthesis to perform elbow extension/flexion and hand opening/closing with the help of a hybrid BCI (hBCI) [ 22 , 23 ]. Hybrid BCIs consist of several input signals in which at least one is obtained via a BCI system (e.g. EEG) and others can be obtained by any other measurement method (e.g. EOG, assistive devices) [ 23 ]. Rohm et al. used the hBCI system to switch between the two different movement methods.
The participant in this single case study was a 41-year old male with an SCI at C4 with a minor overall spasticity. For this study an hBCI was combined with a hybrid neuroprosthesis. This BCI system included Laplacian derivations [ 24 ] from C3 and Cz positions.
At first the end-user underwent FES training, therefore the stimulation frequency was set to 2 - 6 Hz at the beginning to carefully activate the passive structures of the muscles and then the frequency was slowly increased up to 16 Hz until a tetanic muscle contraction was elicited.
After training the participant for 2 months, his muscles developed a level of fatigue resistance that was sufficient for execution of the following tasks. To switch between elbow and hand movement, a classifier had to be trained to distinguish between the two MI classes.
Therefore a short detection of MI switched from hand to elbow control. A longer detection lead to a pause state with muscle stimulation turned off.
The study comprised four different experiments which were the grasp and release test (GRT) [ 20 ], “Picking up a pen and signing a document”, “Eating a pretzel stick” and “Eating an ice cone”. During the GRT the end-user was able to transfer pegs and blocks over the frame of a box several times within a minute. The following experiment was “Picking up a pen and signing a document” which was accomplished by grasping a pen with a lateral grasp from a lower surface, lifting it up to the height of the upper legs, signing a document in front of the participant and finally putting back the pen to its initial position. To realize the “Eating a pretzel stick” task the end-user had to activate certain classes at predefined time points which was harder than expected. Finally, the “Eating an ice cone” task was performed best. Therefore the end-user grabbed the ice cone from a special holder and lifted it to his mouth so he could eat it. The overall average performance of the MI-BCI training was 70.5%.
Hand and Elbow Control
Kreilinger and colleagues [ 25 ] also demonstrated an hBCI system, based on EEG and elbow angle, for the control of the elbow and hand function by applying such a hybrid neuroprosthesis. Nine healthy participants and a male end-user with complete SCI at the level of C5 participated in the study.
EEG electrodes were placed over the motor cortex to record time-frequency patterns of the participants performing motor imagery to get control signals. The participants had a training session to learn imagining movements of their right hand or feet, before online classified time-frequency patterns were used to control the neuroprosthesis by performing short (~1 s) or long (continuous) motor imagery. Short commands were used to trigger the opening or closing of the hand, if the arm was fully flexed or extended or to move the arm to the nearest end position. Long commands were used to continuously move the arm into the direction furthest away from the current position. The position of the elbow was locked by the system, after a continuous command ended. This neuroprosthesis was controlled by using just one active BCI class. The end-user controlled his own arm (see Figure 1(g)) while the healthy participants controlled the arm of a second person, who was not visible for the participants, to avoid proprioceptive feedback but simulate spinal cord injury. Doing so, the moving (foreign) arm was filmed from top (shoulder downwards) and fed back through a screen which was mounted in a horizontal position in front of the participants.
All participants performed ten active sequences, each lasting 3 min with a 1 min break afterwards. During the break, false positive (FP) commands were counted. Five out of nine participants, including the end-user, were able to complete more than half of the sequences. In total 55.5% of all sequences were successfully completed. 8.2 commands/min were triggered during sequences and 4.7 FP/min during breaks.
This work gives a review mainly on the work done by the Graz BCI group in collaboration with the spinal cord injury center at Heidelberg University. The research started with a very basic control of an electrically driven orthosis, which was replaced by FES-based motor neuroprosthetic devices. With several steps and by the inclusion of the environment it was shown that hand as well as elbow movement could be restored and controlled with the help of non-invasive BCIs. Also, these studies showed that the principles work, however, the control was quite cumbersome. Mainly movement intentions of other limbs were used to get control over the target joints. It is desirable to provide more natural control for this type of BCI application. The following roadmap to decode goal-directed arm movements and grasps is currently under investigation (project Feel Your Reach): 1) we want to detect, whether a movement has a direct goal [ 26 ]; 2) then we are going to decode movement trajectories. Here, first results are already available and [ 27 , 28 ]; 3) Different types of grasps need to be differentiated. A more detailed overview can be found here [ 29 ]. All these results now need to be transferred to end-users which is future work. However, currently, also progress is made in terms of non-invasive hand-grasp neuroprosthesis. During the runtime of the European Horizon 2020 MoreGrasp project (http://www.moregrasp.eu) also a new, multi-electrode FES system has been developed. It allows stable finger positioning during supination/pronation, which was not possible with non-invasive systems so far [ 30 ].
When looking a little more to the future then we find out that the restoration of lost arm and hand function with motor neuroprosthesis is not possible in some cases because of denervated muscles in the forearm. This happens when the motor neurons from the spinal cord to the muscle have been injured, too. This means, FES cannot be used to get control over muscle activity. In such cases there are no other options. In many studies which apply intracortical derivations of single unit activity, robotic limbs are controlled [ 9 ]. Of course, this makes things easier, since a robotic arm is much more reliable than a paralyzed arm which gets fatigued, has problems in special positions, low degree of passive range of motion, no major spastic activity, and also sensation. A radical idea here is to amputate the limb and replace it by a robotic arm-hand system [ 31 ] which can help to return to more independence.
Similar systems are already in use in the field of targeted nerve reinnervation [ 32 ]. This is a surgical technique which transfers residual arm nerves to alternative muscle sites. After successful reinnervation, these target muscles produce electromyogram (EMG) signals that can be measured non-invasively and used to control the prosthetic arm and hand device.
To conclude, many developments have been done in the past 15 years and with the current knowledge and experiences and with new concepts discussed above major advances in the field of brain-con- trolled motor neuroprosthetics have been made and can be expected in the future.
This work was partly supported by the European H2020 project MoreGrasp (643955) and the ERC Consolidator Grant “Feel Your Reach” (681231). Many thanks to all end users who participated in these studies and to our collaborator Dr. R. Rupp from SCI center Heidelberg University for this long lasting and fruitful collaboration.