ndependent mobility is central to being able to perform activities of daily living by oneself. However, power wheelchairs are not an option for many people who, due to severe motor disabilities, are unable to use conventional controls. For some of these people, noninvasive brain– computer interfaces (BCIs) offer a promising solution to this interaction problem. Brain-Actuated Wheelchairs Millions of people around the world suffer from mobility impairments, with hundreds of thousands of them relying on power wheelchairs for activities of daily living [1]. However, many patients are not prescribed power wheelchairs either because they are physically unable to control the chair using a conventional interface or because they are deemed incapable of safely operating them [2]. Consequently, it has been estimated that between 1.4 and 2.1 million wheelchair users might benefit from a smart-powered wheelchair if it were able to provide a degree of additional assistance to the driver [3].
Digital Object Identifier 10.1109/MRA.2012.2229936 Date of publication: 8 March 2013
In our research with brain-actuated wheelchairs, we target a population that is or will become unable to use conventional interfaces due to severe motor disabilities. Noninvasive BCIs offer a promising new interaction modality that does not rely on a fully functional peripheral nervous system to mechanically interact with the world and instead uses brain activity directly. However, mastering the use of a BCI, i.e., with all new skills, does not come without a few challenges. Spontaneously performing mental tasks to convey one’s intentions to a BCI can require a high level of concentration, so it would result in a fantastic mental workload if one had to precisely control every movement of the wheelchair. Furthermore, due to the noisy nature of brain signals, we are currently unable to achieve the same information rates that you might get from a joystick, which would make it difficult to wield such levels of control even if one wanted to. Thankfully, we are able to address these issues through the use of intelligent robotics. Our wheelchair uses the notion of shared control to couple the intelligence of the user with the precise capabilities of a robotic wheelchair given the context of the surroundings [4]. It is this synergy that begins to make brain-actuated wheelchairs a potentially viable assistive technology of the not-so-distant future.
March 2013
Brain–Computer Interfaces The electrical activity of the brain can be monitored in real time using an array of electrodes, which are placed on the scalp in a process known as electroencephalography (EEG). To bypass the peripheral nervous system, we need to find some reliable correlates in the brain signals that can be mapped to the intention to perform specific actions. In the next two subsections, we discuss the philosophy of different BCI paradigms before explaining our chosen asynchronous implemenBecause every person tation for controlling the wheelchair. is different, we have to
select features that best
The BCI Philosophy Many BCI implementations rely on the subject reflect the MI task for attending to visual stimuli presented on a screen. each subject. Consequently, researchers are able to detect a specific event-related potential in the EEG, known as the P300, which is exhibited 300 ms after a rare stimulus is presented. For example, in one P300-based BCI wheelchair, the user is presented with a 3 # 3 grid of possible destinations from a known environment (e.g., the bathroom or kitchen in the user’s house) that are highlighted in a standard oddball paradigm [5]. The user then has to focus on looking at the particular place to which they wish to go. Once the BCI has detected the user’s intention, the wheelchair drives autonomously along a predefined route and the user is able to send a mental emergency stop command (if required) with an average 6-s delay. Conversely, another BCI wheelchair, which is also based on the P300 paradigm, does not restrict the user to navigating in known, premapped environments. Instead, in this design, the user is able to select subgoals (such as close left, far right, mid-ahead, etc.) from an augmented reality matrix superimposed on a representation of the surrounding environment [6]. To minimize errors (at the expense of command delivery time), after a subgoal has been preselected, the user then has to focus on a validation option. This gives the users more flexibility in terms of following trajectories of their choice; however, the wheelchair has to stop each time it reaches the desired subgoal and wait for the next command (and validation) from the user. Consequently, when driving to specific destinations, the wheelchair is stationary for more time than it is actually moving (as can be seen in [6, Figure 8]). Our philosophy is to keep as much authority with the users as possible while enabling them to dynamically generate natural and efficient trajectories. Rather than using external stimuli to evoke potentials in the brain, as is done in the P300 paradigm, we allow the user to spontaneously and asynchronously control the wheelchair by performing a motor-imagery (MI) task. Since this does not rely on visual stimuli, it does not interfere with the visual task of navigation. Furthermore,
66
when dealing with motor-disabled patients, it makes sense to use MI since this involves a part of the cortex that may have effectively become redundant, i.e., the task does not interfere with the residual capabilities of the patient. Previously, we demonstrated that it is possible to drive a wheelchair using such a protocol [7]. However, this earlier system relied on an expensive laser scanner to map the environment. In the following section, we show how a combination of relatively cheap sensors is sufficient to provide environmental feedback to the wheelchair controller. Moreover, the original protocol required the user to continuously deliver commands to drive the wheelchair, which resulted in a high user workload. Our current BCI protocol, coupled with shared control (see the “Shared Control Architecture” section), has reduced this workload. In our MI paradigm, the user is required to imagine the kinaesthetic movement of the left hand, the right hand, or both feet, yielding three distinct classes. During the BCI training process, we select the two most distinguishable classes to provide a reliable mapping from the MI tasks to control actions (e.g., imagine left-hand movements to deliver a turnleft command and right-hand movements to turn right). To control our BCI wheelchair, at any moment, the user can spontaneously issue a high-level turn-left or turn-right command. When one of these two turning commands is not delivered by the user, a third implicit class of intentional noncontrol exists, whereby the wheelchair continues to travel forward and automatically avoid obstacles where necessary. Consequently, this reduces the user’s cognitive workload. The implementation is discussed in the “Motion Planning” section. The BCI Implementation Since we are interested in detecting MI, we acquire monopolar EEG at a rate of 512 Hz from the motor cortex using 16 electrodes (see Figure 1). The electrical activity of the brain is diffused as it passes through the skull, which results in a spatial blur of the signals, so we apply a Laplacian filter, which attenuates the common activity between neighboring electrodes and consequently improves our signal-to-noise ratio. After the filtering, we estimate the power spectral density (PSD) over the last second in the band 4–48 Hz with a 2-Hz resolution [8]. It is well-known that when one performs the MI tasks, corresponding parts of the motor cortex are activated, which, as a result of event-related desynchronization, yields a reduction in the mu band power 8–13 Hz over these locations (e.g., the right hand corresponds to approximately C1 and the left hand to approximately C2 in Figure 1). To detect these changes, we estimate the PSD features every 62.5 ms (i.e., 16 times/s) using the Welch method with five overlapped (25%) Hanning windows of 500 ms. Because every person is different, we have to select features that best reflect the MI task for each subject. Therefore, canonical variate analysis is used to select subject-specific features that maximize the separability between the different tasks and that are most stable (according to cross-validation on the training data) [9]. These features are then used to train
•
IEEE ROBOTICS & AUTOMATION MAGAZINE
•
MARCh 2013
a Gaussian classifier [10]. Decisions with a confidence on the probability distribution that are below a given rejection threshold are filtered out. Finally, evidence about the executed task is accumulated using an exponential smoothing probability integration framework [11]. This helps to prevent commands from being delivered accidentally.
GND 1
Fz Wheelchair Hardware Our brain-controlled wheelchair is based on 2 4 5 6 3 a commercially available mid-wheel drive FC3 FC1 FCz FC2 FC4 model by Invacare Corporation. We have 7 8 9 10 11 made the following modifications, which C3 C1 Cz C2 C4 allow us to control the wheelchair directly 13 14 15 12 16 from a laptop computer. CP1 CPz CP2 CP4 CP3 1) We developed a remote joystick module that acts as an interface between a laptop REF computer and the wheelchair’s CANBUSbased control network. 2) We added a pair of wheel-encoders to the central driving wheels to provide the wheelchair with feedback about its own motion. 3) We added an array of ten sonar sensors and two webcams to the wheelchair to provide environmental feedback to the Figure 1. The active electrode placement over the motor cortex for the acquisition of EEG controller. data based on the international 10–20 system. (The triangle at the top represents the nose.) 4) We mounted an adjustable 8-in display to provide visual feedback to the user. 5) We built a power distribution unit to hook up all the two webcams are positioned facing forward directly above sensors, the laptop, and the display to the wheelchair’s each of the front castor wheels. batteries. The complete BCI wheelchair platform is shown in Wheel Encoders Figure 2. The positions of the sonars are indicated by the The encoders return 128 ticks per revolution and are geared white dots in the center of the occupancy grid, whereas the up to the rim of the drive wheels, resulting in a resolution of
Probability of Free Space Green > 70% Black 30–70% Red < 30% User Feedback
Sonars
BCI Feedback Bar (Top)
Occupancy Grid
Camera Streams with Detected Obstacles Highlighted
Figure 2. The complete brain-actuated wheelchair. The wheelchair’s knowledge of the environment is acquired by the fusion of complementary sensors and is represented as a probabilistic occupancy grid. The user is given feedback about the current status of the BCI and about the wheelchair’s knowledge of the environment.
March 2013
•
IEEE rOBOTIcS & aUTOMaTION MaGaZINE
•
67
EEG Auxiliary Laptop (Required for BCI)
User Input Discrete Button Input Other Devices, e.g., Joystick)
acceleration or slippage occur and the odometry does not receive any external correcting factors, then the model will begin to accumulate significant errors [12]. Shared Control Architecture The job of the shared controller is to determine the meaning of the vague, high-level user input (e.g., turn left, turn right, keep going straight), given the context of the surrounding environment [4]. We do not want to restrict ourselves to a known, mapped environment since it may change at any time (e.g., due to human activities), so the wheelchair must be capable of perceiving its surroundings. The shared controller can then determine what actions should be taken based on the user’s input, given the context of the surroundings. The overall robotic shared control architecture is depicted in Figure 3, and we discuss the perception and planning blocks of the controller in the next few subsections. Perception Perception comes naturally to humans, but is difficult to simulate in robotics. To begin with, choosing appropriate sensors is a not a trivial task and tends to result in a tradeoff between many issues, such as cost, precision, range, robustness, sensitivity, complexity of postprocessing, and so on. Furthermore, no single sensor by itself seems to be sufficient. For example, a planar laser scanner may have excellent precision and range, but will only detect a table’s legs, reporting navigable free space between them. Other popular approaches, like relying solely upon cheap and readily available sonar sensors have also been shown to be unreliable for safety-critical applications [14]. To overcome these problems, we propose to use the synergy of two low-cost sensing devices, which will compensate for each other’s drawbacks and complement each other’s strengths. Therefore, we use an array of ten close-range sonars, with a wide detection beam, coupled with two standard off-the-shelf USB webcams, for which we developed an effective obstacledetection algorithm. We then fuse the information from each sensor modality into a probabilistic occupancy grid, as discussed in the “Updating the Occupancy Grid” section. Computer Vision-Based Obstacle Detection The obstacle-detection algorithm is based on monocular image processing from the webcams, which run at 10 Hz. The concept of the algorithm is to detect the floor region and label
Main Laptop Target Acquisition Computer Vision Obstacle Detection
Figure 3. The user’s input is interpreted by the shared controller given the context of the surroundings. The environment is sensed using a fusion of complementary sensors, and the shared controller then generates appropriate control signals to navigate safely on the basis of the user input and the occupancy grid.
2.75 # 10 -3 m translation of the inflated drive wheel per encoder tick. We use this information to calculate the average velocities of the left and the right wheels for each time-step. This feedback is important for regulating the wheelchair control signals, and we also use it as the basis for dead reckoning (or estimating the trajectory that has been driven). We apply the simple differential drive model derived in [12]. To ensure that the model is always analytically solvable, we neglect the acceleration component. In practice, since in this application we are only using odometry to update a 6 # 6-m map, this does not prove to be a problem. However, if large degrees of
(a)
(b)
(c)
(d)
(e)
Figure 4. The obstacle-detection algorithm is based on the computer vision approach posed in [13] but has been adapted for monocular vision. The floor is deemed to be the largest region that touches the base of the image but does not cross the horizon. (a) Original image. (b) Edge detection. (c) Distance transform (exaggerated contrast). (d) Watershed segmentation. (e) Detected obstacles (red).
68
•
IEEE ROBOTICS & AUTOMATION MAGAZINE
•
MARCh 2013
everything that does not fall into this region as an obstacle; we follow an approach similar to that proposed in [13], albeit with monocular vision, rather than using a stereo head. The first step is to segment the image into constituent regions. For this, we use the watershed algorithm, since it is fast enough to work in real time [15]. We take the original image [Figure 4(a)] and begin by applying the well-known Canny edge-detection, as shown in Figure 4(b). A distance transform is then applied such that each pixel is given a value that represents the minimum Euclidean distance to the nearest edge. This results in the relief map as shown in Figure 4(c), with a set of peaks (the farthest points from the edges) and troughs (the edges themselves). The watershed segmentation algorithm itself is applied to this relief map, using the peaks as markers, which results in an image with a (large) number of segments [see Figure 4(d)]. To reduce the number of segments, the adjacent regions with similar average colors are merged. Finally, the average color of the region that has the largest number of pixels along the base of the image is considered to be the floor. All the remaining regions in the image are classified either as obstacles or as navigable floor, depending on how closely they match the newly defined floor color. The result is shown in Figure 4(e), where the detected obstacles are highlighted in red. Since we know the relative position of the camera and its lens distortion parameters, we are able to build a local occupancy grid that can be used by the shared controller, as is described in the following section. Updating the Occupancy Grid At each time-step, the occupancy grid is updated to include the latest sample of sensory data from each sonar and the output of the computer vision obstacle-detection algorithm. We extend the histogram grid construction method described in [16] by fusing information from multiple sensor types into the same occupancy grid. For the sonars, we consider a ray to be emitted from each device along its sensing axis. The likelihood value of each occupancy grid cell that the ray passes through is decremented, while the final grid cell (at the distance value returned by the sonar) is incremented. A similar process is applied for each column of pixels from the computer vision algorithm, as shown in Figure 5. The weight of each increment and decrement is determined by the confidence we have for each sensor at that specific distance. For example, the confidence of the sonar readings being correct in the range 3–50 cm is high, whereas outside this range it is zero (note that the sonars are capable of sensing up to 6 m, but given that they are mounted low on the wheelchair, the reflections from the ground yield a practical limit of 0.5 m). Similarly, the computer vision algorithm only returns valid readings for distances between 0.5 and 3 m. Using this method, multiple sensors and sensor modalities can be integrated into the planning grid. As the wheelchair moves around the environment, the information from the wheel-encoder-based dead-reckoning system is used to translate and rotate the occupancy grid cells
(x, y)
d = f(x, y)
Figure 5. Each column of pixels is scanned from bottom to top in order to detect the nearest obstacle (assuming it intersects with the ground). The estimated distance from the wheelchair to this obstacle is a function of the (x, y) pixel coordinates and the camera distortion parameters.
such that the wheelchair Many BCI implementations remains at the center of the map. In this way, the rely on the subject cells accumulate evidence over time from multiple attending to visual stimuli sensors and sensor modalities. As new cells enter the presented on a screen. map at the boundaries, they are set to “unknown,” or “50% probability of being occupied,” until new occupancy evidence (from sensor readings) becomes available. Motion Planning All the motion planning is done at the level of the occupancy grid, which integrates the data from multiple sensors. We base
Detection Zones
LF2
F2
RF2
LF1 LC1 y
F1
RF1 RC1
x LB1 Wheelchair Figure 6. Diagram of the wheelchair centered in an 8 # 8-m occupancy grid (to scale). The wheelchair obstacle-detection zones are labeled, and the origin of the coordinate system is on the center of the wheelchair’s driving axle. RB1
March 2013
•
IEEE rOBOTIcS & aUTOMaTION MaGaZINE
•
69
our controller on a dynamical system approach to navigation, since this allows us to easily incorporate the notion of obstacles (repellors) and targets (attractors), and results in naturally smooth trajectories [17]. Previously, we implemented such a control strategy on a circular mobile robotic platform, which was successfully controlled by motor-disabled patients using a BCI [18]. With no user input, the dynamic system causes the wheelchair to We demonstrate that move forward and automatically avoid any obstaboth new and cles that it comes across. In practice, this is realized experienced BCI wheelchair by adding repellors into the dynamic system operators are able to according to the obstacle densities in the occucomplete a navigation pancy grid. Rather than simply looking at the dentask successfully. sities in radial directions from the robot, as was sufficient in [18]—to account for the fact that the wheelchair’s shape and motion is more complex than the circular robot— we define a set of N = 10 zones within the occupancy grid, as shown in Figure 6. These zones are split into three sets such that if obstacles were present in them, X c = {RB1, LC1, LF1, LF2} would cause clockwise rotations of the wheelchair, X a = {LB1, RC1, RF1, RF2} would cause anticlockwise rotations, and X n = {F1, F2} would not affect the rotational velocity of the wheelchair. Each zone, z i ! X, has a center (z ix, z iy) and an associated repulsion strength m i < 0 ! K, which is determined according to the position of the zone relative to the wheelchair, such that K = K c , K a , K n . The likelihood of there being an obstacle in each zone is { i ! [0, 1] . The rotational velocity ~ is then
~
whose origin is in the center of the wheelchair’s axis (as shown in Figure 6). Similarly, for the translational velocity, v, each zone has an associated translational repellor, c i < 0 v = v max + / c i { i, i =1 N
v ! [0, + v max] .
(4)
= K ~ / m i S i { i, i =1
N
~
! [- ~ max, + ~ max],
(1) (2)
K~ =
mi ! Kc
/
~ max , | mi |
S i = sgn (- z ix # z iy),
S i ! {- 1, 0, + 1},
(3)
where the constant K ~ ensures that | ~ | is not greater than the maximum possible rotational velocity ~ max . Note that K ~ in (1) assumes that the obstacle-detection zones in X c and X a, and their corresponding K c and K a values, are symmetric, as it is in our case. In general, this makes sense, since you would expect symmetric behavior from the wheelchair for symmetric stimuli. (If this is not the case, one should take K ~ to be the maximum value of K ~c and K ~a , computed using K c and K a , respectively. However, this would result in asymmetric behavior of the wheelchair.) S i simply encodes the sign (direction) of the resultant rotation, assuming that (z ix, z iy) is the Cartesian center of the ith zone, in a coordinate system
70
The c i values are chosen empirically according to the dynamics of the wheelchair and the reliability of the sensors, such that, for example, when the zone F1 reaches 50% occupancy, the wheelchair will stop. Therefore, we set the c that corresponds to zone F1 to be -2vmax, whereas the c that corresponds to zone F2 is -0.75v max . When the user issues a BCI command (either a high-level turn left, or turn right), the wheelchair should turn up to a maximum of 45° in the direction indicated, depending on the environmental context. To achieve this, an additional corresponding virtual attractor zone is placed in the occupancy grid 1 m in front of the wheelchair, at an angle of 45° in the direction indicated by the BCI command. This attractor zone has a corresponding { i = 1.0 , m i = 0.5 , and c i = 0.0, such that in practice it only affects the rotational velocity dynamic system. Note that m is a positive value when acting as an attractor. The attractor remains in the dynamic system until the wheelchair has turned up to 45° in the corresponding direction or a new BCI command is delivered or until a timeout has been reached (in our case 4 s), at which point, it is removed. We extend the dynamic system by exploiting the fact that we have a human in the loop to enable an additional docking behavior. Such a behavior is important if the system is to be useful outside of experimental lab conditions. Therefore, if the occupancy of zone F1 or F2 is greater than an empirically set activation threshold, K T , providing there is no user input, the rotational velocity of the chair will be set to 0 rad–s. The translational velocity will still be controlled by the dynamic system, such that the wheelchair will slow down smoothly and stop in front of the object. At any point, the user is able to deliver a left or right command to initiate an obstacle-avoidance maneuver. If the user remains in a state of intentional noncontrol, once the wheelchair has completed the docking procedure, it will remain stationary and wait for further user input. In the current implementation, the user is not able to stop the chair in free space; instead the chair will stop when it has docked to a potential target. In the future, this control strategy could easily be extended to include an additional BCI command (or another biosignal, in the case of a hybrid approach) to implement an explicit stop signal. Evaluation We demonstrate that both new and experienced BCI wheelchair operators are able to complete a navigation task successfully. Furthermore, unlike in the P300-based systems, not only is the user in continuous spontaneous control of the wheelchair, but the resultant trajectories are smooth and intuitive (i.e., no stopping unless there was an obstacle, and users can voluntarily control the motion at all times).
•
IEEE ROBOTICS & AUTOMATION MAGAZINE
•
MARCh 2013
Manual Condition Target Table 2
BCI Condition Target Table 2 600
Task Completion Time (s)
Manual Benchmark Condition (Left Bars) Versus BCI Condition (Right Bars) Time Moving Time Stationary 500 400 300 200 100 0 s1 s2 Subject Figure 8. The average time required to complete the task for each participant in a benchmark manual condition (left bars) and the BCI condition (right bars). The wheelchair was stationary, awaiting user input, for only a small portion of the trial. s3 s4
Target Table 1
Target Table 1
Finish Position Start Position (a)
Finish Position Start Position (b)
User Command: Turn Right User Command: Turn Left Figure 7. The trajectories followed by subject s3 on (a) one of the manual benchmark trials compared with (b) one of the BCI trials. These trajectories were reconstructed with odometry using the independent reconstruction method [19].
Participants Mastering an MI BCI requires extensive training over a period of weeks or months to generate stable volitional control; it is not simply a case of putting a cap on and starting to drive. Therefore, we performed an initial evaluation with four healthy male subjects, ages 23–28. All subjects were experienced BCI users who had participated in at least 12 h of online MI BCI training and other BCI experiments over the previous few months. They all had some previous experience driving a BCI-based telepresence mobile robot, which requires a higher level of performance than simply moving a cursor on a screen [18]. Subjects s1 and s2 had no previous experience driving a BCI-controlled wheelchair, whereas subjects s3 and s4 had each spent several hours driving the BCI wheelchair. Subject s1 used MI of both feet to indicate “turn left” and of the right hand to mean “turn right”; all the other subjects used lefthand MI to turn left and right-hand MI to turn right. Experiment Protocol As a benchmark, the subject was seated in the wheelchair and was instructed to perform an online BCI session before actually driving. In this online session, the wheelchair remained stationary and the participant simply had to perform the
appropriate MI task to move a cursor on the wheelchair screen in the direction indicated by a cue arrow. There was a randomized balanced set of 30 trials, separated by short resting intervals, which lasted around 4–5 min, depending on the performance of the subject. After the online session, participants were given 15–30 min to familiarize themselves with driving the wheelchair using each of the control conditions: a two-button manual input, which served as a benchmark, and the BCI system. Both input paradigms allowed the users to issue the left and right commands at an intertrial interval of 1 s. The actual task was to enter a large open-plan room through a doorway from a corridor, navigate to two different tables while avoiding obstacles and passing through narrow openings (including other nontarget tables, chairs, ornamental trees, and a piano), and finishing by reaching a second doorway exit of the room (as shown in Figure 7). When approaching the target tables, the participants were instructed to wait for the wheelchair to finish docking to the table, and when it had stopped, issue a turning command to continue on their journey. The trials were counterbalanced such that users began with a manual trial, performed two BCI trials, and finished with another manual trial. Results and Discussion All subjects were able to achieve a remarkably good level of control in the stationary online BCI session, as can be seen in Table 1. Furthermore, the actual driving task was completed successfully by every subject for every run, and no collisions occurred. A comparison between the typical trajectories followed under the two conditions is shown in Figure 7. The statistical tests reported in this section are paired Student’s t-tests. A great advantage that our asynchronous BCI wheelchair brings, compared with alternative approaches like the P300based chairs, is that the driver is in continuous control of the wheelchair. This means that the wheelchair follows natural
March 2013
Table 1. Confusion matrices and accuracy prior to controlling an actual wheelchair. s1 L Left class Right class Accuracy (%) 13 0 93.3 R 2 15 L 12 0 90.0 s2 R 3 15 L 14 0 96.7 s3 R 1 15 L 15 0 s4 R 0 15
100.0
•
IEEE rOBOTIcS & aUTOMaTION MaGaZINE
•
71
trajectories, which are determined in real time by the user (rather than following predefined ones, like in [5]), and also spends a large portion of the navigation time in motion (see Figure 8). This is not the case with some state-of-the-art P300-controlled wheelchairs, where the wheelchair has to spend between 60 and 80% of the maneuver time stationary, waiting for input from the user (compare Figure 8 of this article with [6, Figure 8]). In terms of path efficiency, there was no significant difference (p = 0.6107) among subjects between the distance traveled in the manual benchmark condition (43.1 ! 8.9 m) and that in the BCI condition (44.9 ! 4.1 m). Although the actual environments Our philosophy is to keep were different, the complexity of the navigation as much authority with was comparable with that of the tasks investigated the users as possible, on a P300-based wheelchair in [6]. In fact, the while enabling them to average distance traveled for our BCI condition dynamically generate (44.9 ! 4.1 m) was greater than that in the longest natural and efficient task of [6] (39.3 ! 1.3 m), yet on average our particitrajectories. pants were able to complete the task in 417.6 ! 108.1 s, which was 37% faster than the 659 ! 130 s reported in [6]. This increase in speed might (at least partly) be attributed to the fact that our wheelchair was not stationary for such a large proportion of the trial time. Across subjects, it took an average of 160.0 s longer to complete the task under the BCI condition (see Figure 8, p = 0.0028). This is probably due to a combination of subjects issuing manual commands with a higher temporal accuracy and a slight increase in the number of turning commands that were issued when using the BCI (cf. Figure 7), which resulted in a lower average translational velocity. It should be noted that in the manual benchmark condition, the task completion time varied slightly from subject to subject, as the experiments were carried out on different days, where the changes in lighting conditions affected the computer vision system. On brighter days, some shadows and reflections from the shiny wooden floor caused the wheelchair to be cautious and slow down earlier than on dull days, until the sonars confirmed that there was not an obstacle present. Therefore, it makes more sense to do a within-subjects comparison, looking at the performance improvement or degradation on a given day, rather than comparing absolute performance values between subjects on different days. From Figure 8, it can be seen that for the inexperienced users (s1 and s2), there was some discrepancy in the task completion time between the benchmark manual condition and the BCI condition. However, for the experienced BCI wheelchair users (s3 and s4), the performance in the BCI
72
condition is much closer to the performance in the manual benchmark condition. This is likely to be due to the fact that performing an MI task, while navigating and being seated on a moving wheelchair, is much more demanding than simply moving a cursor on the screen (cf. the stationary online BCI session of Table 1). In particular, aside from the increased workload, when changing from a task where one has to deliver a particular command as fast as possible following a cue to a task that involves navigating asynchronously in a continuous control paradigm, the timing of delivering commands becomes very important. To drive efficiently, the user needs to develop a good mental model of how the entire system behaves (i.e., the BCI, coupled with the wheelchair) [20]. Clearly, through their own experience, subjects s3 and s4 had developed such mental models and were therefore able to anticipate when they should begin performing an MI task to ensure that the wheelchair would execute the desired turn at the correct moment. Furthermore, they were also more experienced in refraining from accidentally delivering commands (intentional noncontrol) during the periods where they wanted the wheelchair to drive straight forward and autonomously avoid any obstacles. Conversely, despite the good online BCI performance of subjects s1 and s2, they had not developed such good mental models and were less experienced in controlling the precise timing of the delivery of BCI commands. Despite this, the use of shared control ensured that all subjects, whether experienced or not, could achieve the task safely and at their own pace, enabling continuous mental control over long periods of time (>400 s, almost 7 min). Conclusions In this article, we saw how a viable brain-actuated wheelchair can be constructed by combining a brain–computer interface with a commercial wheelchair, via a shared control layer. The shared controller couples the intelligence and desires of the user with the precision of the machine. We found that this enabled both experienced and inexperienced users to safely complete a driving task that involved docking to two separate tables along the way. Furthermore, we compared our results with those published on other state-of-the-art brain-controlled wheelchairs that are based on an alternative synchronous stimulus-driven protocol (P300). Our asynchronous MI approach gives users greater flexibility and authority over the actual trajectories driven, since it allows users to interact with the wheelchair spontaneously, rather than having to wait for external cues as was the case in [5] and [6]. Moreover, combining our BCI with a shared control architecture allows users to dynamically produce intuitive and smooth trajectories, rather than relying on predefined routes [5] or having to remain stationary for the majority of the navigation time [6]. Although there was a cost in terms of time for the inexperienced users to complete the task using the BCI input compared with a manual benchmark, the experienced users were able to complete the task in comparable times under
•
IEEE ROBOTICS & AUTOMATION MAGAZINE
•
MARCh 2013
both conditions. This is probably a result of their developing good mental models of how the coupled BCI-shared control system behaves. In summary, the training procedure for spontaneous MI-based BCIs might take a little longer than that for stimulus-driven P300 systems, but ultimately it is very rewarding. After learning to modulate their brain signals appropriately, we demonstrated that both experienced and inexperienced users were able to master a degree of continuous control that was sufficient to safely operate a wheelchair in a real-world environment. They were successful in every case in completing a complex navigation task using mental control over long periods of time. One participant remarked that the MI BCI learning process is similar to that of athletes or musicians training to perfect their skills; when they eventually succeed, they are rewarded with a great sense of self-achievement. The Future We have already begun evaluating our brain-actuated wheelchair with motor-disabled patients in partnership with medical practitioners and rehabilitation clinics, but this is an arduous process that will take significantly longer than the initial trials with healthy subjects for a number of reasons, including that patients tend to take part in fewer sessions per week and generally tire more quickly than healthy participants. This leads us to another exciting new challenge for the future of such shared control systems. Since each user’s needs are not only different but also change throughout the day (e.g., due to fatigue, frustration, etc.), it is not sufficient that a shared control system offers a constant level of assistance. Furthermore, if this assistance is not well matched to the user, it could lead to degradation or loss of function. Therefore, we are developing shared control systems that adapt to the user’s evolving needs, given not only the environmental context, but also the state of the user. This will allow people to use intelligent assistive devices in their day-to-day lives for extended periods of time. Acknowledgments This work was supported by the European ICT Project TOBI (FP7-224631) and the Swiss National Science Foundation through the NCCR Robotics. This article only reflects the authors’ views, and funding agencies are not liable for any use that may be made of the information contained herein. References
[1] A. V. Drongelen, B. Roszek, E. S. M. Hilbers-Modderman, M. Kallewaard, and C. Wassenaar, “Wheelchair incidents,” Rijksinstituut Voor Volksgezondheid en Milieu RIVM, Bilthoven, NL, Tech. Rep. 318902 012, Feb. 2010. [2] A. Frank, J. Ward, N. Orwell, C. McCullagh, and M. Belcher, “Introduction of a new NHS electric-powered indoor/outdoor chair service: Benefits, risks and implications for prescribers,” Clinical Rehab., vol. 14, no. 6, pp. 665–673, 2000. [3] R. C. Simpson, E. F. LoPresti, and R. A. Cooper, “How many people would benefit from a smart wheelchair?” J. Rehab. Res. Develop., vol. 45, no. 1, pp. 53–71, 2008.
[4] T. Carlson and Y. Demiris, “Collaborative control for a robotic wheelchair: Evaluation of performance, attention, and workload,” IEEE Trans. Syst. Man, Cybern. B, vol. 42, no. 3, pp. 876–888, 2012. [5] B. Rebsamen, C. Guan, H. Zhang, C. Wang, C. Teo, M. Ang, and E. Burdet, “A brain controlled wheelchair to navigate in familiar environments,” IEEE Trans. Neural Syst. Rehab. Eng., vol. 18, no. 6, pp. 590–598, Dec. 2010. [6] I. Iturrate, J. Antelis, A. KÜbler, and J. Minguez, “A noninvasive brainactuated wheelchair based on a P300 neurophysiological protocol and automated navigation,” IEEE Trans. Robot., vol. 25, no. 3, pp. 614–627, June 2009. [7] J. del R. Millán, F. Galán, D. Vanhooydonck, E. Lew, J. Philips, and M. Nuttin, “Asynchronous non-invasive brain-actuated control of an intelligent wheelchair,” in Proc. 31st Annu. Int. Conf. IEEE Eng. Med. Biol. Soc., 2009, pp. 3361–3364. [8] J. del R. Millán, F. Renkens, J. Mouriño, and W. Gerstner, “Noninvasive brain-actuated control of a mobile robot by human EEG,” IEEE Trans. Biomed. Eng., vol. 51, no. 6, pp. 1026–1033, 2004. [9] F. Galán, P. W. Ferrez, F. Oliva, J. Guárdia, and J. del R. Millán, “Feature extraction for multi-class BCI using canonical variates analysis,” in Proc. IEEE Int. Symp. Intelligent Signal Processing, 2007, pp. 1–6. [10] J. del R. Millán, P. W. Ferrez, F. Galán, E. Lew, and R. Chavarriaga, “Noninvasive brain-machine interaction,” Int. J. Pattern Recognit. Artif. Intell., vol. 22, no. 5, pp. 959–972, 2008. [11] S. Perdikis, H. Bayati, R. Leeb, and J. del R. Millán, “Evidence accumulation in asynchronous BCI,” Int. J. Bioelectromagnetism, vol. 13, no. 3, pp. 131–132, 2011. [12] G. Lucas. (2013, Jan. 16th). A tutorial and elementary trajectory model for the differential steering system of robot wheel actuators, The Rossum Project. [Online]. Avaliable: http://rossum.sourceforge.net/papers/DiffSteer/ [13] E. Fazl-Ersi and J. Tsotsos, “Region classification for robust floor detection in indoor environments,” in Image Analysis Recognit., M. Kamel and A. Campilho, Eds. Berlin, Germany: Springer-Verlag, 2009, pp. 717–726. [14] T. Dutta and G. Fernie, “Utilization of ultrasound sensors for anti-collision systems of powered wheelchairs,” IEEE Trans. Neural Syst. Rehab. Eng., vol. 13, no. 1, pp. 24–32, 2005. [15] S. Beucher, “The watershed transformation applied to image segmentation,” in Proc. 1Oth Pfefferkorn Conf. Signal Image Processing Microscopy Microanalysis, 16-19 Sept. 1991, Scanning Microscopy Int., Cambridge, U.K., 1992, vol. 6, pp. 299–314. [16] J. Borenstein and Y. Koren, “The vector field histogram—fast obstacle avoidance for mobile robots,” IEEE Trans. Robot. Autom., vol. 7, no. 3, pp. 278–288, 1991. [17] G. Schöner, M. Dose, and C. Engels, “Dynamics of behavior: Theory and applications for autonomous robot architectures,” Robot. Auton. Syst., vol. 16, no. 1, pp. 213–245, 1995. [18] L. Tonin, T. Carlson, R. Leeb, and J. del R. Millán, “Brain-controlled telepresence robot by motor-disabled people,” in Proc. IEEE Annu. Int. Conf. Engineering Medicine Biology Society EMBC, 2011, pp. 4227–4230. [19] S. ten Hagen and B. Krose, “Trajectory reconstruction for self-localization and map building,” in Proc. IEEE Int. Conf. Robotics Automation, 2002, vol. 2, pp. 1796–1801. [20] D. Norman, The design of everyday things, New York: Doubleday Business, 2002.
Tom Carlson, Ecole Polytechnique Fédérale de Lausanne, Lausanne, Switzerland. E-mail: tom.carlson@epfl.ch. José del R. Millán, Ecole Polytechnique Fédérale de Lausanne, Lausanne, Switzerland. E-mail: jose.millan@epfl.ch.
