- Open Access
(C)overt attention and visual speller design in an ERP-based brain-computer interface
© Treder and Blankertz; licensee BioMed Central Ltd. 2010
- Received: 12 February 2010
- Accepted: 28 May 2010
- Published: 28 May 2010
In a visual oddball paradigm, attention to an event usually modulates the event-related potential (ERP). An ERP-based brain-computer interface (BCI) exploits this neural mechanism for communication. Hitherto, it was unclear to what extent the accuracy of such a BCI requires eye movements (overt attention) or whether it is also feasible for targets in the visual periphery (covert attention). Also unclear was how the visual design of the BCI can be improved to meet peculiarities of peripheral vision such as low spatial acuity and crowding.
Healthy participants (N = 13) performed a copy-spelling task wherein they had to count target intensifications. EEG and eye movements were recorded concurrently. First, (c)overt attention was investigated by way of a target fixation condition and a central fixation condition. In the latter, participants had to fixate a dot in the center of the screen and allocate their attention to a target in the visual periphery. Second, the effect of visual speller layout was investigated by comparing the symbol Matrix to an ERP-based Hex-o-Spell, a two-levels speller consisting of six discs arranged on an invisible hexagon.
We assessed counting errors, ERP amplitudes, and offline classification performance. There is an advantage (i.e., less errors, larger ERP amplitude modulation, better classification) of overt attention over covert attention, and there is also an advantage of the Hex-o-Spell over the Matrix. Using overt attention, P1, N1, P2, N2, and P3 components are enhanced by attention. Using covert attention, only N2 and P3 are enhanced for both spellers, and N1 and P2 are modulated when using the Hex-o-Spell but not when using the Matrix. Consequently, classifiers rely mainly on early evoked potentials in overt attention and on later cognitive components in covert attention.
Both overt and covert attention can be used to drive an ERP-based BCI, but performance is markedly lower for covert attention. The Hex-o-Spell outperforms the Matrix, especially when eye movements are not permitted, illustrating that performance can be increased if one accounts for peculiarities of peripheral vision.
- Stimulus Onset Asynchrony
- Motor Imagery
- Amyotrophic Lateral Sclerosis Patient
- Covert Attention
- Visual Periphery
A brain-computer interface (BCI) based on event-related potentials (ERPs) exploits the fact that the neural processing of a stimulus can be modulated by attention. In particular, attention to an event can enhance the positive and negative peaks of the ERP time-locked to this event. ERP-based BCIs attempt to detect these modulations to infer the stimulus that the user intended to choose. Often, the BCI is implemented in an oddball paradigm, wherein rare target events are interspersed with frequent nontarget events. The first such device was introduced by Farwell and Donchin . The authors coined the name P300-speller to refer to the fact that classification was mainly based on the P300 component, a large positivity occurring at 300-500 ms post-stimulus upon rare events. A number of variations of the original speller have been developed, and it has also been adapted to non-visual modalities by using auditory [2–6] and tactile  stimulation.
The classical Farwell and Donchin speller consists of a 6 × 6 symbol matrix wherein symbols are arranged within rows and columns. We will refer to this kind of speller as the Matrix. Throughout the course of a trial, the rows and columns are intensified (flashed) one after the other in a random order. Since a given target symbol has a chance of 1/6 of being intensified, it constitutes a rare event or oddball. In Farwell and Donchin's study, healthy participants were able to communicate about 12 bits or an equivalent of 2.3 symbols/min. In the past decades, classification techniques improved [8–11] and practical communication rates including feedback of 5.82 symbols/min have been reported , but this information throughput is still not competitive when compared to conventional communication means such as speech, typing, or handwriting. In other words, current ERP-based BCIs do not seem to be viable tools for healthy users. Therefore, most BCIs are tailored for use by patients deprived of other means of communication, such as amyotrophic lateral sclerosis (ALS) patients, who suffer from a neurodegenerative disease characterized by a progressive loss of motor function [13–15]. However, most successful implementations, such as the Matrix speller, use a spatial layout wherein the to-be-chosen symbols are placed at different spatial locations. Hitherto, it was unclear whether or not these spellers rely on eye movements. If they do then devices that measure eye movements directly (such as eyetrackers) might outperform visual BCIs. In fact, there is a body of evidence corroborating the efficacy of dwell-time based gaze interaction in a clinical context [16–19]. For healthy users, information throughput of about 10 words per minute has been reported .
The present study addresses the question whether an ERP-based BCI is ultimately dependent on eye movements (i.e., overt attention), or whether it can also detect attention deployed in the visual periphery (i.e., covert attention). This is a key issue because, first, the focus of covert attention cannot be inferred from eye movement data. Second, successful communication using ERP-based visual spellers has been demonstrated in ALS patients [21, 22], but in progressed stages of the disease, oculomotor control can deteriorate. Dysfunctions in smooth pursuit, slowing of fixations, nystagmus, and abnormalities in Bell's phenomenon have been observed [23–25], as well as corresponding neurophysiological damage to oculomotor nuclei . For patients suffering from these symptoms, communication using eyetrackers might collapse. Using covert attention in the visual periphery, however, is complicated by the fact that peripheral vision is subject to some peculiarities that should be taken into account in the visual design of the BCI. One of these peculiarities is the decline of spatial acuity with increasing visual eccentricity. Human detail vision is limited to the fovea, the small central portion of the visual field subtending about 2° of visual angle. Beyond the fovea, spatial acuity drops rapidly as a function of eccentricity. For a part, this is due to the anisotropic distribution of photoreceptors in the retina. For instance, cone photoreceptors subserving photopic (day) vision are densely packed in the fovea where spatial acuity is high. With increasing eccentricity, rods subserving scotopic (night) vision become abundant, with less and less cones interspersed. Another factor adding to the limited peripheral acuity is that the responses of rods are usually pooled to increase sensitivity to light at the expense of spatial resolution; while the 1:1 correspondence in the fovea allows for maximal spatial resolution, the ratio of photoreceptors to ganglion cells can be as low as 130:1 in the periphery . This implies that users might not be able to resolve and identify targets if they are located in the far periphery. Another peculiarity of peripheral vision is the so-called crowding effect [28–31]. Crowding refers to the phenomenon that the identification of objects in the visual periphery-is hampered if they are surrounded by similar objects. It has been suggested that crowding is caused by an inaccuracy in deploying spatial attention in the periphery, resulting in misbinding of features belonging to different objects .
Due to its visual design, the classical Matrix speller is inevitably affected by these peculiarities. The Matrix contains many symbols which are hard to allocate attention to in the periphery. One could up-size the symbols with increasing eccentricity to compensate for the decline of visual acuity, but this would then increase the crowding effect because the symbols get crammed together. The only way to scale-up element size and to counteract crowding at the same time is to decrease the number of symbols. Unfortunately, in the classical paradigm, this would leave the user with less degrees of freedom in communication. In contrast, both premises can be met with the Hex-o-Spell [33–35]. By means of a two-levels selection process, the Hex-o-Spell preserves a large vocabulary even though the number of symbols in the display is small. The original Hex-o-Spell consists of a central circle surrounded by six hexagons. Each hexagon represents a quintet of alphanumerical symbols. By means of motor imagery, the user rotates a central arrow and then chooses one of the hexagons. Upon choice, the symbols in the hexagon are expanded into the other hexagons and the user again uses mental imagery to choose the desired symbol. In this study, we adapt this two-level BCI design to an oddball paradigm and we compare its efficacy to the efficacy of the Matrix. Hex-o-Spell has some visually desirable properties. First, at each level, it displays only a few large symbols. This can prevent the detrimental effects of both declining spatial acuity and crowding. Second, the arrangement of hexagons is optimal with respect to crowding. Crowding is most serious if elements are placed on a line extending radially from the fixation point; it is minimal in configurations wherein elements are placed in a circular fashion, such as the hexagons of the Hex-o-Spell.
To investigate both modes of spatial attention (overt, covert) and both kinds of spellers (Hex-o-Spell, Matrix), we used a 2 × 2 within-subjects design. As benchmarks for the efficacy of each speller-attention pairing, we measured counting accuracy, ERPs, and classification performance in a copy-spelling task. In the ERP analysis, we investigated a number of different evoked and event-related components. In particular, in addition to the P3 component, we considered P1, N1, P2, and N2. P1, N1 and P2 are associated with automated stimulus processing that is affected by early attentional processes . N2 is assumed to be related to the processing of deviant stimuli . Despite the rather step-motherly treatment of these early components in earlier articles on ERP-based BCIs, there have been consistent reports that they are modulated in visual oddball tasks, first shown by  and corroborated by later studies [39, 40]. Following the ERP analysis, we will present the results of offline classification using linear discriminant analysis (LDA) with shrinkage of the covariance matrix. Preliminary results of this study have been previously presented at a BCI workshop . At a recent workshop, a similar study has been reported for the Matrix speller .
Thirteen participants (9 males and 4 females), aged 21-43 years (μ = 29.5) and naïve with respect to ERP-based BCIs, took part in the experiment. All had normal or corrected-to-normal vision and they received money for their participation. All participants gave written consent and the study was performed in accordance with the Declaration of Helsinki.
In the Matrix, symbols were arranged on a grid with a size of 500 × 500 px2 (13.96° × 13.92°). In order to match the total number of symbols in the Hex-o-Spell (30), the speller comprised 6 rows and 5 columns (Figure 2a). Symbol height was 40 px (1.12°), or 65 px (1.82°) when intensified (an increase of 62.5%) with width depending on the particular symbol. Intensification was row-wise or column-wise. The Hex-o-Spell features selection as a two-stage process, wherein first a symbol group is selected (Figure 2b). Upon choice of a symbol group, the speller descends to the second level (Figure 2c), where the individual target symbol can be selected (Figure 2d). Note that, since there are 6 discs but only 5 symbols, one disc is empty; the purpose of the empty disc is to enable users to return to the top-level in case the wrong group has been selected. Discs had a size of 148 × 148 px2 (4.15° × 4.14° of visual angle), or 200 × 200 px2 (5.61° × 5.59°) when intensified (an increase of 35.1%). Unlike in the Matrix, discs in the Hex-o-Spell were intensified one by one. The discs were spatially arranged at the corners of an (invisible) hexagon with a diameter of 440 px (about 12.28°).
Participants were seated in a comfortable chair at a distance of about 60 cm from the screen, which is the optimal operational range for the eyetracker. Instruction was given both in written and verbal form. Participants were instructed to relax their muscles and to try to avoid eye movements during the course of a trial. After EEG preparation and calibration of the eyetracker, they completed a practice phase for the Matrix and for the Hex-o-Spell in the overt attention condition. After this, the experiment commenced and EEG was recorded for offline analysis. Participants engaged in a copy-spelling task, whereby they had to copy 5-6 letter words. There was a set of nine German words, chosen such that each letter in the English alphabet was covered 1-3 times. Each word was repeated 4 times (once for each subcondition). When a new word was introduced, it was shown on the screen prior to the start of the trial. Subsequently, a trial started with a 4-seconds auditory countdown, during which participants had time to identify the location of the target. The current word was always shown in a box above the speller with the current letter being highlighted (Figure 2). After the countdown, the intensification phase started, lasting for about 30s. The task of the participant was to silently count the number of intensifications of the target symbol. For the Matrix, 10 sequences were presented, whereby every row and every column was intensified exactly once in a single sequence (6 rows + 5 columns = 11 intensifications per sequence). The order of intensifications was pseudo-randomized as there had to be at least two intermittent intensifications before a particular intensification was repeated. Furthermore, to obtain meaningful behavioral data, some variation to the number of intensifications of a target was introduced. The sequences had a prequel and a sequel (both not used in the analysis) of 11 intensifications each, whereby intensifications were allowed to repeat. This added up to a total of 132 intensifications per trial. For the Hex-o-Spell, the sequences were evenly spread across the two hex levels. At the first level (group level), a symbol group had to be selected. Analogous to the Matrix, there were 10 sequences of 6 intensifications each, with prequels and sequels containing repetitions. At the second level (symbol level), the target symbol had to be selected, again with 10 sequences, and again preceded and followed by sequences with repetitions. Since, in the second level, there was also an empty disc, there was a total of 144 intensifications per trial. The number of target intensifications, however, was the same for both spellers.
For both spellers, the duration of a single intensification was 100 ms (or an equivalent of 6 frames). Stimulus onset asynchrony (SOA), that is, the time between the onsets of subsequent intensifications, amounted to 166 ms (10 frames). At the end of each trial, participants entered their count via the computer keyboard. The next trial commenced when participants pressed the enter key. In the overt attention condition, they had to fixate the target symbol or disc. In the covert attention condition, a central fixation dot was shown throughout the trial and participants had to strictly fixate the dot while counting the intensifications of the target. To assure proper fixation, eye movements were monitored online. If a fixation of a location other than the designated location (i.e., the target symbol or the fixation dot) was detected, a warning tone was presented and the trial was aborted; upon a button press, the trial was started again, using new intensification sequences.
The experiment was split into blocks of 3 words each, whereby breaks were given between the blocks. The order of the blocks was randomized, albeit with the constraint that each speller type was introduced first in the overt attention condition, not in the more difficult covert attention condition. The total number of blocks amounted to 2 (spellers) × 2 (attention) × 3 (unique blocks) = 12. The two spellers were implemented in the open-source BCI framework Pyff  and remote-controlled via Matlab. Both spellers are available on the Pyff website .
Event-related potentials (ERPs)
Analysis of N1 amplitudes revealed larger amplitudes for targets than for nontargets (Status, F = 178.29, p < .001) and larger amplitudes for overt attention than for covert attention (Attention, F = 87.16, p < .001). Main effects of Speller (p = .403), Electrode (p = .698), and the other interactions were not significant. Like for P1, N1 amplitude modulation was different for the two modes of attention (Attention × Status, F = 119.07, p < .001); there was significant modulation for overt attention for both spellers (Status, F = 174.35, p < .001), but for covert attention modulation was significant for the Hex-o-Spell (Status, F = 9.14, p < .01) but not for the Matrix (p = .163).
P2 amplitudes were larger for targets than for nontargets (Status, F = 120.23, p < .001), larger for the Hex-o-Spell than for the Matrix (Speller, F = 10.34, p < .01), and larger for overt attention than for covert attention (Attention, F = 38.79, p < .001). P2 amplitude modulation was different for the two modes of attention (Attention × Status, F = 77.47, p < .001), and it was also different for the two different spellers (Speller × Status, F = 5.15, p < .05). In particular, modulation was stronger for overt attention than for covert attention, and stronger for Hex-o-Spell than for the Matrix. In the covert attention condition, there was no significant modulation of P2 amplitude for the Matrix (p = .354), but it was significant for the Hex-o-Spell (Status, F = 11.88, p < .01). The effect of Electrode (p = .18) and the other interactions were not significant.
N2 amplitudes were larger for targets than for nontargets (Status, F = 36.25, p < .001), and larger for overt attention than for covert attention (Attention, F = 15.75, p < .001). N2 amplitude modulation was different for the two modes of attention (Attention × Status, F = 38.71, p < .001), and these differences differed for the two spellers (Attention × Speller × Status, F = 4.25, p < .05). Amplitude modulation was higher for overt attention than for covert attention. In the covert attention condition, it was still significant for both the Matrix (Status, F = 4.53, p < .05) and the Hex-o-Spell (Status, F = 10.59, p < .01). However, for Hex-o-Spell, it was in the opposite direction (i.e., smaller amplitudes for targets than for nontargets). Although there was no significant effect of Electrode (p = .486), there were significant interactions, namely Attention × Electrode (F = 3.07, p < .05), and there was a significant three-way interaction Attention × Target × Electrode, F = 4.25, p < .05). The effect of Speller (p = .958) and the other interactions were not significant.
P3 amplitudes were larger for targets than for nontargets (Status, F = 129.52, p < .001). Amplitude modulation was higher for overt attention than for covert attention (Attention × Status, F = 7.75, p <.01). Main effects of Attention (p = .708), Speller (p = .109), Electrode (p = .491), and the other interactions were not significant.
In addition to these analyses, we also investigated the effects of attention and speller on difference amplitudes (i.e., ERP amplitude to target intensification minus ERP amplitude to nontarget intensifications). Because, as Figure 6 shows, not only target but also nontarget amplitudes were usually different across conditions, difference amplitudes give a better picture of the magnitude of amplitude modulation. For all ERP components under investigation, that is, P1, N1, P2, N2, and P3, we found that amplitudes are modulated more under overt attention than under covert attention (Attention, F-values 90.22, 160.81, 121.67, 43.91, and 13.65, respectively, with all p-values < .001). For positive components P1 and P2, we found overall stronger modulations for the Hex-o-Spell than for the Matrix (Speller, F-values are 19.64 and 8.09, respectively, p < .01). For the N2 and P3 components, amplitude modulations in the overt attention condition were not significantly different for the two spellers (p-values .6438 and .516, respectively), but they were stronger for the Hex-o-Spell than for the Matrix in the covert attention condition (F = 28.67, p < .001, and F = 5.23, p < .05, respectively). Regarding the N1 component, there was no effect of Speller on difference amplitude (p = .1).
The ERP analysis in the previous section showed that there is a number of ERP components that is modulated by attention. A BCI operates by detecting these modulations and inferring whether a target or a nontarget was intensified. For offline classification, we used linear discriminant analysis (LDA) with shrinkage of the covariance matrix. A recent article on ERP analysis showed that shrinkage is a potent tool to counteract the bias encountered in settings with high-dimensional feature vectors and comparably small training sets, and it was shown to be at least as good as step-wise LDA (Blankertz B, Lemm S, Treder MS, Haufe S, Müller KR: Single-trial analysis and classification of ERP components - a tutorial, submitted). As said in the Procedure, there were three blocks of trials for each attention-speller pairing. For classification, the first block was taken as training set and the second and third blocks were taken as test set. EEG was downsampled to 100 Hz and baseline corrected using a 170 ms pre-stimulus interval. In contrast to the ERP analysis, all epochs were used for classification. The feature vector consisted of 55 spatial features × 7 temporal features = 385 spatio-temporal features. Temporal features were automatically extracted using a heuristic searching for peaks in the point-biserial correlation coefficient between targets and nontargets. A single binary (target versus nontarget) classifier was trained. To choose a symbol in the Matrix speller, the row (out of 6 rows) and the column (out of 5 columns) with maximum classifier outputs were selected, and the target symbol was given by their intersection . For the Hex-o-Spell, the selection process was similar. At the group level, the group (out of 6 groups) with the highest classifier output was chosen, and at the symbol level, the symbol (out of 6 symbols) with the highest classifier output was chosen.
Behavioral, neurophysiological, and classification indices unanimously attest an advantage (i.e., less errors, larger ERP amplitude, better classification) of overt attention over covert attention, and an advantage of Hex-o-Spell over Matrix. Using overt attention, spelling success is mainly based on visually evoked potentials (VEPs) measured at occipital and parieto-occipital sites. This confirms earlier conjectures [38, 49] and it is also in accordance with  who showed that classifying on posterior electrodes in addition to the classical P3 sites improves classification performance. For a part, the comparably limited amount of information carried in the P3 component is due to the fast pace of the BCI used in the present study. In BCIs with longer SOAs, P3 components tend to be much more pronounced both in terms of amplitude and temporal extent [10, 39]. In covert attention mode, classification is mainly based on the P3 component, but there is also a clear modulation of P2 amplitude for the Hex-o-Spell. In the face of these results, the term P300-BCI, which is often used in the literature, seems inadequate if not misleading. We advocate the use of the term ERP-based BCI to put emphasis on the fact that there is a multitude of ERP components that is affected by attention and that is exploited by classifiers. The rest of the General discussion addresses the role of (c)overt attention and discusses aspects that might be important in the design of visual spellers.
Overt versus covert attention
If visual ERP-based BCIs are to have more than a shadowy existence in clinical practice, they have to form a viable alternative to eyetrackers. Currently, the detection of eye movements is quicker, easier, and more accurate than the detection of ERP modulations, and there are commercial plug-and-play eyetrackers tailored for users with motor deficiencies. Using eyetrackers, a spelling rate of 10 words per minute can be obtained with unimpaired eye movements. For ERP-based spellers, a recent study reported a spelling rate of 1.2 symbols per minute using a 6 × 6 Matrix speller with ALS patients , which is markedly lower. As a side note, notice that eyetrackers and BCIs do not need to be mutually exclusive systems in general. For instance,  demonstrated a hybrid system based on both eyetracking and BCI wherein targets were selected by eye gaze and an action was triggered via motor imagery.
In patients with impaired control of eye movements, however, reliable communication via eyetrackers can break down. But, since the neural systems underlying overt attention shifts and covert attention shifts are not identical , such patients might still be able to use covert attention. Our study shows that ERP-based visual spellers can be driven in both modes of attention, so they might replace or complement eyetrackers in these situations. Unfortunately, the accuracy obtained for covert attention in the present offline analysis is too low to be a viable means of communication. This implies that before ERP-based spellers are suitable for clinical practice, both classification  and visual design need to improve markedly. The latter point is addressed next.
Visual speller design
Using covert attention, classification performance is low for both kind of spellers. For the Matrix, peak performance is about 40%. For the Hex-o-Spell, it is about 60%, which amounts to a relative increase of 50%. This illustrates that taking into account the peculiarities of peripheral vision can substantially boost performance. There is a number of aspects related to visual speller design that can be differentiated, namely the spatial arrangement of the elements on the screen, the visual properties of the elements, the intensification type, and the intensification sequence. We will address these points one by one.
As explicated in the Introduction, the deployment of spatial attention in the visual periphery is complicated by effects of crowding and decline of spatial acuity. Hex-o-Spell is less affected by these effects than the Matrix, because it features a small number of large elements instead of a large number of small elements, and because elements are arranged in a circular fashion, which has been shown to reduce crowding. In addition, the circular arrangement of elements around a central point in the Hex-o-Spell allows for a straightforward transition from a screen-centered representation to a body-centered representation in other modalities. For instance, [5, 6] recently presented a spatial auditory paradigm wherein the user was located in the center of a ring of six loudspeakers. Users had to focus their auditory attention on one of the loudspeakers to choose a symbol. Due to the conceptual similarities between the visual Hex-o-Spell and the spatial auditory paradigm, users could probably switch more easily between the paradigms than when they had started with the Matrix.
With respect to the design of the individual elements, size matters. Not only can large elements be identified more easily in the periphery, there is also evidence that P3 amplitudes are positively correlated with stimulus intensity , which is in line with the fact that we found larger ERP amplitudes for the Hex-o-Spell than for the Matrix. Larger amplitudes might be of particular importance for clinical application, because an attenuation of ERP amplitudes is generally observed in ALS patients [14, 55, 56]. With respect to intensification type, there is a number of visual feature dimensions along which an intensification can be defined, for instance, luminance, size, form, orientation, color, and motion. In this study, we used size enhancement because it allows for maximum contrast both when the symbols are enhanced and when they are not. This is especially important when objects are presented in the visual periphery. Successful applications of other types of intensification have been reported in the literature, such as orientation, motion onset and illusory triangles [57–59]. Flipping the orientation of a rectangle located behind each symbol in a Matrix speller produced better classification results than the classical luminance intensification . Furthermore, when intensifications were defined by motion onset, the positive and negative components in the 160-300 ms interval were discriminative for the task . Interestingly, this interval was also the most informative interval in our overt attention condition. Actually, in contrast to luminance enhancement, both orientation flips and size enhancement imply that the contours of the symbols are displaced, which can give rise to a conscious percept of apparent motion. Stimulus intensity could be increased by using multiple features simultaneously for intensification; conversely, using different intensifications for different elements might increase their discriminability if a classifier is trained on each element separately.
With respect to intensification sequence, target-to-target distance (i.e., the number of nontargets between successive intensifications of the target) has been shown to affect the amplitude of the P3 component, with smaller amplitudes for more frequent targets [58, 60]. This is in accordance with the fact that larger SOAs yield larger ERP amplitudes, because with larger SOAs the temporal separation between successive target intensifications increases. Hence, it is desirable to use target sequences wherein targets are not repeatedly intensified without nontargets in between. In the Matrix, the problem is inevitable because if a row intensification is followed by a column intensification (or vice versa), it is possible that the target lies at the intersection of these two intensifications. This is not the case for the Hex-o-Spell, where elements are individually intensified.
In the literature, there have also been other approaches to make visual spellers less dependent on eye movements  presented a speller whereby four different words were presented at the same spatial location in an alternating fashion. Both healthy users and ALS patients were able to communicate, which means that spatial attention is not necessary for visual spelling. The disadvantage of this paradigm, however, is that the sequential presentation of targets requires very long trials to maintain a large vocabulary. An alternative to sequential presentation of symbols might be simultaneous presentation at the same location. For instance, for the SSVEP paradigm, it was demonstrated that users can reliably choose between two superimposed dot patterns rotating in opposite directions  or between two superimposed gratings . Again, however, the vocabulary is necessarily small with overlapping targets, which illustrates how difficult it is to reconcile independence from eye movements with high information throughput.
A few limitations warrant consideration. First, whether or not ALS patients with impaired eye movements can reliably employ covert spatial attention for BCI control has to be verified in clinical studies. Second, classification performance of the Matrix is compared to the performance of the Hex-o-Spell on basis of offline data. An online study would give a more accurate estimate of the performance of these two spellers.
For patients with intact control of eye movements, eyetrackers are the device-of-choice, at least if information throughput is the evaluation criterion. The target group of ERP-based spellers therefore is patients with impaired eye movements. Our study shows that the performance of visual spellers deteriorates if one switches from overt to covert attention, but it also shows that performance can be increased using innovative spellers that take into account the peculiarities of peripheral vision.
We would like to thank Michael Tangermann and Thorsten Dickhaus for helpful comments on an earlier version of the manuscript.
- Farwell L, Donchin E: Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. Electroencephalogr Clin Neurophysiol. 1988, 70: 510-523. 10.1016/0013-4694(88)90149-6.View ArticlePubMedGoogle Scholar
- Furdea A, Halder S, Krusienski DJ, Bross D, Nijboer F, Birbaumer N, Kübler A: An auditory oddball (P300) spelling system for brain-computer interfaces. Psychephysiology. 2009, 46: 617-625. 10.1111/j.1469-8986.2008.00783.x.View ArticleGoogle Scholar
- Kübler A, Furdea A, Halder S, Hammer EM, Nijboer F, Kotchoubey B: A brain-computer interface controlled auditory event-related potential (p300) spelling system for locked-in patients. Annals of the New York Academy of Sciences. 2009, 1157: 90-100. 10.1111/j.1749-6632.2008.04122.x.View ArticlePubMedGoogle Scholar
- Nijboer F, Furdea A, Gunst I, Mellinger J, McFarland D, Birbaumer N, Kübler A: An auditory brain-computer interface (BCI). J Neurosci Methods. 2008, 167: 43-50. 10.1016/j.jneumeth.2007.02.009.View ArticlePubMedGoogle Scholar
- Schreuder M, Tangermann M, Blankertz B: Initial results of a high-speed spatial auditory BCI. Int J Bioelectromagnetism. 2009, 11 (2): 105-109. http://ijbem.k.hosei.ac.jp/volume11/number2/1102008.pdfGoogle Scholar
- Schreuder M, Blankertz B, Tangermann M: A New Auditory Multi-class Brain-Computer Interface Paradigm: Spatial Hearing as an Informative Cue. PLoS ONE. 2010, 5 (4): e9813-10.1371/journal.pone.0009813.PubMed CentralView ArticlePubMedGoogle Scholar
- Brouwer AM, van Erp J: A tactile P300 BCI and the optimal number of tactors: effects of target probability and discriminability. Proceedings of the 4th International BCI Workshop and Training Course. 2008, 280-285.Google Scholar
- Donchin E, Spencer KM, Wijesinghe R: The Mental Prosthesis: Assessing the speed of a P300-Based Brain-Computer Interface. IEEE Trans Rehabil Eng. 2000, 8 (2): 174-179. 10.1109/86.847808.View ArticlePubMedGoogle Scholar
- Krusienski DJ, Sellers EW, Cabestaing F, Bayoudh S, McFarland DJ, Vaughan TM, Wolpaw JR: A comparison of classification techniques for the P300 Speller. J Neural Eng. 2006, 3 (4): 299-305. 10.1088/1741-2560/3/4/007.View ArticlePubMedGoogle Scholar
- Sellers E, Krusienski D, McFarland D, Vaughan T, Wolpaw J: A P300 event-related potential brain-computer interface (BCI): the effects of matrix size and inter stimulus interval on performance. Biol Psychol. 2006, 73: 242-252. 10.1016/j.biopsycho.2006.04.007.View ArticlePubMedGoogle Scholar
- Rakotomamonjy A, Guigue V: BCI competition III: dataset II ensemble of SVMs for BCI P300 speller. IEEE Trans Biomed Eng. 2008, 55: 1147-1154. 10.1109/TBME.2008.915728.View ArticlePubMedGoogle Scholar
- Lenhardt A, Kaper M, Ritter H: An adaptive P300-based online brain-computer interface. IEEE Trans Neural Syst Rehabil Eng. 2008, 16: 121-130. 10.1109/TNSRE.2007.912816.View ArticlePubMedGoogle Scholar
- Abrahams S, Goldstein LH, Kew JJM, Brooks DJ, Lloyd CM: Frontal lobe dysfunction in amyotrophic lateral sclerosis: a PET study. Brain. 1996, 119: 2105-2120. 10.1093/brain/119.6.2105.View ArticlePubMedGoogle Scholar
- Hanagasi HA, Gurvit IH, Ermutlu N, Kaptanoglu G, Karamurseld S, Idrisoglu HA, Emre M, Demiralp T: Cognitive impairment in amyotrophic lateral sclerosis: evidence from neuropsychological investigation and event-related potentials. Cognitive Brain Research. 2002, 14: 234-244. 10.1016/S0926-6410(02)00110-6.View ArticlePubMedGoogle Scholar
- Massman PJ, Sims J, Cooke N, Haverkamp LJ, Appel V: Prevalence and correlates of neuropsychological deficits in amyotrophic lateral sclerosis. Journal of Neurology, Neurosurgery & Psychiatry. 1996, 61: 450-455.View ArticleGoogle Scholar
- Adjouadi M, Sesin A, Ayala M, Cabrerizo M: Remote eye gaze tracking system as a computer interface for persons with severe motor disability. Computers Helping People with Special Needs. 2004, 3118: 761-769.View ArticleGoogle Scholar
- Calvo A, Chiò A, Castellina E, Corno F, Farinetti L, Ghiglione P, Pasian V, Vignola A: Eye tracking impact on quality-of-life of ALS patients. Computers Helping People with Special Needs. 2008, 5105: 70-77. full_text.View ArticleGoogle Scholar
- Itoh K, Aoki H, Hansen JP: A comparative usability study of two Japanese gaze typing systems. Proceedings of the 2006 Symposium on Eye Tracking Research & Applications. 2006, 59-66.View ArticleGoogle Scholar
- Shi F, Gale A, Purtly K: A new gaze-based interface for environmental control. Universal Access in Human-Computer Interaction. Ambient Interaction. 4th International Conference on Universal Access in Human-Computer Interaction. 2007, 996-1005.View ArticleGoogle Scholar
- Majaranta P, MacKenzie S, Aula A, Räihä KJ: Effects of feedback and dwell time on eye typing speed and accuracy. Universal Access in the Information Society. 2006, 5 (2): 199-208. 10.1007/s10209-006-0034-z.View ArticleGoogle Scholar
- Sellers EW, Donchin E: A P300-based brain-computer interface: initial tests by ALSpatients. Clinical Neurophysiology. 2006, 117: 538-548. 10.1016/j.clinph.2005.06.027.View ArticlePubMedGoogle Scholar
- Nijboer F, Sellers EW, Mellinger J, Jordan MA, Matuz T, Furdea A, Halder S, Mochty U, Krusienski DJ, Vaughan TM, Wolpaw JR, Birbaumer N, Kübler A: A P300-based brain-computer interface for people with amyotrophic lateral sclerosis. Clinical Neurophysiology. 2008, 119: 1909-1916. 10.1016/j.clinph.2008.03.034.PubMed CentralView ArticlePubMedGoogle Scholar
- Esteban A, DeAndres C, Gimenetz-Roldan S: Abnormalities of Bell's phenomenon in amyotrophic lateral sclerosis. A clinical and electrophysiological evaluation. Journal of Neurology, Neurosurgery & Psychiatry. 1978, 41: 690-698.View ArticleGoogle Scholar
- Palmowski A, Jost WH, Osterhage J, Prudlo J, Käsmann B, Schimrigk K, Ruprecht KW: Augenbewegungsstörungen bei Amyotropher Lateralsklerose--Bericht über zwei Patienten. Klinische Monatsblätter für Augenheilkunde. 1995, 206: 192-201. 10.1055/s-2008-1035424.View ArticleGoogle Scholar
- Shapley R, Hawken M, Ringach DL: Dynamics of orientation selectivity in the primary visual cortex and the importance of cortical inhibition. Neuron. 2003, 38 (5): 689-699. 10.1016/S0896-6273(03)00332-5.View ArticlePubMedGoogle Scholar
- Okamoto K, Hira S, Amari M, Iizuka T, Watanabe M, Murakami N, Takatama M: Oculomotor nuclear pathology in amyotrophic lateral sclerosis. Acta Neuropathologica. 1992, 85: 458-462.Google Scholar
- Rodieck RW: The first steps in seeing. 1998, Massachusetts: Sinauer AssociatesGoogle Scholar
- Bouma H: Interaction effects in parafoveal letter recognition. Nature. 1970, 226: 177-178. 10.1038/226177a0.View ArticlePubMedGoogle Scholar
- Feng C, Jiang Y, He S: Horizontal and vertical asymmetry in visual spatial crowding effects. Journal of Vision. 2007, 7: 1-10. 10.1167/7.2.13.View ArticlePubMedGoogle Scholar
- Toet A, Levi DM: The two-dimensional shape of spatial interaction zones in the parafovea. Vision Research. 1992, 32: 1349-1357. 10.1016/0042-6989(92)90227-A.View ArticlePubMedGoogle Scholar
- van den Berg R, Roerdink JBTM, Cornelissen FW: On the generality of crowding: Visual crowding in size, saturation, and hue compared to orientation. Journal of Vision. 2007, 7: 1-11. 10.1167/7.2.14.View ArticleGoogle Scholar
- Strasburger H: Unfocussed spatial attention underlies the crowding effect in indirect form vision. Journal of Vision. 2005, 5: 1024-1037. 10.1167/5.11.8.View ArticlePubMedGoogle Scholar
- Blankertz B, Krauledat M, Dornhege G, Williamson J, Murray-Smith R, Müller KR: A Note on Brain Actuated Spelling with the Berlin Brain-Computer Interface. Universal Access in HCI, Part II, HCII 2007, Volume 4555 of LNCS. Edited by: Stephanidis C. 2007, Berlin Heidelberg: Springer, 759-768.Google Scholar
- Müller KR, Blankertz B: Toward noninvasive Brain-Computer Interfaces. IEEE Signal Process Mag. 2006, 23 (5): 125-128. 10.1109/MSP.2006.1708426.View ArticleGoogle Scholar
- Williamson J, Murray-Smith R, Blankertz B, Krauledat M, Müller KR: Designing for uncertain, asymmetric control: Interaction design for brain-computer interfaces. International Journal of Human-Computer Studies. 2009, 67 (10): 827-841. 10.1016/j.ijhcs.2009.05.009.View ArticleGoogle Scholar
- Näätänen R, Picton T: The N1 wave of the human electric and magnetic response to sound: a review and an analysis of the component structure. Psychophysiology. 1987, 24 (4): 375-425. 10.1111/j.1469-8986.1987.tb00311.x.View ArticlePubMedGoogle Scholar
- Näätänen R, Gaillard AWK: Tutorials in event-related potential research: Endogenous components. Amsterdam: North Holland 1983 chap. The orienting reflex and the N2 deflection of the ERP, 119-141.Google Scholar
- Blankertz B, Curio G: BCI Competition 2003 Results: Description of one of the winning algorithms on data set IIb (web document). 2003, [The algorithm is using classification on 'spatio-temporal features from visual cortex']., http://www.bbci.de/competition/ii/results/blankertz_iib_desc.pdfGoogle Scholar
- Allison BZ, Pineda JA: Effects of SOA and flash pattern manipulations on ERPs, performance, and preference: implications for a BCI system. International Journal of Psychophysiology. 2006, 59 (2): 127-140. 10.1016/j.ijpsycho.2005.02.007.View ArticlePubMedGoogle Scholar
- Müller MM, Hillyard S: Concurrent recording of steady-state and transient event-related potentials as indices of visual-spatial selective attention. Clin Neurophysiol. 2000, 111: 1544-1552. 10.1016/S1388-2457(00)00371-0.View ArticlePubMedGoogle Scholar
- Treder M, Venthur B, Blankertz B: (C)overt Attention and P300-Speller Design. Poster at the BBCI Workshop 'Advances in Neurotechnology', Berlin. 2009Google Scholar
- Brunner P, Joshi S, Briskin S, Wolpaw JR, Bischof H, Schalk G: Does the P300 speller depend on eye-gaze?. Poster at the TOBI Workshop 'Integrating brain-computer interfaces with conventional assistive technology', Graz. 2010Google Scholar
- Troxler IPV: Über das Verschwinden gegebener Gegenstände innerhalb unseres Gesichtskreises. Ophthalmologische Bibliothek. 1804, 2: 1-53.Google Scholar
- Kanai R, Kamitani Y: Time-locked perceptual fading induced by visual transients. Journal of Cognitive Neuroscience. 2003, 15: 664-672. 10.1162/jocn.2003.15.5.664.View ArticlePubMedGoogle Scholar
- Gibert G, Attina V, Mattout J, Maby E, Bertrand O: Size enhancement coupled with intensification of symbols improves speller accuracy. Proceedings of the 4th International BCI Workshop and Training Course. 2008, 250-255.Google Scholar
- Martens SM, Hill NJ, Farquhar J, Schölkopf B: Overlap and refractory effects in a brain-computer interface speller based on the visual P300 event-related potential. J Neural Eng. 2009, 6: 026003-10.1088/1741-2560/6/2/026003.View ArticlePubMedGoogle Scholar
- Venthur B, Blankertz B: A Platform-Independent Open-Source Feedback Framework for BCI Systems. Proceedings of the 4th International Brain-Computer Interface Workshop and Training Course 2008. 2008, Verlag der Technischen Universität Graz, 385-389.Google Scholar
- Pyff - a Pythonic framework for BCI systems. http://bbci.de/pyff
- Kaper M, Meinicke P, Grossekathoefer U, Lingner T, Ritter H: BCI Competition 2003-Data set IIb: support vector machines for the P300 speller paradigm. Trans Biomed Eng. 2004, 51 (6): 1073-1076. 10.1109/TBME.2004.826698.View ArticleGoogle Scholar
- Krusienski D, Sellers E, McFarland D, Vaughan T, Wolpaw J: Toward enhanced P300 speller performance. J Neurosci Methods. 2008, 167: 15-21. 10.1016/j.jneumeth.2007.07.017.PubMed CentralView ArticlePubMedGoogle Scholar
- Popescu F, Fazli S, Badower Y, Blankertz B, Müller KR: Single Trial Classification of Motor Imagination Using 6 Dry EEG Electrodes. PLoS ONE. 2007, 2 (7): 10.1371/journal.pone.0000637.Google Scholar
- Posner MI: Orienting of attention. Quarterly Journal of Experimental Psychology. 1980, 32: 2-25.View ArticleGoogle Scholar
- Martens SMM, Leiva JM: A generative model approach for decoding in the visual event-related potential-based brain-computer interface speller. J Neural Eng. 2010, 7 (2): 26003-10.1088/1741-2560/7/2/026003.View ArticlePubMedGoogle Scholar
- Covington JW, Polich J: P300, stimulus intensity, and modality. Electroencephalography and Clinical Neurophysiology. 1996, 100: 579-584. 10.1016/S0168-5597(96)96013-X.View ArticlePubMedGoogle Scholar
- Münte TF, Tröger MC, Nusser I, Wieringa BM, Johannes S, Matzke M, Dengler R: Alteration of early components of the visual evoked potential in amyotrophic lateral sclerosis. Journal of Neurology. 1998, 245: 206-210. 10.1007/s004150050206.View ArticlePubMedGoogle Scholar
- Vieregge P, Wauschkuhn B, Heberlein I, Hagenah J, Verleger R: Selective attention is impaired in amyotrophic lateral sclerosis--a study of event-related EEG potentials. Cognitive Brain Research. 1999, 8: 27-35. 10.1016/S0926-6410(99)00004-X.View ArticlePubMedGoogle Scholar
- Guo F, Hong B, Gao X, Gao S: A brain computer interface based on motion-onset VEPs. Conference Proceedings of the IEEE Engineering in Medicine and Biology Society. 2008, 2008: 4478-4481.Google Scholar
- Hill J, Farquhar J, Martens S, Bießmann F, Schölkopf B: Effects of Stimulus Type and of Error-Correcting Code Design on BCI Speller Performance. Advances in Neural Information Processing Systems. 2009, 21:Google Scholar
- Macenka T, Braunstein V, Kober S, Neuper C: The Kanizsa P300-speller: a new way to spell. Poster at the TOBI Workshop 'Integrating brain-computer interfaces with conventional assistive technology', Graz. 2010Google Scholar
- Polich J, Ellerson PC, Cohen J: P300, stimulus intensity, modality, and probability. Int J Psychophysiol. 1996, 23: 55-62. 10.1016/0167-8760(96)00028-1.View ArticlePubMedGoogle Scholar
- Zhang D, Maye A, Gao X, Hong B, Engel AK, Gao S: An independent brain-computer interface using covert non-spatial visual selective attention. J Neural Eng. 2010, 7: 16010-10.1088/1741-2560/7/1/016010.View ArticlePubMedGoogle Scholar
- Allison B, McFarland D, Schalk G, Zheng S, Jackson M, Wolpaw J: Towards an independent brain-computer interface using steady state visual evoked potentials. Clin Neurophysiol. 2008, 119 (2): 399-408. 10.1016/j.clinph.2007.09.121.PubMed CentralView ArticlePubMedGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.