Crossroads The ACM Magazine for Students

Sign In

Association for Computing Machinery

Magazine: Features From brains to bytes

From brains to bytes

By ,

Full text also available in the ACM Digital Library as PDF | HTML | Digital Edition

Tags: Human computer interaction (HCI)

back to top 

Science fiction has long been fascinated by brain-computer interfaces (BCI)—the use of sensors to identify brain states. From Andre Maurois' 1938 story "The Thought-Reading Machine," in which a professor stumbles on a machine that reads people's thoughts, to the recent blockbuster Avatar, where humans control surrogate bodies with their minds, the public is captivated by the interaction between the human brain and the computers created by those brains.

Although most people are likely to conjure images of Neo's frightening "head port" from The Matrix before they dream of a university student wearing an elastic cap studded with electrodes, the media has closely followed less sinister, if also less all-powerful, BCI research. In the past year, University of Wisconsin-Madison's Brain-Twitter interface received Time Magazine's honor as the no. 9 invention of the year. Furthermore, as brain-imaging technology has become more portable and less expensive, the human-computer interaction (HCI) community has begun to bring science fiction closer to reality.

back to top  Mind Matters

In the larger field of human-computer interaction, we are often concerned with the bandwidth of interaction between a user and the computer. How can we give the computer more information, and more relevant information? How can the computer give us more information without overloading our sensory systems?

Using a mouse in addition to a keyboard increases the bandwidth from the user to the computer by augmenting the type and number of commands the computer can recognize. An application that uses audio increases the bandwidth from the computer to the user, by adding to the type of information the computer can output. Seen in this context, brain-computer interfaces present an opportunity to expand the user-to-computer bandwidth in a unique and powerful way. Instead of identifying explicit actions, we can detect intent. Instead of evaluating action artifacts, we can recognize purpose. Even more interesting, we may be able to understand the user's needs before the user can articulate them.

"As brain-imaging technology has become more portable and less expensive, the HCI community has begun to bring science fiction closer to reality."

But this is all far in the future. On the wide continuum between analyzing electroencephalo-graphs to Avatar mind-machines, where are we now? And perhaps more importantly, where are we going?

In this article, we will discuss several directions for research into brain-computer interaction, and the relative merits using these brain measurements to give the user direct or passive control of computer interfaces. We will also introduce projects across the world that offer a glimpse into the future of BCIs.

Imagine the following scenario:

It's 9:30 p.m., and you're driving in heavy traffic through Boston, unsure of where you're going. Your phone is alerting you of text messages that your sister is sending every 30 seconds. Your GPS is commanding you to turn in half a mile, then a tenth of a mile, but still, you cannot tell which of the six exits off the rotary to take. To make things worse, the radio commercials are blaring, but you are focused on the road, and uncomfortable taking your hand off the wheel to reach for the volume.

Unlike the head-ports in the fantastical world of The Matrix, this scenario is close to our current reality. Also unlike a fantastical movie world, this is the world in which any brain-computer interface will have to function. But how could a brain-computer interface deal with this situation without adding to the already full spectrum of sensory input? Researchers are taking steps toward answering this question and designing BCIs to meet our needs.

Several approaches to BCIs are being pursued using current brain-sensing technology. These systems detect biological changes occurring naturally during the operator's activity. Interface designers can use this information to deduce the operator's state and translate it into commands that adjust the computer accordingly. Changes to the interface can be the result of a direct—voluntary—input, or a passive measure.

back to top  Direct Control Interfaces

Brain interfaces that allow direct control often replace a user's normal motor function (generally used to move mouse cursors or type on a keyboard), and are currently the dominant strain of research in brain-computer interaction. Direct control involves a structured mental activity that results in an explicit command to the computer. To move your mouse to the right, you might imagine moving your hand to the right.

These direct-control interfaces rely on the fact that the brain activity occurring when you move your hand to the right is very similar to the activity that occurs when you imagine moving your hand to the right. This consistency can be used to pair mental "movements" with commands: when participants imagine waving their arms up and down, for example, the volume on their phone might mute, or the zoom level on their screen might change.

Using this mechanism, we can imagine a world for our car scenario in which direct control interfaces are commonly available to everyone.

You have trained yourself to produce specific brain activity to control different technologies in the car (much like how we learn to touch type). Without taking your eyes off the road, you decide to silence your phone, and perform mental arithmetic, which the brain-computer interface recognizes as the command for muting the phone. You imagine swinging your right arm up and down, and the device also recognizes this as a command, turning the radio down. You imagine yourself moving your left pinky, and again, the device recognizes your brain state, redrawing the GPS map in more detail. Through the entire process, your eyes never leave the road, your hands never leave the steering wheel, and you never say a word.

If this sort of control seems contrived, it is because most direct-control interfaces are currently geared toward disabled users, such as paralyzed patients, or people with severe physical disabilities. In disabled users, familiar, physical mental commands can be repurposed to perform other tasks, as these motor skills are not available. However, if we look further into the future, users may not need to perform mental gymnastics, instead using the thought processes that occur in the brain anyway to control external devices.

To survey current direct control research, we further divide the topic into two brain-imaging techniques: invasive and non-invasive. Although brain sensing is not limited to direct control interfaces, we are not aware of any passive systems that use invasive techniques.

back to top  Invasive

Invasive BCIs involve implanting microelectrodes into the grey matter of the brain during neurosurgery in an effort to capture brain activity more accurately. While significant, successful work is being performed on invasive techniques at research facilities such at Brown University and University Wisconsin-Madison, there are many risks, difficulties and limitations involved. Aside from the difficulty of inserting sensors directly into the brain, there is the risk of scar tissue formation, infection to the brain, and the unknown long-term stability of implanted micro-electrodes. Depending on the task and the participant, accomplishing a task using brain states can require long training, and some users may never reach the desired level of reliability.

Despite this, any improvement in communication is worthwhile and rewarding for paralyzed users, who may not have voluntary control over any muscles at all. Invasive BCIs can provide remarkable new opportunities for disabled users; although currently such systems are limited to fairly rudimentary communication such as choosing pictures, letters, or cursor directions, some potential advantages for future applications include repairing damaged sight and providing new functionality to those with paralysis.

back to top  Non-Invasive

Non-invasive direct control technologies, unlike their invasive counterparts, use external systems, such as electro-encephalography (EEG) or functional near-infrared spectroscopy (fNIRS), to measure brain activity. See Figure 1 for an example. Non-invasive imaging technologies are commonly used in a number of fields, including medicine and psychology, and provide valuable insight into brain activity without requiring surgery or implantation. As a result, they are an attractive option for researchers who want portable and flexible systems for measuring and interpreting this data.

EEG measures brain activity using electrodes placed on the scalp that detect electrical potential caused by neurons firing, providing researchers with information about activation in numerous regions of the brain. The complexity of these systems varies with the number of electrodes used and the techniques to process the data captured.

"One recent example of a BCI that uses EEG is a wheelchair that can be controlled through brain activity."

One advantage of this system is that the data recorded has a high temporal resolution. The system can detect brief changes in brain activity, in the millisecond realm. Additionally, it is possible to buy EEG systems that are small, lightweight, and portable. However, a number of limitations affect the utility of the data collected from EEG systems. For example, the EEG system is sensitive to muscle contraction, limiting the user's physical movements during cognitive activity measurements. Additionally, these systems have low spatial resolution. It can be difficult or impossible to determine which precise region of the brain is being activated.

There are many successful direct control paradigms using EEG signal, which include visually evoked potentials (SSVEP), P300-based BCI, motor imagery, event-related synchronization and slow cortical potentials. A great example of work in this direction can be seen by reading Wolpaw et al.'s 2002 article in Clinical Neurophysiology [3]. The P300-based BCI allows selection by taking advantage of brain activity, called a P300, that occurs when your intended target is highlighted. This has been used successfully to create a spelling application.

One recent example of a BCI that uses EEG is a wheelchair that can be controlled through brain activity, created by Brice Rebsamen et al. The researchers created a list of paths to locations in a small apartment and then presented those target locations to users. To select a target, the users were instructed to focus on that target when it was presented to them. After several minutes of training with a participant, the system could detect the desired location with almost perfect accuracy.

Clearly, there are limitations to this system, such as predetermining a list of routes and targets, which may not be possible in large or complex environments. However, this work is an example of the possibilities for EEG systems to be developed and incorporated into more sophisticated technologies.

Functional near-infrared spectroscopy (fNIRS) is a vastly different technology than EEG in that it measures blood flow changes instead of electrical activity. It uses optical wires to emit near-infrared light, which is then refracted from the tissue of the head, including the brain (see Figure 2 for an example of the device). Sensors in the system detect changes in oxygenated and deoxygenated blood in that region [1]. This technology is marked by a high degree of spatial resolution but is less temporally sensitive to changes in brain activity than EEG.

The brain data recorded by fNIRS is less susceptible to movement artifacts and can be used in combination with computer usage. fNIRS has another notable advantage: it has a shorter setup time than EEG, which makes it a more practical option for use in research, government work, and commercial applications. Additionally, the part of the fNIRS system placed on the scalp or forehead is typically small and therefore less bothersome to users than other brain measurement technologies.

Ruldolph L. Mappus IV and his colleagues at the Georgia Institute of Technology have used fNIRS to develop a technology for drawing letters using data collected from brain activity. The subjects were instructed to trace the letters "C" and "J" by performing different mental tasks to control the drawing direction. The researchers noted some difficulties with their design, including the slow fNIRS response time for changes in mental activity. However, the researchers indicated that, in the future, they believe that this work can be expanded to provide users with a broad range of tools to draw using fNIRS technologies.

back to top  Passive Bcis

The previous examples have helped illustrate direct control BCIs, which use brain activity as the primary input device, but which often require considerable user training to generate specific brain states. However, these technologies have a reduced relevance in the ordinary computer environment. A healthy student has little need to hook herself up to an EEG in order to move a cursor. It's easier, faster, and less error-prone to simply use a mouse. With this in mind, our research at Tufts University has turned to passive BCIs, interfaces that detect brain activity that occurs naturally during task performance [2]. Passive BCIs focus on the brain as a complementary source of information, an additional input used in conjunction with conventional computer inputs such as the mouse or keyboard.

"Brain-computer interfaces present an opportunity to expand the user-to-computer bandwidth... Instead of identifying explicit actions, we can detect intent. Instead of evaluating action artifacts, we can recognize purpose."

Generally, passive BCIs use non-invasive measuring techniques such as EEG and fNIRS. fNIRS, as previously mentioned, requires little in the way of set up and imposes relatively few physical or behavioral restraints on the user. Because many passive BCIs aim to observe brain signals that can be used in a relatively ordinary computer task environment, these qualities make fNIRS particularly attractive as a measuring tool.

How would a future of passive BCIs impact our previous car example? Imagine now that you're wearing the headband shown in Figure 3:

You drive, think, and behave normally, as you would before the BCI was introduced. Brain-sensing devices determine that you are mentally overloaded, juggling the phone, GPS, radio, and driving simultaneously. As a result, the system gently simplifies the map on your GPS. You may not have a clear understanding of the neighborhood, but you won't miss this turn. The system subtly turns down the interior dashboard lights to prevent glare on the windshield. Finally, since the sender of the text messages had previously been designated as low-priority, the system silences the phone, waiting for a better time to let you know about the messages.

The principal advantage of passive BCIs is that they operate using the brain activity that occurs naturally during task performance, and so they do not add to the user's task load (in this example, the driver does not have to think about waving her arms or any other extraneous mental process). Currently, we are working towards a similar class of BCIs to create adaptive environments.

In a recent experiment designed to lead to adaptive interfaces, participants played Pac-Man at two different levels of difficulty, while their brain activity was measured using fNIRS. Statistical analysis of the fNIRS measurements allowed the participants to be classified both by their playing state (playing versus rest) and difficulty level (easy versus hard) with high accuracy. Using this data, the interface could be subtly adapted to the task at hand. If the user is resting, play quiet, soothing music. If the user is playing the game, speed up the pace and volume of music to make the experience more intense and enjoyable. Other research at Tufts has examined fNIRS use in an ordinary computer environment, as well as determined differences in semantic and syntactic workload.

Passive BCIs also lead to a completely different paradigm for interaction with the user. It is no longer acceptable to use bold, explicit reactions to brain states that we make in direct control interfaces. Instead, we find ourselves drawn toward gentle, more implicit changes to the interface. Fading, overlaying information, or changing an application's screen real estate are options that, if done slowly enough, may affect the user's focus in a positive way.

While this type of research is still in its infancy, there are countless examples of possible adaptive interfaces that could be used with fNIRS measurements. For example, at any given moment during the day, a stock broker could be overwhelmed with work or experiencing a lull in activity; either way, he must always be aware of the market. A display that gently changes visualizations of the market according to the user's workload could be invaluable. If the stockbroker is not currently exerting a high mental workload, the display can show his stock visualization as highly detailed, giving the stockbroker as much information as possible. See Figure 4. If the stockbroker is working exceptionally hard at his email and cannot afford to be distracted by complex stock information, the display can simply lower the level of detail. In this way, the broker will still recognize major changes in the data without getting bogged down in the details.

We can imagine a class of interfaces that work to benefit the general population, systems that dynamically filter streams of information (Twitter, RSS, email) to accommodate the workload of the user, systems that dynamically adjust teaching methods to best suit a child's learning style.

back to top  Looking Ahead

BCIs currently in development have the potential to open worlds of communication and mobility to disabled users; further up the pipeline, BCIs have the potential to adjust our constantly-busy environments to provide us with the best possible chance of completing our tasks. Whether this is successful navigation to a new place, remotely commanding a robot, or buying all the items on a grocery list, BCIs hold the promise of performing tasks or changing environ-ments in the real world with no physical manipulation at all.

back to top  References

1. Chance, B., Anday, E., Nioka, S., Zhou, S., Hong, L., Worden, K., Li, C., Murray, T., Ovetsky, Y., and Thomas, R. 1998. A novel method for fast imaging of brain function, non-invasively, with light. Optics Express, 411–423.

2. Cutrell, E., and Tan, D. S. 2008. BCI for passive input in HCI. Paper presented at the ACM CHI'08 Conference on Human Factors in Computing Systems Workshop on Brain-Computer Interfaces for HCI and Games.

3. Wolpaw, J.R., Birbaumer, N., Mcfarland, D.J., Pfurtscheller, G., and Vaughan, T.M. Brain-computer interfaces for communication and control. Clinical Neurophysiology, 113 (6). 2002, 767–791.

back to top  Authors

Evan Peck is a PhD candidate in the Human-Computer Interaction lab at Tufts University. His research focuses on creating adaptive brain-computer interfaces to improve user experience with commonplace computer tasks.

Krysta Chauncey is currently a National Research Council Research Associate at the U.S. Army Natick Soldier Research, Development & Engineering Center in collaboration with the Brain-Computer Interface Project at the Tufts Human-Computer Interface Laboratory. Her primary interest is in applying the methods and knowledge of cognitive neuroscience to the development of brain-computer interfaces, particularly those which use electrophysiological or optical brain data as an auxiliary source of information to improve the user's task performance or experience.

Audrey Girouard is a PhD candidate in Computer Science at Tufts University, specializing in HCI. She is currently fascinated by passive brain computer interfaces, studying the use of brain imagery to enhance HCI for all. She is the laureate of the PostGratudate Scholarship from NSERC.

Rebecca Gulotta is a senior at Tufts University and a research assistant in the Tufts Human-Computer Interaction Lab. She studies Human Factors Engineering and Computer Science.

Francine Lalooses is a computer science doctoral candidate at Tufts University, where she is studying human computer interaction and specifically brain-computer interaction. She currently works at The MITRE Corporation while attending Tufts.

Erin Treacy Solovey is a computer science doctoral candidate at Tufts University, where she is interested in next-generation interaction techniques. She is currently investigating the use of brain sensors to detect signals that users naturally give off in order to augment standard input devices.

Doug Weaver is a master's student at Tufts University. Currently, he is participating in research developing adaptive brain-computer interfaces.

Robert Jacob is a professor of computer science at Tufts University, where his research interests are new interaction media and techniques and user interface software. He was also a visiting professor at the Universite Paris-Sud and at the MIT Media Laboratory, in the Tangible Media Group. He is a member of the editorial board of Human-Computer Interaction and the ACM Transactions on Computer-Human Interaction. He was elected to the ACM CHI Academy in 2007.

back to top  Footnotes

DOI: http://doi.acm.org/10.1145/1764848.17664858

back to top  Figures

F1Figure 1. Unlike some BCI systems that require inserting devices directly into the brain, EEG electrode caps are worn externally.

F2Figure 2. An fNIRS sensor with five light sources and one detector.

F3Figure 3. fNIRS sensors placed on the forehead can be non-intrusive when secured by a simple headband.

F4Figure 4. Stockbrokers might use BCIs to project appropriate market visualization based on how much distraction they can handle at the moment. The image on the left is what a broker might see when engaged in low amounts of work, while the more simplified version might be more appropriate when the mental workload is greater.

back to top 

©2010 ACM  1528-4972/10/0600  $10.00

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.

The Digital Library is published by the Association for Computing Machinery. Copyright © 2010 ACM, Inc.


There are no comments at this time.


To comment you must create or log in with your ACM account.


HCI Resources

HCI Resources & Bibliography



Brain-Computer Interface: technology that reads your mind (more or less)


Sounds produced by your body, such as when you touch your arm, which makes a distinctive noise that a computer can distinguish


Electroencephalography: a BCI that measures electrical signals from sensors on your scalp


enabling grids for e-science