Ubiquitous Sensing
from Keio's Faculty of Science and Technology Challenging the visualization of
“thought and thinking”
– Seeing, observing and examining brain waves
(electroencephalogram:EEG)

Signal processing is being used everywhere in modern society

Brain-computer interfaces can measure our physical/mental states, such as pleasure/pain, stress and sleepiness by measuring and analyzing brain waves (electroencephalogram:EEG). Avatar systems can detect movements of our face and changes in facial expression using a compact web camera and project them instantaneously onto an avatar on the monitor screen . . . explained Associate Professor Mitsukura who focuses on the development of these innovative systems.

“By simply wearing this device on your forehead, it is possible to measure your pleasure/pain, likes/dislikes, stress, sleepiness, or degree of your concentration or interest. It may be safe to say that this one is the most simplified and easiest-to-use device of its kind available today. Of course, its performance capability is never outdone by its category.” Saying so, Dr. Mitsukura showed a headband-like electroencephalograph (EEG). The EEG is designed to send information from the headband-like sensor to a smart phone and instantaneously displays the measurement result on the smart phone screen.
If you are feeling stress, for example, the white face icon on the screen will turn blue and the degree of stress will be shown as a percentage. Likewise, if you start up an application for the state of concentration or sleepiness, as a test subject you will be able to clearly see your constantly changing state of concentration or sleepiness in real time.
Raw data obtained is also displayed on the screen simultaneously. The waveforms shown in line graphs represent frequency bands, such as α, β and θ waves, that are languages commonly used to indicate brain waves. According to Dr. Mitsukura, it is possible to measure a subject’s state by analyzing a combination of his/her brain wave frequencies.
She continues, “You wear a sensor on your forehead to measure the state of what is called the FPI, the area of the brain in the left-side frontal lobe that governs emotions and sensitivity. At this time very small voltages in the microvolt order are measured. Types of brain waves can be known by converting these voltages into frequencies. Brain wave frequencies are limited to the 1~30Hz range. One’s physical/mental states can be known from combinations of pieces of such slight pieces of information.”
Take the sense of touch, for example. You can judge that you “like” an object if the value of a combination of ○Hz and ×Hz is large, while you “dislike” the object if the value of a combination of △ Hz and ◇Hz is small. Finding out such rules and allowing the particular state to be identified instantaneously via the optimization method are the greatest features of this system.
In addition to the EEG (electroencephalograph), brain-measuring instruments available today include the f-MRI (functional magnetic resonance imaging) and NIRS (near-infrared spectroscopy). But each of these systems has its own measurement target. To examine the state of the brain, the f-MRI targets brain blood flow while the NIRS is dedicated to brain blood oxygen level. Here, we have to remember that brain waves quickly respond to any psychological change that occurs. As such, real-time measurement is fit for brain wave measurement.
Furthermore, it has been a major drawback with conventional brain wave measuring instruments that accurate measurement is not possible because the subject must have several electrodes adhered to his/her scalp, which is timeconsuming while wearing the instrument itself is stressful. But Dr. Mitsukura’s device is easy to wear and enables what a person is feeling to be measured accurately. In fact, the device already began to be used as a marketing research tool to determine the degree of interest in sound, taste, new products and so on.

However, is it really possible to obtain accurate measurement results using such a simple device? No problem. The signal processing technology, Associate Professor Mitsukura’s field of expertise, enables measurement by the device well comparable in accuracy to other types of measuring instruments.
She says, “Most of simplified EEGs adopted in games operate by acquiring myoelectric signals that are generated, say, when you move your eyelids, and they do not deserve being called EEGs. A lot of noises are mixed into brain waves, most of which are myoelectric noises, making it very difficult to selectively pick out pure brain waves. In fact, myoelectric noises are not eliminated with most of the simplified instruments available on the market today. Our device, therefore, is designed to eliminate such noises instantaneously and enable accurate measurements.”
In her early career, Dr. Mitsukura had long engaged in the study of formulating behavior of industrial plants and equipment for years. She says that the experiences she had nurtured through the work of formulating such behavior proved to be a great asset for the development of the current system.
“The impetus for my taking up the development of this system was an encounter with an ALS (amyotrophic lateral sclerosis) patient. As the disease progresses, ALS patients eventually become unable to communicate with the outside world except by the movement of their eyeballs. When the patient who used our EEG system became able to express his Yes/No, his family members were very pleased, which fueled my motivation. I was determined to accelerate this research project by all means. In the future, I aim to make this device an even more simplified and easy-to-use system – one that will allow users to express what they are thinking directly into letters,” she remarks.

The Mitsukura lab has also developed another signal processing-based system. It is an innovative avatar system. With this system, a web camera attached to a PC recognizes movements of a person’s face, allowing an avatar on the PC screen to follow the movements in real time.
“We took up this research project with the aim of tracking facial movements and expressions within 0.1 second. The key behind the success of this system was that we enhanced computation speed by narrowing down tracking points – both ends of the eyes and both ends of the mouth – to facilitate instantaneous reactions,” she explains.
As the avatar for this system, she used Hatsune Miku and uploaded it to the YouTube video-sharing website. This attempt was rewarded with over three million views. As a result, she is flooded with numerous inquiries from interested companies regarding its use for animations, promotional events and CMs, among others.
She adds “Also, we are handling the development of various other systems, such as a system for scrolling the PC screen with a whistle and a technology for moving a wheelchair using myoelectricity.”
All of these innovative systems are based on signal processing technology. In conclusion, Ms. Mitsukura vigorously mentioned that she would like to continue to develop systems that could contribute to society by leveraging signal processing technology.

When I was a senior, I had an opportunity to participate in an international academic conference on telecommunications, where I won the best paper prize and the best presentation prize. In relation to the paper I would read at the conference, I had run into a great wall (though it’s not too serious a problem as I look back at it today) and had to sit up sleepless for several days in a row. However, the moment I won the prizes, all the hardships I had experienced were blown away and replaced with utmost delight, which I still remember vividly. Through this experience, I learned the importance of dedicated effort and willpower necessary for solutions when I come face to face with a problem.
Later, an encounter with Professor Norio Akamatsu during my service with Tokushima University marked an important turning point for my career as a researcher. When we were bothered with noise from a nearby construction site, I spoke to Dr. Akamatsu, saying, “If we produce sound of opposite phase, the noise outside disappears, doesn’t it?” This is nothing special, but Dr. Akamatsu praised me as being always conscious of my own research in relation to daily matters. He even guaranteed that by maintaining such an attitude I would be able to grow into a full-fledged researcher. This word of encouragement has supported me as a researcher ever since. As I continued the work of converting sounds and visual images into frequencies and formulating them day after day, I found myself capable of connecting various phenomena with frequencies, and by merely hearing sounds or seeing images I naturally became able to make out what frequency components are contained there. For example, by a casual look at ripples on the water surface, I became able to intuitively realize what waves are contained and how they can be expressed in a formula. Now everything around me appears as a frequency or a formula – almost an occupational hazard you might say.
Incidentally, at my lab, students jokingly say for example, “Look! Curls in that person’s hair can be expressed by the XX function. Don’t you think so?” It may be safe to say that one’s mathematical sense can be cultivated in greater part through training.
I may have wandered off the point. Now back to the point. Thanks to the advice given by Dr. Akamatsu, I came to engage in research into brain waves. Back in those days, I was totally absorbed in matters related to frequencies. In my doctor’s course, I focused on research into frequency analysis of facial visual images. This project aimed to distinguish individuals by formulating visual images of faces. Individual faces can be distinguished by comparing formulas . . . Don’t you think it’s interesting? After all, I was able to earn a doctor’s degree in a year and a half.
By the way, among the students who were concerned with Dr. Akamatsu is Dr. Shuji Nakamura, who is famous as a researcher of blue LED. Surrounded by such highly progressive researchers, I can say I was extremely happy. Indeed, I have been blessed with good teachers, good researchers and good friends around me.

Day after day, I went to the lab and devoted myself to research work until 2:00 a.m. As a short sleeper by nature, I could refresh myself after five hours of sleep, so such a lifestyle was not hard at all. What’s more, it was my daily routine to enjoy jogging with Dr. Akamatsu in the early evening. Thanks to that habit, I enjoy marathons even today. Every year I take part in major marathon races like the Shonan International Marathon.

When I have to make any decision, I make it a rule to judge by likes/dislikes. Of course I choose what I like, so there is no room for stress to accumulate, I believe. Whenever I find something interesting, I challenge it single-mindedly and with all my energy. When I was working for Okayama University, I even assumed the post of CTO (Chief Technology Officer) for a venture business for a voice compression project. I frequented Tokyo because the venture business’ head office was located in Tokyo and I also had several non-researcher friends in Tokyo. So I had no anxiety at all when I decided to move from Okayama to Tokyo.
To tell the truth, I am the younger sister of identical twins. My elder sister, Hiroe, passed away years ago. The loss of my sister largely changed my way of life. I made up my mind to live a double life including that of my sister – appreciating both pleasures and pains.

Maybe running keeps me in good shape. I’m confident in my stamina. On Sundays, I try to run as much as I can. Whenever I take official overseas trips, I bring my jogging shoes with me and run around town so that I can feel the atmosphere of the town firsthand. Running erases the fatigue of journey from Japan to that country. I often invite my students to join in running while also advising them to learn well and play well like myself.

The atmosphere of Keio University is truly unrestricted and there are many researchers of colorful characters, so I feel at home here and am enjoying an unrestricted research life. It’s also wonderful that Keio has a culture of caring for each and every student attentively. This is why I like Keio. In years ahead as in the past, I’d like to continue to give full play to my creative imagination – based on signal processing technology – and energetically pursue research themes that will benefit society.