Finding pleasant things and doing my best whether in research or pleasure
Dr. Mitsukura is tackling the development of innovative systems based on signal processing technology. She is a short sleeper, but never cuts corners whether it is research or pleasure. The source of her restless energy seems to come from her forward-looking attitude of finding pleasant things and thoroughly pursuing them.
Ms. Mitsukura pursues research related to multimedia signal processing and Bio-Signal analysis, using Bio-Signal processing, brain wave (electroencephalogram:EEG) analysis, visual image processing, semantic analysis of visual images and impression analysis as keywords. In 1999, she became a research assistant for Tokushima University Department of Information Science and Intelligent Systems; in 2002, she became a full-time lecturer for Okayama University Information Education Course; in 2005, she became an associate professor for Tokyo University of Agriculture and Technology in 2007; and in 2011 she moved to Keio University assuming the current position as an associate professor for the Faculty of Science and Technology.
Featured in this issue is Associate Professor Yasue Mitsukura, who pursues the development of innovative systems by making the most of signal processing technology.
The future that analysis of ubiquitous Bio-Signals will open
Brain-computer interfaces can measure our physical/mental states, such as pleasure/pain, stress and sleepiness by measuring and analyzing brain waves (electroencephalogram:EEG). Avatar systems can detect movements of our face and changes in facial expression using a compact web camera and project them instantaneously onto an avatar on the monitor screen . . . explained Associate Professor Mitsukura who focuses on the development of these innovative systems.
“By simply wearing this device on your forehead, it is possible to measure your pleasure/pain, likes/dislikes, stress, sleepiness, or degree of your concentration or interest. It may be safe to say that this one is the most simplified and easiest-to-use device of its kind available today. Of course, its performance capability is never outdone by its category.” Saying so, Dr. Mitsukura showed a headband-like electroencephalograph (EEG). The EEG is designed to send information from the headband-like sensor to a smart phone and instantaneously displays the measurement result on the smart phone screen.
If you are feeling stress, for example, the white face icon on the screen will turn blue and the degree of stress will be shown as a percentage. Likewise, if you start up an application for the state of concentration or sleepiness, as a test subject you will be able to clearly see your constantly changing state of concentration or sleepiness in real time.
Raw data obtained is also displayed on the screen simultaneously. The waveforms shown in line graphs represent frequency bands, such as α, β and θ waves, that are languages commonly used to indicate brain waves. According to Dr. Mitsukura, it is possible to measure a subject’s state by analyzing a combination of his/her brain wave frequencies.
She continues, “You wear a sensor on your forehead to measure the state of what is called the FPI, the area of the brain in the left-side frontal lobe that governs emotions and sensitivity. At this time very small voltages in the microvolt order are measured. Types of brain waves can be known by converting these voltages into frequencies. Brain wave frequencies are limited to the 1~30Hz range. One’s physical/mental states can be known from combinations of pieces of such slight pieces of information.”
Take the sense of touch, for example. You can judge that you “like” an object if the value of a combination of ○Hz and ×Hz is large, while you “dislike” the object if the value of a combination of △ Hz and ◇Hz is small. Finding out such rules and allowing the particular state to be identified instantaneously via the optimization method are the greatest features of this system.
In addition to the EEG (electroencephalograph), brain-measuring instruments available today include the f-MRI (functional magnetic resonance imaging) and NIRS (near-infrared spectroscopy). But each of these systems has its own measurement target. To examine the state of the brain, the f-MRI targets brain blood flow while the NIRS is dedicated to brain blood oxygen level. Here, we have to remember that brain waves quickly respond to any psychological change that occurs. As such, real-time measurement is fit for brain wave measurement.
Furthermore, it has been a major drawback with conventional brain wave measuring instruments that accurate measurement is not possible because the subject must have several electrodes adhered to his/her scalp, which is timeconsuming while wearing the instrument itself is stressful. But Dr. Mitsukura’s device is easy to wear and enables what a person is feeling to be measured accurately. In fact, the device already began to be used as a marketing research tool to determine the degree of interest in sound, taste, new products and so on.
Fig.1 Detection of sleepiness by means of brain wave analysis
Degree of sleepiness can be detected online by easily obtaining brain waves and analyzing them instantaneously.
However, is it really possible to obtain accurate measurement results using such a simple device? No problem. The signal processing technology, Associate Professor Mitsukura’s field of expertise, enables measurement by the device well comparable in accuracy to other types of measuring instruments.
She says, “Most of simplified EEGs adopted in games operate by acquiring myoelectric signals that are generated, say, when you move your eyelids, and they do not deserve being called EEGs. A lot of noises are mixed into brain waves, most of which are myoelectric noises, making it very difficult to selectively pick out pure brain waves. In fact, myoelectric noises are not eliminated with most of the simplified instruments available on the market today. Our device, therefore, is designed to eliminate such noises instantaneously and enable accurate measurements.”
In her early career, Dr. Mitsukura had long engaged in the study of formulating behavior of industrial plants and equipment for years. She says that the experiences she had nurtured through the work of formulating such behavior proved to be a great asset for the development of the current system.
“The impetus for my taking up the development of this system was an encounter with an ALS (amyotrophic lateral sclerosis) patient. As the disease progresses, ALS patients eventually become unable to communicate with the outside world except by the movement of their eyeballs. When the patient who used our EEG system became able to express his Yes/No, his family members were very pleased, which fueled my motivation. I was determined to accelerate this research project by all means. In the future, I aim to make this device an even more simplified and easy-to-use system – one that will allow users to express what they are thinking directly into letters,” she remarks.
Fig.2 Signal processing has much to do with our daily lives
Bio-Signals, visual images, and sounds – each of these represents an aggregate of signals on a certain scale. Signal processing is a process of grasping the characteristics of these signals, classifying them into patterns, and imparting a meaning to each pattern. When it comes to living bodies, we generally deal with signals ranging in voltage from micro-volts to milli-volts; for visual images, signals ranging in RGB value from 0 to 255; and for sounds, signals ranging from 20Hz to 20kHz. Studies of analyzing these signals have much to do with a wide variety of products found in our daily lives.
The Mitsukura lab has also developed another signal processing-based system. It is an innovative avatar system. With this system, a web camera attached to a PC recognizes movements of a person’s face, allowing an avatar on the PC screen to follow the movements in real time.
“We took up this research project with the aim of tracking facial movements and expressions within 0.1 second. The key behind the success of this system was that we enhanced computation speed by narrowing down tracking points – both ends of the eyes and both ends of the mouth – to facilitate instantaneous reactions,” she explains.
As the avatar for this system, she used Hatsune Miku and uploaded it to the YouTube video-sharing website. This attempt was rewarded with over three million views. As a result, she is flooded with numerous inquiries from interested companies regarding its use for animations, promotional events and CMs, among others.
She adds “Also, we are handling the development of various other systems, such as a system for scrolling the PC screen with a whistle and a technology for moving a wheelchair using myoelectricity.”
All of these innovative systems are based on signal processing technology. In conclusion, Ms. Mitsukura vigorously mentioned that she would like to continue to develop systems that could contribute to society by leveraging signal processing technology.
Fig.3 An online avatar system allows you to transform into a popular avatar in an instant
Only if you have a common inexpensive camera (shown in photo is a camera for Playstation) and a PC, you can analyze movements and expressions of your face in an instant and express them using an avatar.
Listening to what Associate Professor Yasue Mitsukura has to say
That’s correct. Although I had moved from one workplace to another to date, I have at last found a place in which I can settle in peace. I chose the scientific course as a high school student presumably because I was raised in a family where my father is a man of science and mother from the medical field. In fact, bookshelves at my home were filled with books on mathematics, physics and medicine – no picture books and the like for children. Even now the scene of my father studiously reading such books on holidays is printed on my memory. I still remember the day when I dropped and broke a piece of glassware. At the time my father explained to his small daughter earnestly and in detail about “why it can be broken.” His explanation was so interesting that I broke various other things merely for the sake of interest – my mother scolded me. Raised in such a family atmosphere, it was natural that I chose a scientific high school and then specialized in electric/electronic studies at a university. I moved from one place to another to study – at Okayama Prefectural University until I completed my master’s course, earned a doctor’s degree at Tokushima University, studied at the University of Tokyo’s Graduate School of Medicine, worked at Okayama University and Tokyo University of Agriculture and Technology, and finally moved to Keio University in 2011.
When I was a senior, I had an opportunity to participate in an international academic conference on telecommunications, where I won the best paper prize and the best presentation prize. In relation to the paper I would read at the conference, I had run into a great wall (though it’s not too serious a problem as I look back at it today) and had to sit up sleepless for several days in a row. However, the moment I won the prizes, all the hardships I had experienced were blown away and replaced with utmost delight, which I still remember vividly. Through this experience, I learned the importance of dedicated effort and willpower necessary for solutions when I come face to face with a problem.
Later, an encounter with Professor Norio Akamatsu during my service with Tokushima University marked an important turning point for my career as a researcher. When we were bothered with noise from a nearby construction site, I spoke to Dr. Akamatsu, saying, “If we produce sound of opposite phase, the noise outside disappears, doesn’t it?” This is nothing special, but Dr. Akamatsu praised me as being always conscious of my own research in relation to daily matters. He even guaranteed that by maintaining such an attitude I would be able to grow into a full-fledged researcher. This word of encouragement has supported me as a researcher ever since. As I continued the work of converting sounds and visual images into frequencies and formulating them day after day, I found myself capable of connecting various phenomena with frequencies, and by merely hearing sounds or seeing images I naturally became able to make out what frequency components are contained there. For example, by a casual look at ripples on the water surface, I became able to intuitively realize what waves are contained and how they can be expressed in a formula. Now everything around me appears as a frequency or a formula – almost an occupational hazard you might say.
Incidentally, at my lab, students jokingly say for example, “Look! Curls in that person’s hair can be expressed by the XX function. Don’t you think so?” It may be safe to say that one’s mathematical sense can be cultivated in greater part through training.
I may have wandered off the point. Now back to the point. Thanks to the advice given by Dr. Akamatsu, I came to engage in research into brain waves. Back in those days, I was totally absorbed in matters related to frequencies. In my doctor’s course, I focused on research into frequency analysis of facial visual images. This project aimed to distinguish individuals by formulating visual images of faces. Individual faces can be distinguished by comparing formulas . . . Don’t you think it’s interesting? After all, I was able to earn a doctor’s degree in a year and a half.
By the way, among the students who were concerned with Dr. Akamatsu is Dr. Shuji Nakamura, who is famous as a researcher of blue LED. Surrounded by such highly progressive researchers, I can say I was extremely happy. Indeed, I have been blessed with good teachers, good researchers and good friends around me.
Day after day, I went to the lab and devoted myself to research work until 2:00 a.m. As a short sleeper by nature, I could refresh myself after five hours of sleep, so such a lifestyle was not hard at all. What’s more, it was my daily routine to enjoy jogging with Dr. Akamatsu in the early evening. Thanks to that habit, I enjoy marathons even today. Every year I take part in major marathon races like the Shonan International Marathon.
When I have to make any decision, I make it a rule to judge by likes/dislikes. Of course I choose what I like, so there is no room for stress to accumulate, I believe. Whenever I find something interesting, I challenge it single-mindedly and with all my energy. When I was working for Okayama University, I even assumed the post of CTO (Chief Technology Officer) for a venture business for a voice compression project. I frequented Tokyo because the venture business’ head office was located in Tokyo and I also had several non-researcher friends in Tokyo. So I had no anxiety at all when I decided to move from Okayama to Tokyo.
To tell the truth, I am the younger sister of identical twins. My elder sister, Hiroe, passed away years ago. The loss of my sister largely changed my way of life. I made up my mind to live a double life including that of my sister – appreciating both pleasures and pains.
Therefore, it is my rule to do research work with all my might and enjoy pleasures with all my might as well. Unless my work schedule is too tight or I have a business trip to take, I set aside Sundays as free days to enjoy myself to the fullest. I sometimes amuse myself, almost sleepless, from 23:00 of Saturday to 23:59 of Sunday. Frequenting drinking places not caring about money, shopping lavishly, enjoying outdoors with friends for camping, bathing, snowboarding, climbing, etc. For example, my holiday schedule for this month is already full (laughter). No matter how much I may enjoy a holiday, I never carry fatigue to the following day. It’s my principle. Enjoying a lot of delicious food is another pleasure although I specialize in eating (laughter). Eating well, learning well, playing well, being moved well by good things, laughing well, and shedding tears occasionally . . . This is my lifestyle.
Maybe running keeps me in good shape. I’m confident in my stamina. On Sundays, I try to run as much as I can. Whenever I take official overseas trips, I bring my jogging shoes with me and run around town so that I can feel the atmosphere of the town firsthand. Running erases the fatigue of journey from Japan to that country. I often invite my students to join in running while also advising them to learn well and play well like myself.
Another way of getting rid of stress is to meet people. If former workplaces are included, 14 years have already passed since my Mitsukura lab. was established, sending many graduates into the world. Many of these graduates who completed their doctor’s courses are now frontline researchers. Meeting them at times gives me a good stimulus. Graduates of undergraduate courses are also actively at work at manufacturing companies and mass media, so meeting them provides me with excellent opportunities of information exchange. The moment I look at their lively faces, I can forget my fatigue and unpleasant things.
The atmosphere of Keio University is truly unrestricted and there are many researchers of colorful characters, so I feel at home here and am enjoying an unrestricted research life. It’s also wonderful that Keio has a culture of caring for each and every student attentively. This is why I like Keio. In years ahead as in the past, I’d like to continue to give full play to my creative imagination – based on signal processing technology – and energetically pursue research themes that will benefit society.
Student : Dr. Mitsukura is very stoic when it comes to research. But even when I come out with an unexpected idea, she generously allows me to go ahead, saying, “Why don’t you try it?” And she is kind enough to follow up my endeavor with proper advice, which is very nice. All of us at the lab are following Dr. Mitsukura’s example – tackling both research work and pleasure with all our might.
(Reporter & and text writer: Madoka Tainaka)