I’ve always done my best in everything, whether it is study, sports or hobbies. What I am today is nothing less than the result of this way of life
Brought up as the first son of an electrical shop owner in town, Dr. Aoki spent his childhood familiarizing himself with all kinds of electrical appliances and PCs. In his high school and college days, he was extremely active in extracurricular activities such as judo, rugby, band and so on. Indeed, throughout his life he has tackled study, sports and hobbies with all his might and produced a number of achievements in research as well. He is one of the teachers who enjoy an overwhelming popularity among Keio students. How did Dr. Aoki become a researcher? We asked him about his personal background and his career as a researcher.
Born in Takasaki City, Gunma Prefecture, Mr. Aoki graduated from Waseda University Department of Applied Physics in 1996, then completed the doctoral course at Waseda University Graduate School of Science and Engineering (Physics and Applied Physics) in 2001. Doctor of Engineering. After serving as a research assistant at Waseda University Faculty of Science and Engineering, he became an assistant professor for Shibaura Institute of Technology Department of Information and Engineering, then assumed the current position as an associate professor at Department of Electronics and Electrical Engineering, Keio University Faculty of Science and Technology in 2008. In 2013, he concurrently became a board member of IDEAQUEST Inc. and continues to spearhead research aiming to put Keio-initiated image sensing technologies into practical use in the medical field. His specialties include perceptual information processing, intelligent robotics, media informatics/database, measurement engineering, and medical systems.
Featured in this issue is Associate Professor Yoshimitsu Aoki, who pursues research into image information sensing for application to a wide range of fields.
Understanding contexts from human, object and spatial information
Our modern daily lives are brimming with all kinds of visual images and image information thanks to the spread of digital cameras, security cameras and the like. To put it another way, these visual images and information are serving as our vital sensors on understanding society. Associate Professor Yoshimitsu Aoki focuses his energy on the effective use of leading-edge image sensing technology to develop systems beneficial to society. We asked Dr. Aoki about a variety of advanced technologies he is working on at his lab.
Dr. Aoki of the Department of Electronics and Electrical Engineering is currently engaged in an attempt to apply visual images from digital cameras and video cameras for sensing. He says he is interested in developing a system that allows meaningful pieces of information to be extracted automatically, which he wants to use for the benefit of society.
For example, functions such as extracting a particular individual’s facial features to automatically focus on them and functions for recognizing one’s own child and bringing the camera into focus on the child have already been put into practical use with digital cameras and video cameras. Furthermore and steps ahead, he aims to enable the computer to understand a move a subject is going to make, a given situation and even the meaning or import of a whole particular scene.
“I’m targeting not only humans but also recognition of colors and shapes of objects as well as spatial and environmental recognition. In other words, I’m trying to understand the whole of a given scene by using images to recognize the three – human, object and spatial – elements. By combining multiple pieces of information, I think it’ll be possible to understand the context of what’s going on there. Suppose there is a person who is extending his hand. Context understanding in this case means distinguishing whether he is going to shake hands with someone else or is trying to get hold of something,” remarks Dr. Aoki.
Almost everyone carries a smart phone or a tablet device now, casually taking snapshots and videos and uploading them on the Internet. Out on the streets, security cameras are installed everywhere, replacing human eyes. Sensing these images and using them to develop an intelligent sensing system with humanlike senses, sensitivity and thinking, something that can comprehend complex matters and make forecasts . . . this was Dr. Aoki’s motive to take up this research theme.
But why images . . . ? He explains as follows: “The primary merit of using images lies in that images allow us to conduct simultaneous, wide-range measurement of more than one person. Another advantage is that, unlike common sensors, images do not have to be carried by the persons concerned, thus placing no restrictions on their actions.
On the other hand, the use of images can involve privacy problems. Images are also useless unless the target can be captured by camera due to blind spots and so on. Furthermore, it is difficult to process moving images in realtime because of their large volume of information. In this age of overflowing image information, why can’t we make full, more effective use of images? . . . This is the ever-stronger voice from the real world.”
Fig.1 Image sensing technologies from Aoki lab
Aoki lab’s core technologies cover: a variety of sensing technologies targeting humans for measurement and recognition; and imagery measurement and recognition technologies targeting objects and spaces. Results of sensing thus obtained are applied for automatic image analysis, scene understanding, extraction of sensitivity information, and medical diagnosis support systems, among others.
Above all, Dr. Aoki focuses on a technology with the ability to detect and track a particular individual in images. “By applying this technology to security cameras, it will become possible to identify a suspicious person or detect a lost person. If used with security cameras at convenience stores, for example, it will recognize what product items customers are interested in – an application for marketing research.”
Dr. Aoki is also tackling a more challenging theme in relation to sports events such as soccer or American football games, where players of each team wear the same uniforms with different numbers. The challenge here is to detect movements and/or positions of a particular player (or the ball) from among many players, relying on his number only. These technologies can be applied for the understanding of tactics and prediction of plays. In fact, Dr. Aoki is receiving business inquiries from pro sports teams, TV stations, game software makers and so on.
“For this technology, we employ a method based on modeled human postures, which allows the computer to learn beforehand information about how positions of a person’s body parts, e.g. from the head to shoulders, will change. In doing so, we compress the information volume significantly by using a technique known as ‘principal component analysis,’ which is the secret,” continues Dr. Aoki.
Principal component analysis is a mathematical technique to show in a limited number of indices the features of given multidimensional data without destroying important information. By using this technique, it becomes possible to express an individual’s body shape features from a limited volume of information. Furthermore, by establishing a link model for a human body from the upper to the lower halves, it is also possible to track an individual’s movements and postures in real-time and robustly.
“Thanks to this technology, I can now obtain more detailed information about human postures, such as determining to what product on a shelf a person is trying to extend his/her hand. In addition, by applying this knowledge I could successfully develop an innovative technology capable of instantaneously restoring one’s three-dimensional body shape from only two pieces of image showing his/her front and side,” he says.
In this manner, the excellence of Dr. Aoki-initiated technologies lies in that they can speedily process highly complex movements and shapes using an extremely limited volume of information, which can be applied to actual systems.
Fig.2 Flow of human behavior recognition from image information
This process begins with modeling of an individual’s features based on a limited amount of information. The modeled features are given to computer as prior knowledge and are compared stochastically with features obtained from visual images. This enables detection and tracking of a target that has human-like features. Expectations are high for applying this technology for a wide range of fields, such as analysis of buying behavior at shops, analysis of sports tactics, recognition of everyday behavior, and even the prediction of next actions to be taken.
The essence of Dr. Aoki’s research initiatives can be described as pursuit of real-time processing of information using minimal volume of information. What lies beyond? . . . It should be practical application of the research results. What he has in mind as application outlets for his research results are fields, such as sports, entertainment, apparel, medical and social welfare.
Examples of potential medical application are measurement systems for the detection of babies’ respiratory failures and for the swallowing ability of the elderly. For the former, the system is intended to measure subtle up-and-down movements of the chest, and for the latter it will measure movements of the Adam’s apple. The greatest advantage is that both of these measurement systems will have no invasiveness.
Another feature of the Aoki lab is that it engages in research into a sensitivity information processing system designed to measure human sensitivity. For example, suppose one accesses an EC (electronic commerce) web site. In this case, the system will estimate the user’s personal taste for clothing and recommend a product most suited for his/her sense or liking.
“At my lab, 60 to 70 percent of the research activities are joint projects with businesses. This means expectations are high among businesses for our image sensing technologies,” concludes Dr. Aoki. Being close to our everyday life, it seems only natural that expectations continue to grow for practical application of these technologies.
Associate Professor Yoshimitsu Aoki
Right. My parents were running an electrical appliance shop. As an elementary school boy, it was almost my daily routine to drop in at the shop after school. So as far as I can remember, I spent most of my childhood surrounded by all kinds of electrical appliances. A so-called “electrical appliance freak,” whenever my father opened his mouth, he talked only about electrical appliances – even at home! It was only natural that I became interested in electrical appliances. What’s more, my father liked novelties. Back in those days, for example, plasma TVs were recently launched products costing millions of yen, but money was no object; he bought one and proudly displayed it at the front of the shop. An electrical shop is a community-oriented business by nature, but my father’s shop seems to have been a trendsetting one.
Another example typifying my father’s shop was that it was quick to handle PCs and opened the whole of its second floor as a PC-dedicated space, where PCs from various makers were made available and visitors were welcome to freely try a variety of software packages. On my way from elementary school, I often visited the PC floor together with my friends in the neighborhood, enjoying time programming games while referring to the “Basic Magazine.” It was in those days that I vaguely felt that I would choose the scientific course in the future.
In the meantime, I loved sports of various kinds. As an elementary school boy, I took up soccer and swimming; when I entered junior high school, I was cheated by a “tricky” senior (Laughter) to join the volleyball club eventually to become the captain. Having joined the club, I was really shocked to find that almost all of its members were bad boys. The only good thing about being the club member was that I learned how to deal with bad boys, I should say (Laughter).
To tell the truth, I was at a loss which school to choose, Honjo High School of Waseda University or Shiki High School of Keio University. Given my personal character, I finally chose Waseda’s Honjo High School because of its “unrefined” school color. Years later, I’m now teaching at Keio University – back in those days I didn’t even dream of becoming a Keio teacher.
Being a high school attached to Waseda University meant that there would be no university entrance exam, so I was able to adsorb myself in extracurricular club activities. In fact, my school life was extremely busy – setting up a cheering party together with my seniors; playing active roles as a judo club member; beating a drum at a light music club; at the school cultural festival, I held positions in three different bands, and then appeared on cheering party’s stage during the night festival. What is deeply impressed on my mind is that we staged a music performance as a copy band of the then popular “Seikima-II” in a demon-like imitation of that band. That said, I also studied as hard as other students. Incidentally, the theme of my high school graduation thesis was: “Energy toward the 21st century – The necessity of solar energy as an alternative energy in place of fossil fuels.” In those days, I was convinced that petroleum would be completely depleted by the dawn of the 21st century.
Actually not. When I was in the third year of high school, I was a bit in doubt as to which course to choose, the Faculty of Political Science and Economics or Faculty of Science and Engineering. But I finally carried out my original intention and chose the latter. Of the many departments of the faculty, I had a vague longing for architectural science but gave up the idea quickly because I knew I lacked a sense of design. Then what attracted my interest instead was the study of space physics, so I chose the Department of Physics and Applied Physics. If I remember correctly, I chose Applied Physics because I thought it would be the better way for approaching practical use.
That said, I became absorbed in sports again. I joined the faculty’s rugby club; I spent the first one and a half years of the campus life with practice, games and drinking parties day in, day out. In September when I was a sophomore, however, I suffered a serious injury. As a rugby beginner, I was allowed to take part in games only as a sophomore. During a game against the University of Tokyo, I had a neck bone broken when I was making a tackle. It was the first cervical vertebra. But it were the second cervical vertebra or any of the lower neck bones, one half of my body would have been paralyzed.
I was confined in hospital for three months, ruining the latter half of the second year on campus. Although my neck was fixed with gypsum, as I became accustomed to it, I became able to move around inside the hospital and even began exercises using a treadmill and muscle training machines as part of rehabilitation. Looking back at my life, I think my health condition was best when I left the hospital (Laughter). Despite such circumstances, my classmates were kind enough to lend me their notebooks; I was somehow able to clear the new year exams and become a junior without staying two years as a sophomore.
Incidentally, although I give up rugby soon after the accident, I returned to the rugby club from the middle of the third year.The last game as a senior was against the Keio (Faculty of Science and Technology) team. Coming from behind, our team finally won a victory with a try during overtime. The game took place nowhere else but on the rugby field on this Yagami campus of Keio! I cherish this memory even today.
Following the serious injury I suffered during the rugby game, as a junior I began to feel it difficult to understand theoretical physics-related subjects. At that time I attended a “Principles of Measurement” lecture by Professor Shuji Hashimoto who would later become my teacher, when I was impressed with the depth of measurement. My interest suddenly shifted from science to engineering. What made the Hashimoto lab a unique one among the many labs in the Applied Physics Department was the fact it was dealing with exciting themes such as facial image processing and humanoid robots. I didn’t think about joining the Hashimoto lab seriously. My intention was an easygoing one. Since the lab was so popular, I thought it might be easier for me to find employment with a company if I failed in the graduate school exam. In fact, a number of my seniors at the rugby club found employment with trading firms, and becoming a trading firm businessman seemed a rather attractive idea because I liked talking and drinking with other people and was confident in my stamina as well.
Fortunately, I passed the graduate school exam; at the graduate school I took up the study into facial image recognition and synthesis. For example, one of my studies concerned simulated images showing changes in facial appearances and teeth occlusion before and after orthodontic treatment.
I made a presentation of the results of this research at a meeting of the Japanese Academy of Facial Studies, which attracted the attention of a professor of Kyushu University Faculty of Dentistry and led to joint research. Joint research activity with Kyushu University continued until I worked on my doctoral thesis and even after I moved to Shibaura Institute of Technology as an assistant professor. I was lucky to have the opportunities to experience, in the early stages of my career, such joint research projects and advising students of the image study group.
Later I obtained a doctoral degree and had served as a research assistant at Waseda University, then proceeded to Shibaura Institute of Technology (SIT – College of Information Science and Engineering) as an assistant professor in 2002 almost at the same time as I got married. In the first year at SIT, I took care of ten students, and a cumulative total of 80 students up to the end of the academic year 2007. In the early years of my service there, most students did not advance to graduate school. But eventually I was able to nurture my lab where approximately 70 percent of its member students moved up to graduate school. To make it possible, I worked hard to improve the lab’s research environment. Endeavors included holding research presentation meetings jointly with other labs both inside and outside of SIT, and initiating joint research projects for the purpose of obtaining research funds. Incidentally, joint research presentation meetings continue to be held even to this day.
I can see many aspects in common between Keio and Waseda presumably because as the leading private universities Keio and Waseda have improved each other through friendly rivalry over the years. If asked about a visible difference between the two, I’d dare to say that Keio students are smarter than Waseda’s, good or bad. A good thing about Keio is that senior students take good care of their juniors. Keio is also complete with systems that encourage academic pursuits. Three of our lab’s doctoral students studied overseas taking advantage of these systems.
Two of my children – an elementary school fourth-grader boy and a kindergarten middle-grader girl – are practicing judo. So I get rid of stress by joining them in judo practice once a week. I video all of judo matches of my small ones, editing the videos and analyzing tactics myself (Laughter). Sometime in the future, I’d like to take up research into image-based automatic analysis to identify judoists’ center of gravity while they are practicing.
Some words from students
Student : A hot-blooded person with a bit of severity, Dr. Aoki is a truly wonderful teacher who is always and sincerely considerate of us students. In particular, he always makes the best possible effort to prepare an ideal research environment for us. He was once a rugby player himself. He reminds us of the image of Mr. Takizawa, the hero of the “School Wars” TV drama series (Laughter). From Dr. Aoki, all of us are learning many things . . . the delight and severity of learning, among others.
(Reporter & text writer: Madoka Tainaka)