Image Sensing
from Keio's Faculty of Science and Technology A new world opened up
by advanced image recognition technology

I’d like to obtain useful information from visual images and use it to benefit society

Our modern daily lives are brimming with all kinds of visual images and image information thanks to the spread of digital cameras, security cameras and the like. To put it another way, these visual images and information are serving as our vital sensors on understanding society. Associate Professor Yoshimitsu Aoki focuses his energy on the effective use of leading-edge image sensing technology to develop systems beneficial to society. We asked Dr. Aoki about a variety of advanced technologies he is working on at his lab.

Dr. Aoki of the Department of Electronics and Electrical Engineering is currently engaged in an attempt to apply visual images from digital cameras and video cameras for sensing. He says he is interested in developing a system that allows meaningful pieces of information to be extracted automatically, which he wants to use for the benefit of society.
For example, functions such as extracting a particular individual’s facial features to automatically focus on them and functions for recognizing one’s own child and bringing the camera into focus on the child have already been put into practical use with digital cameras and video cameras. Furthermore and steps ahead, he aims to enable the computer to understand a move a subject is going to make, a given situation and even the meaning or import of a whole particular scene.
“I’m targeting not only humans but also recognition of colors and shapes of objects as well as spatial and environmental recognition. In other words, I’m trying to understand the whole of a given scene by using images to recognize the three – human, object and spatial – elements. By combining multiple pieces of information, I think it’ll be possible to understand the context of what’s going on there. Suppose there is a person who is extending his hand. Context understanding in this case means distinguishing whether he is going to shake hands with someone else or is trying to get hold of something,” remarks Dr. Aoki.
Almost everyone carries a smart phone or a tablet device now, casually taking snapshots and videos and uploading them on the Internet. Out on the streets, security cameras are installed everywhere, replacing human eyes. Sensing these images and using them to develop an intelligent sensing system with humanlike senses, sensitivity and thinking, something that can comprehend complex matters and make forecasts . . . this was Dr. Aoki’s motive to take up this research theme.
But why images . . . ? He explains as follows: “The primary merit of using images lies in that images allow us to conduct simultaneous, wide-range measurement of more than one person. Another advantage is that, unlike common sensors, images do not have to be carried by the persons concerned, thus placing no restrictions on their actions.
On the other hand, the use of images can involve privacy problems. Images are also useless unless the target can be captured by camera due to blind spots and so on. Furthermore, it is difficult to process moving images in realtime because of their large volume of information. In this age of overflowing image information, why can’t we make full, more effective use of images? . . . This is the ever-stronger voice from the real world.”

Above all, Dr. Aoki focuses on a technology with the ability to detect and track a particular individual in images. “By applying this technology to security cameras, it will become possible to identify a suspicious person or detect a lost person. If used with security cameras at convenience stores, for example, it will recognize what product items customers are interested in – an application for marketing research.”
Dr. Aoki is also tackling a more challenging theme in relation to sports events such as soccer or American football games, where players of each team wear the same uniforms with different numbers. The challenge here is to detect movements and/or positions of a particular player (or the ball) from among many players, relying on his number only. These technologies can be applied for the understanding of tactics and prediction of plays. In fact, Dr. Aoki is receiving business inquiries from pro sports teams, TV stations, game software makers and so on.
“For this technology, we employ a method based on modeled human postures, which allows the computer to learn beforehand information about how positions of a person’s body parts, e.g. from the head to shoulders, will change. In doing so, we compress the information volume significantly by using a technique known as ‘principal component analysis,’ which is the secret,” continues Dr. Aoki.
Principal component analysis is a mathematical technique to show in a limited number of indices the features of given multidimensional data without destroying important information. By using this technique, it becomes possible to express an individual’s body shape features from a limited volume of information. Furthermore, by establishing a link model for a human body from the upper to the lower halves, it is also possible to track an individual’s movements and postures in real-time and robustly.
“Thanks to this technology, I can now obtain more detailed information about human postures, such as determining to what product on a shelf a person is trying to extend his/her hand. In addition, by applying this knowledge I could successfully develop an innovative technology capable of instantaneously restoring one’s three-dimensional body shape from only two pieces of image showing his/her front and side,” he says.
In this manner, the excellence of Dr. Aoki-initiated technologies lies in that they can speedily process highly complex movements and shapes using an extremely limited volume of information, which can be applied to actual systems.

The essence of Dr. Aoki’s research initiatives can be described as pursuit of real-time processing of information using minimal volume of information. What lies beyond? . . . It should be practical application of the research results. What he has in mind as application outlets for his research results are fields, such as sports, entertainment, apparel, medical and social welfare.
Examples of potential medical application are measurement systems for the detection of babies’ respiratory failures and for the swallowing ability of the elderly. For the former, the system is intended to measure subtle up-and-down movements of the chest, and for the latter it will measure movements of the Adam’s apple. The greatest advantage is that both of these measurement systems will have no invasiveness.
Another feature of the Aoki lab is that it engages in research into a sensitivity information processing system designed to measure human sensitivity. For example, suppose one accesses an EC (electronic commerce) web site. In this case, the system will estimate the user’s personal taste for clothing and recommend a product most suited for his/her sense or liking.
“At my lab, 60 to 70 percent of the research activities are joint projects with businesses. This means expectations are high among businesses for our image sensing technologies,” concludes Dr. Aoki. Being close to our everyday life, it seems only natural that expectations continue to grow for practical application of these technologies.

To tell the truth, I was at a loss which school to choose, Honjo High School of Waseda University or Shiki High School of Keio University. Given my personal character, I finally chose Waseda’s Honjo High School because of its “unrefined” school color. Years later, I’m now teaching at Keio University – back in those days I didn’t even dream of becoming a Keio teacher.

Following the serious injury I suffered during the rugby game, as a junior I began to feel it difficult to understand theoretical physics-related subjects. At that time I attended a “Principles of Measurement” lecture by Professor Shuji Hashimoto who would later become my teacher, when I was impressed with the depth of measurement. My interest suddenly shifted from science to engineering. What made the Hashimoto lab a unique one among the many labs in the Applied Physics Department was the fact it was dealing with exciting themes such as facial image processing and humanoid robots. I didn’t think about joining the Hashimoto lab seriously. My intention was an easygoing one. Since the lab was so popular, I thought it might be easier for me to find employment with a company if I failed in the graduate school exam. In fact, a number of my seniors at the rugby club found employment with trading firms, and becoming a trading firm businessman seemed a rather attractive idea because I liked talking and drinking with other people and was confident in my stamina as well.
Fortunately, I passed the graduate school exam; at the graduate school I took up the study into facial image recognition and synthesis. For example, one of my studies concerned simulated images showing changes in facial appearances and teeth occlusion before and after orthodontic treatment.
I made a presentation of the results of this research at a meeting of the Japanese Academy of Facial Studies, which attracted the attention of a professor of Kyushu University Faculty of Dentistry and led to joint research. Joint research activity with Kyushu University continued until I worked on my doctoral thesis and even after I moved to Shibaura Institute of Technology as an assistant professor. I was lucky to have the opportunities to experience, in the early stages of my career, such joint research projects and advising students of the image study group.
Later I obtained a doctoral degree and had served as a research assistant at Waseda University, then proceeded to Shibaura Institute of Technology (SIT – College of Information Science and Engineering) as an assistant professor in 2002 almost at the same time as I got married. In the first year at SIT, I took care of ten students, and a cumulative total of 80 students up to the end of the academic year 2007. In the early years of my service there, most students did not advance to graduate school. But eventually I was able to nurture my lab where approximately 70 percent of its member students moved up to graduate school. To make it possible, I worked hard to improve the lab’s research environment. Endeavors included holding research presentation meetings jointly with other labs both inside and outside of SIT, and initiating joint research projects for the purpose of obtaining research funds. Incidentally, joint research presentation meetings continue to be held even to this day.

I can see many aspects in common between Keio and Waseda presumably because as the leading private universities Keio and Waseda have improved each other through friendly rivalry over the years. If asked about a visible difference between the two, I’d dare to say that Keio students are smarter than Waseda’s, good or bad. A good thing about Keio is that senior students take good care of their juniors. Keio is also complete with systems that encourage academic pursuits. Three of our lab’s doctoral students studied overseas taking advantage of these systems.