Passion for computers and virtual reality

Dr. Sugimoto says that during his childhood he did not have a TV set at home. It may have been his counteraction to such a home environment that he became crazy about computers as a junior high school student. His devotion to computers continued to grow so much so that he came into a spotlight as a major player in inter-college virtual reality contests during his college days. His current hobby is the creation of CG design, indicating his endless enthusiasm for computers. In Dr. Sugimoto’s lifestyle, virtually no gap can be seen between his likes and professional research work, with which his students seem to empathize. His lab is always full of students’ vigor.


Maki Sugimoto

Department of Information and Computer Science

Dr. Sugimoto graduated from Iida High School in Nagano Prefecture, Japan. He received his Doctor of Philosophy in Engineering from The University of Electro-Communications. He was a visiting researcher of NTT Communication Science Laboratories, a research fellow of the Japan Society for the Promotion of Science, and a visiting scholar of MIT Computer Science and Artificial Intelligence Laboratory. He became a senior assistant professor in Graduate School of Media Design, Keio University in 2008. In 2011, he assumed the current position as an assistant professor in Department of Information and Computer Science, Faculty of Science and Technology, Keio University.

Introducing Researchers

Featured in this issue is Assistant Professor Maki Sugimoto, who pursues research into new forms of interface using augmented reality (AR).

Styles of communication being transformed by augmented reality

What is AR that superposes virtual environment upon real environment?

Recent years have seen application of Augmented Reality becoming increasingly popular along with the keyword “AR” in a wide range of fields such as smart phone applications, games, and advertising/sales promotional tools. Here, let’s have a look at several concrete examples to overview the initiatives being developed by Assistant Professor Maki Sugimoto who pioneers the development of new styles of AR and pursues research into nextgeneration communication tools.

Spread of AR technology is accelerating in pace with its ever-expanding application to entertainment and advertisement.

Augmented Reality (AR) is the research field Dr. Sugimoto of the Department of Information and Computer Science is currently dedicated to. AR is a field deriving from Virtual Reality (VR) with which we are familiar in SF movies and games.
“While VR studies mainly aim to create sensory information that immerses users into a virtual environment built within the computer, AR aims to augment sensory information by superposing information from the computer upon that of our actual living environment (see Fig. 1 (Right)). In other words, by superposing a computer-created virtual environment upon the actual world in real time, AR allows us to access the virtual environment in a more natural way,” as Dr. Sugimoto outlines his field of study.
A recent momentum that aroused interest in AR among the general public was the impact of the animation “Coil – A Circle of Children” televised on NHK (Japan Broadcasting Corporation) in 2007. The animation depicts scenes of children experiencing AR in their daily lives using a compact HMD (Head Mounted Display).
Meanwhile, “ARToolKit” software, developed by Professor Hirokazu Kato of Nara Institute of Science and Technology, is also worthy of special mention as having contributed to the spread of AR. It can make a virtual character or object appear on the screen after camerascanning a pattern printed on a sheet of paper. Given its availability as an open source, many character animations produced by means of ARToolKit have been contributed to video-sharing websites, while video games and advertising/ sales promotional tools based on similar technologies are also spreading lately.
“For example, there is a tool capable of drawing a computer graphics model of a vehicle’s body on the screen if you hold the vehicle’s promotional pamphlet over the camera, allowing you to confirm the vehicle’s running state or internal structure. Competitive games combining a card game and CG are also rising in popularity. Thus, AR has now become a vital technology indispensable to advertising and entertainment.”

Fig.1 Human-computer interfaces taking physical presence and spatiality into account
(Left) With this “Stickable Bear” interface, the bear-shaped robots of a size fitting in the palm of a hand support communication via gestures. The bear shape was created using a 3D printer.
(Right) This is an attempt to realize markerless tracking of location and/or attitude using a range image camera in a Spatial Augmented Reality environment. A texture projected by the projector follows a three-dimensional shape.

Not only on a display screen but fusion in a real environment as well

You may talk about “AR” broadly, but its techniques vary widely. Currently most popular is a system known as “video see-through.” Like the above-mentioned ARToolKit, the video see-through system realizes AR by superposing on the display the information created on the computer upon images taken by a camera. Meanwhile, the system known as “optical seethrough” uses a transmissive display and is equipped with a half mirror and a see-through-type HMD. It is capable of superposing CG on real images that are optically visible. Furthermore, the system known as “spatial AR” can transform a real environment by projecting virtual images onto the real environment. The “Projection Mapping” event held in the newly renovated Tokyo Station Building in 2012 is a typical example of spatial AR.  “AR techniques are also very useful in work support. Application of AR to work support has already begun – for example, a patient’s excision area can be clearly indicated by superposing a prephotographed CT or MRI image onto his or her affected part at the time of surgical operation. Thanks to the recent spread of smart phones and personal digital assistants (PDA), the number of AR-based services easily available to general consumers is increasing. “Layar” combining location data and a camera is a fine example”  Dr. Sugimoto continues, “With the recent spread of AR, it is now perceived as a broader technology that can not only augment sensory information on the display screen but can also directly transform the real environment itself. This makes AR studies all the more interesting and worthwhile.”

Measuring and controlling robot movements using an optical ruler

Dr. Sugimoto is currently working on the development of a dynamic type of spatial AR, in which visual information is projected onto the real environment.
He adds, “By using devices such as projectors and robots, I’m trying to projec t information contained in the computer directly onto a real environment.”
“For example, in the ‘Augmented Coliseum’ competitive shooting game using vehicle-type robots, robot movements are controlled by identifying accurate locations and attitudes of the robots mounted with optical sensors that can read fiducial images projected from the projector (see Fig. 2(Above)). By dynamically projecting a so-to-speak “optical ruler,” the system realizes a highly accurate AR environment.”
The greatest merit of this system is that it is possible to identify highly accurate locations of the robots only by using information read by the five robot-mounted sensors. Compared with common image sensors, the system allows computer computational complexity to be reduced significantly while also making it possible to move the robots in real time.
Dr. Sugimoto is also proceeding with a research project to achieve real-time synchronization, in an AR environment, of movements of a robot at hand with those of a robot in a remote location.
“Suppose you are doing a desk study to determine the arrangement of things inside a building. In this case, if you moved an object in front of you, a similar object in front of a person in a remote location would move in synchronization. This surely will enhance your sense of reality.”
In this relation, he is also engaged in the development of a communication tool that allows an optical sensor mounted on a compact bear-shaped robot to read information on the computer screen and make various gestures in accordance with the information (see Fig. 1(Left)).
“What I want to realize by using augmented reality is flexible interface between humans and information. By presenting information with spatiality while setting much value on physical nature in the real environment, I would like to give concrete shape to abstracted information, thus contributing to smoother communication among people,” concludes Dr. Sugimoto.

Fig. 2 Robots acting in harmony with AR environment
(Above) Based on the concept of Display-Based Computing, the entertainment environment “Augmented Coliseum” is capable of measuring and controlling by means of a projector and optical sensors mounted on vehicle-type robots.
(Below left) This is a projection system for building up a spatial AR environment. Collaboration with robots has been made easy by exerting ingenuity on projection patterns.
(Below right) This is a remote-controllable vehicle-type robot based on the video see-through AR technology. It is possible to control the robot while confirming future predictive images that have taken into account the interaction between the surrounding environment and the robot.

(Reporter & text writer : Madoka Tainaka)


Assistant Professor Maki Sugimoto has to say

Gaining interest in computers as a junior high school student

We heard that you are from Iida City, Nagano Prefecture. What was your childhood like back then?

Both of my parents graduated from universities of fine arts, and they were working in the same industrial design firm in Tokyo. One day, they were compelled to enjoy a self-sufficient life in the countryside and moved from Tokyo to Nagano. Thanks to their education policy of raising a child unrestrictedly in an environment of superb nature, I had lived a life without TV up to the age of elementary school upper grades. Indeed, as a small boy I spent everyday running about fields and hills.

Does it mean you studied very little?

My mother was then teaching at a cram school in Nagano, so she earnestly tried to teach me English and mathematics. But I was always slipping out of her hands because I didn’t like to be taught face-to-face by my own mother (laughter).
Perhaps as retaliation to the life without TV, I came to take a special interest in computers when I was a junior high student. Toward the end of elementary school, I encountered computers for the first time in life when I visited the home of my cousin. There I found and touched an MSX, a computer for beginners. Since then, even before owning a computer of my own, I bought a computer magazine every month and was absorbed in reading them.

It was after I went to high school that I actually got a personal computer of my own. It was a high school admission congratulatory present from my father, who became unable to let my computer fever pass unnoticed. During my high school days, I belonged to a physics circle, where I spent after-school hours with other circle members devising computer programs. For example, we defined an original markup language and developed a software program with a presentation function combining text and illustrations – something like what we call PowerPoint today. We used the software to prepare materials to be presented on the occasion of the school cultural festival. In fact, our program securely supported even animation functions. After I obtained my own computer, I became even more fascinated by informatics so much so that a technical book on lossless compression became my favorite reading during my high school days. By the way, I still cherish the memorable computer that my father bought me.

Deciding to choose a researcher's career taking the opportunity of a contest in which I participated as a univetsity freshman

What made you decide to choose a researcher’s career?

As an undergraduate freshman, I took part in the International Collegiate Virtual Reality Contest (IVRC), an event presided over by the then University of Tokyo Professor Susumu Tachi (now Special Research Professor of Keio University, and Professor Emeritus of the University of Tokyo). This turned out to be the first opportunity for me to think about a career as a researcher.
The seniors in the circle that I joined upon university admission were the contest winner for the preceding year, whom aroused my interest. What my circle seniors developed then was a device called “Virtual Bobsleigh.” It was just before the opening of the Nagano Winter Olympics. They made a model bobsleigh, with which one could experience virtual bobsledding. To tell the truth, my seniors didn’t want to participate in that year’s contest because they had overworked themselves in the contest the preceding year. But I persistently persuaded them and could somehow make an entry as a team.
My first production as a freshman was a tele-existence system that creates a sense of self-projection to a target object placed in a water tank. Though it was the first ever challenge for me, we were honored with an encouragement award of The Virtual Reality Society of Japan. From the second year on, we also made entries with the help of many friends as well as a professor who became the supervisor of my graduation thesis. In the third year, our team created a system that allowed one to walk around freely inside a computer memory unit’s folder structure expressed as a 3D environment. This system won both the encouragement award and technical award. The technical award was worthy of special mention as it had been monopolized over the years by the Tokyo Institute of Technology’s Robotics Study Society, famous for their achievements in the development of excellent systems. I was elated when our team won this award.
Through entries in this contest, I was blessed with opportunities to work with front-line researchers, such as Professor Taro Maeda and Professor Hideyuki Ando (both currently at Osaka University) who became colleagues after completion of my master’s course, and Professor Masahiko Inami (now at Keio University) who was my mentor of my doctoral course. It was a truly valuable experience, awakening me to the excitement of studies.
In my doctoral studies, it was particular significant that I could help develop the basic approach of measuring optical device locations in real time by using the intensity gradient index (index for density of light), which I devised originally. During that time, I was working with Professor Inami in research into optical measurement via space-time division multiplex. This achievement was favorably recognized by SIGGRAPH Emerging Technologies and other forefront academic societies, which in turn gave me confidence and further fueled my enthusiasm to continue to follow a researcher’s career.

You are now involved in IVRC administration, aren’t you?

I turned to the event’s administrative side when I was a senior, and now I’m supporting it as an executive committee member. This year we celebrate the 20th milestone for the annual IVRC event. Over the past two decades, entry contents have become very diverse in pace with the development of technologies. The event has also produced a great number of VR and AR researchers including myself. Since it is a very challenging and meaningful opportunity, I’m recommending my lab students to actively participate in IVRC. I’d like to see continued participation of as many prospective researchers as possible in the future.
I also had the experience of designing the IVRC webpage though I’m not responsible for it now. I undertook the webpage design because I enjoy CG design as a hobby and to get away from research work. During my student days, I even spent my living expenses worth several months to buy very expensive 3D CG software. Back in those days, it was a decision as desperate as making a leap in the dark. I am very envious of today’s students because at educational institutions today, it is possible for students to take advantage of a variety of highly functional software programs for free.
The fact is I’m making effective use of CG, which I have created as a pastime, for presentation materials and web contents. So this pursuit combines utility and a hobby. The other day, I happened to find in NASA’s website a CG model of the “Hinode” (Solar-B) satellite that I created for the National Astronomical Observatory of Japan when I was a doctoral student. I am very happy to see my own CG works being enjoyed and used by many people.
Recently, my lab introduced a “3D printer” (3D shaping equipment) as an application of CG design. It is capable of making fixtures for devices that we use for research on the spot by designing CG models. I’m extremely delighted that my hobby of CG design is actually useful in our studies.

Of course, not everything in my life, whether it is my research or pastime, revolves around a computer. Since three or four years ago, our team of researchers has been making entry in “Eco-Run” – an auto event in which participating light vehicles run on the Fuji Speedway racing course. At my home, I have a home theater of 130-inch LCD projector, with which I view SF movies and animations from time to time. At the end of each semester, I invite my students to my home for a drinking party while enjoying movies. My most favorite are the works (animations) of Makoto Shinkai. Being so much attracted to such visual entertainment is perhaps retaliation to my childhood that I spent without TV (laughter).

I'd link my students to take up anything positive and put it in practice as “their own challenge”

Currently how many students does your lab have?

A total of ten – four undergraduate and six graduate students. They are all gentle, positive, and well brought up. Indeed, they are really worth teaching. In the Faculty of Science and Technology, we have just produced our first alumni, but some of them found employment with advertising agencies and other mass media-related companies. This may be an indication that AR will find increasing application as useful media. I’d also like to point out that at Keio University researchers are very active in research activities off campus as well. It is often the case that being involved with teachers from other departments at Keio through administration meetings provides me with various stimuli. There are also many opportunities for interactions beyond academic specialties. All in all, Keio offers an ideal environment, for sure. But it should be kept in mind that students could hardly learn what they are taught at classes if they remained passive. This is why I’d like my students to take up any hobby related to their interested fields and put it in practice as “their own challenge.” Especially when it comes to the field of informatics, you are able to shape your own ideas readily with only a computer and sufficient enthusiasm. I sincerely hope that students will enjoy a meaningful student life by beginning with their interests.

Just a word from a student

Student : Whenever we students approach Prof. Sugimoto for consultation, he responds to us with very kind advice, putting himself in our shoes. Our lab is always has a friendly atmosphere, so we can communicate very frankly with each other.

(Reporter & and text writer: Madoka Tainaka)