Augmented Reality (AR)
from Keio's Faculty of Science and Technology Human-computer interaction
based on spatial information

Styles of communication being transformed by augmented reality

Recent years have seen application of Augmented Reality becoming increasingly popular along with the keyword “AR” in a wide range of fields such as smart phone applications, games, and advertising/sales promotional tools. Here, let’s have a look at several concrete examples to overview the initiatives being developed by Assistant Professor Maki Sugimoto who pioneers the development of new styles of AR and pursues research into nextgeneration communication tools.

Augmented Reality (AR) is the research field Dr. Sugimoto of the Department of Information and Computer Science is currently dedicated to. AR is a field deriving from Virtual Reality (VR) with which we are familiar in SF movies and games.
“While VR studies mainly aim to create sensory information that immerses users into a virtual environment built within the computer, AR aims to augment sensory information by superposing information from the computer upon that of our actual living environment (see Fig. 1 (Right)). In other words, by superposing a computer-created virtual environment upon the actual world in real time, AR allows us to access the virtual environment in a more natural way,” as Dr. Sugimoto outlines his field of study.
A recent momentum that aroused interest in AR among the general public was the impact of the animation “Coil – A Circle of Children” televised on NHK (Japan Broadcasting Corporation) in 2007. The animation depicts scenes of children experiencing AR in their daily lives using a compact HMD (Head Mounted Display).
Meanwhile, “ARToolKit” software, developed by Professor Hirokazu Kato of Nara Institute of Science and Technology, is also worthy of special mention as having contributed to the spread of AR. It can make a virtual character or object appear on the screen after camerascanning a pattern printed on a sheet of paper. Given its availability as an open source, many character animations produced by means of ARToolKit have been contributed to video-sharing websites, while video games and advertising/ sales promotional tools based on similar technologies are also spreading lately.
“For example, there is a tool capable of drawing a computer graphics model of a vehicle’s body on the screen if you hold the vehicle’s promotional pamphlet over the camera, allowing you to confirm the vehicle’s running state or internal structure. Competitive games combining a card game and CG are also rising in popularity. Thus, AR has now become a vital technology indispensable to advertising and entertainment.”

You may talk about “AR” broadly, but its techniques vary widely. Currently most popular is a system known as “video see-through.” Like the above-mentioned ARToolKit, the video see-through system realizes AR by superposing on the display the information created on the computer upon images taken by a camera. Meanwhile, the system known as “optical seethrough” uses a transmissive display and is equipped with a half mirror and a see-through-type HMD. It is capable of superposing CG on real images that are optically visible. Furthermore, the system known as “spatial AR” can transform a real environment by projecting virtual images onto the real environment. The “Projection Mapping” event held in the newly renovated Tokyo Station Building in 2012 is a typical example of spatial AR. 　“AR techniques are also very useful in work support. Application of AR to work support has already begun – for example, a patient’s excision area can be clearly indicated by superposing a prephotographed CT or MRI image onto his or her affected part at the time of surgical operation. Thanks to the recent spread of smart phones and personal digital assistants (PDA), the number of AR-based services easily available to general consumers is increasing. “Layar” combining location data and a camera is a fine example” 　Dr. Sugimoto continues, “With the recent spread of AR, it is now perceived as a broader technology that can not only augment sensory information on the display screen but can also directly transform the real environment itself. This makes AR studies all the more interesting and worthwhile.”

Dr. Sugimoto is currently working on the development of a dynamic type of spatial AR, in which visual information is projected onto the real environment.
He adds, “By using devices such as projectors and robots, I’m trying to projec t information contained in the computer directly onto a real environment.”
“For example, in the ‘Augmented Coliseum’ competitive shooting game using vehicle-type robots, robot movements are controlled by identifying accurate locations and attitudes of the robots mounted with optical sensors that can read fiducial images projected from the projector (see Fig. 2(Above)). By dynamically projecting a so-to-speak “optical ruler,” the system realizes a highly accurate AR environment.”
The greatest merit of this system is that it is possible to identify highly accurate locations of the robots only by using information read by the five robot-mounted sensors. Compared with common image sensors, the system allows computer computational complexity to be reduced significantly while also making it possible to move the robots in real time.
Dr. Sugimoto is also proceeding with a research project to achieve real-time synchronization, in an AR environment, of movements of a robot at hand with those of a robot in a remote location.
“Suppose you are doing a desk study to determine the arrangement of things inside a building. In this case, if you moved an object in front of you, a similar object in front of a person in a remote location would move in synchronization. This surely will enhance your sense of reality.”
In this relation, he is also engaged in the development of a communication tool that allows an optical sensor mounted on a compact bear-shaped robot to read information on the computer screen and make various gestures in accordance with the information (see Fig. 1(Left)).
“What I want to realize by using augmented reality is flexible interface between humans and information. By presenting information with spatiality while setting much value on physical nature in the real environment, I would like to give concrete shape to abstracted information, thus contributing to smoother communication among people,” concludes Dr. Sugimoto.

Both of my parents graduated from universities of fine arts, and they were working in the same industrial design firm in Tokyo. One day, they were compelled to enjoy a self-sufficient life in the countryside and moved from Tokyo to Nagano. Thanks to their education policy of raising a child unrestrictedly in an environment of superb nature, I had lived a life without TV up to the age of elementary school upper grades. Indeed, as a small boy I spent everyday running about fields and hills.

My mother was then teaching at a cram school in Nagano, so she earnestly tried to teach me English and mathematics. But I was always slipping out of her hands because I didn’t like to be taught face-to-face by my own mother (laughter).
Perhaps as retaliation to the life without TV, I came to take a special interest in computers when I was a junior high student. Toward the end of elementary school, I encountered computers for the first time in life when I visited the home of my cousin. There I found and touched an MSX, a computer for beginners. Since then, even before owning a computer of my own, I bought a computer magazine every month and was absorbed in reading them.

As an undergraduate freshman, I took part in the International Collegiate Virtual Reality Contest (IVRC), an event presided over by the then University of Tokyo Professor Susumu Tachi (now Special Research Professor of Keio University, and Professor Emeritus of the University of Tokyo). This turned out to be the first opportunity for me to think about a career as a researcher.
The seniors in the circle that I joined upon university admission were the contest winner for the preceding year, whom aroused my interest. What my circle seniors developed then was a device called “Virtual Bobsleigh.” It was just before the opening of the Nagano Winter Olympics. They made a model bobsleigh, with which one could experience virtual bobsledding. To tell the truth, my seniors didn’t want to participate in that year’s contest because they had overworked themselves in the contest the preceding year. But I persistently persuaded them and could somehow make an entry as a team.
My first production as a freshman was a tele-existence system that creates a sense of self-projection to a target object placed in a water tank. Though it was the first ever challenge for me, we were honored with an encouragement award of The Virtual Reality Society of Japan. From the second year on, we also made entries with the help of many friends as well as a professor who became the supervisor of my graduation thesis. In the third year, our team created a system that allowed one to walk around freely inside a computer memory unit’s folder structure expressed as a 3D environment. This system won both the encouragement award and technical award. The technical award was worthy of special mention as it had been monopolized over the years by the Tokyo Institute of Technology’s Robotics Study Society, famous for their achievements in the development of excellent systems. I was elated when our team won this award.
Through entries in this contest, I was blessed with opportunities to work with front-line researchers, such as Professor Taro Maeda and Professor Hideyuki Ando (both currently at Osaka University) who became colleagues after completion of my master’s course, and Professor Masahiko Inami (now at Keio University) who was my mentor of my doctoral course. It was a truly valuable experience, awakening me to the excitement of studies.
In my doctoral studies, it was particular significant that I could help develop the basic approach of measuring optical device locations in real time by using the intensity gradient index (index for density of light), which I devised originally. During that time, I was working with Professor Inami in research into optical measurement via space-time division multiplex. This achievement was favorably recognized by SIGGRAPH Emerging Technologies and other forefront academic societies, which in turn gave me confidence and further fueled my enthusiasm to continue to follow a researcher’s career.

I turned to the event’s administrative side when I was a senior, and now I’m supporting it as an executive committee member. This year we celebrate the 20th milestone for the annual IVRC event. Over the past two decades, entry contents have become very diverse in pace with the development of technologies. The event has also produced a great number of VR and AR researchers including myself. Since it is a very challenging and meaningful opportunity, I’m recommending my lab students to actively participate in IVRC. I’d like to see continued participation of as many prospective researchers as possible in the future.
I also had the experience of designing the IVRC webpage though I’m not responsible for it now. I undertook the webpage design because I enjoy CG design as a hobby and to get away from research work. During my student days, I even spent my living expenses worth several months to buy very expensive 3D CG software. Back in those days, it was a decision as desperate as making a leap in the dark. I am very envious of today’s students because at educational institutions today, it is possible for students to take advantage of a variety of highly functional software programs for free.
The fact is I’m making effective use of CG, which I have created as a pastime, for presentation materials and web contents. So this pursuit combines utility and a hobby. The other day, I happened to find in NASA’s website a CG model of the “Hinode” (Solar-B) satellite that I created for the National Astronomical Observatory of Japan when I was a doctoral student. I am very happy to see my own CG works being enjoyed and used by many people.
Recently, my lab introduced a “3D printer” (3D shaping equipment) as an application of CG design. It is capable of making fixtures for devices that we use for research on the spot by designing CG models. I’m extremely delighted that my hobby of CG design is actually useful in our studies.