The Power of Imagination from Keio's Faculty of Science and Technology Human-Agent Interaction and Science Fiction

The Potential of Communication Agents

Robot and AI characters that complement society

Human-Agent Interaction (HAI) is a field closely related to that of social robots. Recently, people have been finding more opportunities to come into close contact with robots of different sorts, whether in the form of pets or the kinds that serve you meals at restaurants. When they encounter these robots, humans naturally feel inclined to read their emotions, sense their intentions, and develop attachments to them. HAI is a field broadly studying any interaction between humans and artificial objects that suggests some intrinsic “humanness” or intentionality. This research is rapidly growing in importance, especially as many companies are introducing AI agents that can plan and provide customer support on their own, automate routine tasks, and analyze data.
Osawa has launched a series of research projects that explore new forms of interaction between humans and human-like agents. One prime example is an agent that was showcased in 2017, conveying emotions and information by capturing the movements of human eyes. Originally, when he was working on giving household appliances human features, Osawa thought, “Maybe by capturing the essence of human interaction and embodying that I can provide psychological support...”
He focused on the eyes, given their ability to convey emotions, and developed a device called “AgencyGlass,” which acts as a substitute for the eyes’ communicative ability (Fig. 1). The camera attachment captures the face and gaze of the person or people in front of it and automatically moves the glasses’ computer-generated eyes accordingly. Thus, when a visually impaired person wears the device, their eyes appear to move naturally, helping alleviate potential discomfort in social situations. Vibrations are also transmitted to the glasses when the eyes move, allowing the wearer to turn their head accordingly. While they would not be able to see any image, the visually impaired wearer will be able to adjust the direction of their gaze to the object of the conversation, resulting in a smoother interaction.

Fig. 1 “AgencyGlass,” glasses to anthropomorphize human
When worn by someone with visual impairment, the eyes in the glasses move naturally in response to the conversation going on as shown in the figure above. The device can be used in conversations with multiple people, allowing for smooth interactions.

One successful example of energizing a reading community is the “Book Introduction Robot,” used at an elementary school in Tsukuba City (Fig. 2). A child first enters the information of a book they want to introduce such as its title and an introduction into a computer. Then, when someone clicks on the book title on the screen, the robot introduces the book orally, complete with its own facial expressions and gestures. The ability to set the robot's facial expressions, from how it moved to its color, was a big hit. Osawa said, “I was happy to see the older children taking the initiative to teach the younger ones how to use the robot.” He calls a robot in which the agent not only provides help, but also enhances the scene itself with its presence, a “catalytic agent.”
　They are also developing a learning system for children. This system is designed to flip the script, having students learn by teaching the agent, rather than having the agent fulfill the teaching role. “These agents do not necessarily have to be smarter than humans. In fact, they just need to be a bit ‘less smart’ than humans. Our relationship with catalytic agents is one of the research themes we will explore moving forward.”

Fig. 2 Book introduction robot
A child introduces a book, and the robot presents on it on their behalf. The audience lauded the robot’s ease of use and adorable characters. The robot also helped enliven communication between children.

The second major pillar of Osawa’s research is the theme of social intelligence on the agents’ part. This research mostly pertains to the software that makes use of programming and algorithms, and tackles whether AI can understand the intentions of others and cooperate with them. One example of where AI can use this skill is a party game called Werewolf where participants talk to each other to find out who the “werewolves” hidden among them are. There has also been research on a game called Hanabi. Here, players are given a hand of cards that they cannot see and cooperate with one another by guessing what cards they have based on the other players’ hints. The key is to read the other player’s intentions, cooperate with them, or even double-cross them at times. This is the behavior that Osawa is researching how to make AI mimic.
AI has difficulty reading the people’s intentions like humans do, but if it acts in accordance with others’ personalities, it will perform better. “We need to design AIs that match human personalities,” says Osawa. “This includes AI that are bold risk-takers and those that are more cautious, quick-decision makers and ones that agonize over tiny minutiae.” He is working to improve performance through repeated gaming playtests with human subjects.

The third pillar of Osawa’s research is the application of science fiction. Agents developed to support humans may encounter problems when they are put to use in society. In fact, the people who have explored and dug into this issue the most are science fiction authors. Through his immense experience with communication of sci-fi writers, Osawa realized that it could hold the key to new idea for social agents.
Figure 3 shows a classification of different AI characters in science fiction. He classified them according to several parameters such as independence, friendliness, network size, and language ability, and used clustering and principal component analysis to find that there are four main types of AI depicted in science fiction. “Human- and machine-type agents have been around for a long time, but we can see the potential for new agents in the realm of buddy- and infrastructure-type agents, which lie somewhere in between them,” says Osawa. The Science Fiction Research & Development Center was established within Keio to serve as a base for this research.
　Osawa is also researching “Sci-fi prototyping,” a method for identifying issues through science fiction. Usually, companies do “forecasting,” which is looking at the current situation to identify future challenges. Sci-fi prototyping involves “backcasting.” This is where science fiction writers and other relevant parties such as companies think about the future and its story, then work backwards to determine the route to get there. Using this method, future challenges and ideas that were difficult to identify using conventional methods will emerge in plain sight.
　A sticking point for this research the difficulty in deciding evaluation criteria. Choosing a standard for what is “acceptable” or “human-like” is complicated. There is also a large variation in results due to differences between individuals. Still, Osawa believes that “If even one person in ten can be saved, it should be done.” Moreover, he adds, “It is difficult because we are dealing with people, but that is the most interesting part.”
We look forward to Osawa’s research for a bright future where humans and agents can thrive alongside each other.

When I was in elementary school, my friend’s older brother introduced me about notion of programming, and I noticed that I could make my own games. My dream was to create a game like Super Mario Bros., which I was totally hooked on at the time. Then, in junior high and high school, I joined a programming club and had a lot of fun showing my friends the games that I created and playing them together. It was a joy to make the characters in my games move in such a lifelike fashion. As I researched more about the subject, I learned about artificial intelligence in a book titled Shintaisei to Conpyūta (lit. Embodiment and Computers), which was a separate volume of the magazine bit. I wanted to work more in that field, so after high school I entered the Faculty of Science and Technology at Keio University.
There, I joined two student clubs: the Keio Robotics Society and the Science Fiction Society. When I was in my third year as an undergraduate and trying to decide what laboratory to join, a senior member of the Robotics Society recommended that working with Professor Imai would be a good option based on my interests, so I joined the Anzai-Imai Laboratory.

For a while, I was working with a communication robot in the lab, obtaining information from its speech and sensors and defining what its actions meant. Later, when I was a master’s student, I received an award for my research on anthropomorphized household appliances at the HAI Symposium. The appliances would give a tutorial to users. It was an unexpected discovery that when home appliances explain their own usage, measuring the user's eye movements revealed that the same speech can be understood differently depending on the character's position and settings.

My research on anthropomorphism was selected for a JSPS Research Fellowship and the “Presto” project of the National Institute of Science and Technology (JST). I continued my research as a visiting researcher at the National Institute of Informatics, a visiting researcher at the Massachusetts Institute of Technology AgeLab in the United States, a research associate at Keio University, and an assistant professor at Tsukuba University before returning to Keio in 2022, which is my current position.
I thought that anthropomorphism was the key to advancing AI research. Going forward, I wanted to do research on social intelligence in situations that involve responding to others, like having the AI cooperate with or betray you. During my time at the University of Tsukuba, I made use of the Werewolf game while collaborating with people from different research backgrounds to hone our ideas. We formed a voluntary-based group to continue this research, and its members are currently active at the forefront of various research fields.

The HAI Symposium has been held annually since its inception in 2006. From 2013, the symposium evolved into the HAI International Conference and is now held annually in Europe or Asia, taking place in countries such as Spain, South Korea, Singapore, Germany, New Zealand, Sweden, and the United Kingdom. It was even held online during the COVID-19 pandemic. This year I am one of the general chairs with the event being held in the fall at Keio's Hiyoshi campus.
The amount of investment in the US and in China towards AI research is off the charts. It is often said that Japan is lagging behind, but I believe it has a significant advantage in AI research in that it can collaborate in fields where it is strong, such as in quantum computing, materials engineering, chemistry, and biology. HAI is one such example. Japan is ahead of other countries in its research and implementation of communication robots, and its synergy with character culture and virtual agents is also a strength.

Japan has a particularly vast treasure trove of visual and interactive fiction. Manga is arguably the best in the world in terms of quality and quantity. Japanese anime and video games have also had a huge influence internationally. You can find excellent media content in places like the US, Europe, and China, but overall, when you consider the community around secondary creation, you’ll discover in Japan an environment for creative works that is truly diverse.
Just as the Comiket convention, a symbol of doujinshi (self-published print material) culture, branched off from the Japan Science Fiction Convention, Japan’s wide range of works stems from SF culture. There’s a lot that makes science fiction appealing, but one of them is that it’s the world-building that directs the stories, rather than the people themselves. Because of that, rather than being restricted to depicting people as they are today, it has the advantage of being an avenue for doing thought experiments freely.
Interviewer: How will research with SF develop in the future?
Osawa: For artificial intelligence research, we believe science fiction will be the next theme to nurture AI research. I think one challenge is figuring out how machines can support or even replace vision-building endeavors like sci-fi prototyping. To this end, we hope to discover new synergies by studying the science fiction as a genre.
Sci-fi research is being conducted in various fields; in the US there’s been much business-related research, and in China there’s been much research on brain science and psychology. However, in terms of diversity, I believe that Keio University's Science Fiction Research & Development Center, or the “SF Center,” is the best. I don't think there are many other areas where researchers and scholars in engineering, cognitive science, aesthetics, philosophy, management, and science communication can come together and conduct research allowing them to create a vision of AI by mapping it through science fiction, or evaluate the results of sci-fi prototyping.

For artificial intelligence research, we believe science fiction will be the next theme to nurture AI research. I think one challenge is figuring out how machines can support or even replace vision-building endeavors like sci-fi prototyping. To this end, we hope to discover new synergies by studying the science fiction as a genre.
Sci-fi research is being conducted in various fields; in the US there’s been much business-related research, and in China there’s been much research on brain science and psychology. However, in terms of diversity, I believe that Keio University's Science Fiction Research & Development Center, or the “SF Center,” is the best. I don't think there are many other areas where researchers and scholars in engineering, cognitive science, aesthetics, philosophy, management, and science communication can come together and conduct research allowing them to create a vision of AI by mapping it through science fiction, or evaluate the results of sci-fi prototyping.

The SF Center was recently launched in 2024. Given the history and development of science fiction in art and creative works, researchers in literature, engineering, and art are able to collaborate and explore the value of storytelling, as well as explore the future of sci-fi as a means of finding innovation in human society. We have six groups that include the AI Group and the Cognitive Science Group. We have attracted incredible talent with, for example, the head of the Literature Group being Professor Susumu Niijima from the Faculty of Economics. I believe that Keio, as a university, has the flexibility to allow those in the humanities, sciences, or engineering, to easily collaborate on themes such as science fiction. I feel that we have the framework in place to tackle new challenges.
It was the well-known Japanese sci-fi author Sakyo Komatsu who proposed the idea of having “Science Fiction Studies,” but until now there have been few places that have taken it up as an academic discipline. Now preparations are underway for this line of research at the SF Center. This is probably possible only because Keio is the best private university in Japan. Rather than eyeing what the government wants and following what everyone is doing at the time, I think an advantage of being a private school is the independence to take a chance to seek out what is needed for the next generation.

I was like them myself, and I honestly want people to be free to do what they want. In particular, I would like to tell them, “Don't be afraid to try something new. It will be more fun in the end.” Of course, it is also important to develop competence in existing fields. When you are really new, no one will recognize you at first, but in my experience, what’s really great is that if you work hard for a few years, someone will recognize you and become your ally. If, after looking deep in yourself, you are convinced that you can make a case for what you want to do, I urge you to keep going without ever giving up. I believe that Keio is a place where this is possible.

● He is very friendly. He knows a lot about the science fiction works, video games, and anime from my generation. I enjoy talking with him. He even joins events like softball tournaments. Our similar interests have made it easier for me to communicate my research, which has been helpful. He is readily available for in-depth consultation and gives crucial advice. (2nd year doctoral student)
● The scope of his research is quite broad, and I think it is amazing that he is applying his AI knowledge to so many different areas. There’s a sense of openness in his laboratory, and it is easy to come up with ideas freely. (2nd year master's student)