A graduate student visits Prof. NISHIMOTO’s Laboratory Decoding the Brain 2021.11.25

What is "Decoding the Brain?" To find out, we followed graduate student, Ryuon HIGUCHI (Laboratory of KOKORO-Biology Group) to the Nishimoto lab.

HIGUCHI Ryuon (D2): Professor NISHIMOTO, what motivated you to become a researcher?

Prof. NISHIMOTO Shinji: I think it was about the time I became interested in data processing machines and began to wonder how my own brain worked when I was a high school student. What intrigued me was that living things and machines basically process information the same way, though the brains of living things are far more complex. With that idea in mind, I entered the School of Engineering Science and began my research into the visual cortex of the brain under Prof. OHZAWA Izumi (retired March 2021). After obtaining a degree, I spent 7 years at the University of California, Berkeley, to undergo my post-doctorate research on visual processing in the brain, then returned to Japan when CiNet was established.

HIGUCHI: I see. I also joined the School of Engineering Science because of my interest in robots and computers. However, because I gradually developed an interest in the function and structure of the brains of living things, I knocked on the door to Prof. Yagi’s neuroscience lab, which researches ways to understand the mind. I am now studying the cell adhesion proteins, called protocadherin, expressed in mouse nerve cells there. Dare I say that our research in brain science is on a similar wavelength? Seriously, though, could you tell me about your interest in using human subjects for your research in neuroscience?

Prof. NISHIMOTO: I mainly use human subjects because I want to know how the human brain works. When I was in graduate school under Prof. Ozawa, I researched information represented in neuronal units using animal models. At that time, I became more interested in higher-order visual functions, so I shifted my focus toward human brains. I am conducting my research to get closer to understanding how information is processed in the human brain while also using the data from the results of experiments on the nervous systems of animal models as a reference since all neurons are basic units of neural information processing.

HIGUCHI: One of the major challenges in the lab I belong to, the Yagi Lab, is asking what the “mind” means for the brain science field. We assume that even with the diversification of genes, the uniqueness of each neuron may contribute to the creation of each mind. What are you specifically aiming for in your research?

Prof. NISHIMOTO: Ultimately, we want to elucidate the brain’s entire range of information processing functions that support “human life and living.” While we aren’t nearly there yet, especially regarding visual functions, our mid-to-long-term theme is to use higher brain cognitive functions as a foothold for further research.

HIGUCHI: Regarding your most recent research, I’ve heard that it is becoming possible to decode brain activity when viewing an image and being able to reproduce that… can this also be done in animal models?

Prof. NISHIMOTO: Basically, yes. We use an MRI to measure brain activity in human subjects, but for animal models, we can use more powerful methods such as calcium imaging, which allow us to make more detailed recordings. Although using an MRI is advantageous when decoding an entire brain, there are many methods that can only be used with animal models to view the minute details of the brain.

HIGUCHI: It will be very interesting if we could visualize what non-human animals see. Think of what kind of applicable technology and social contributions can be made if we can capture that!

Prof. NISHIMOTO: Right. We are also working toward social implementations based on human research, such as analyzing and extracting visual information from brain activity and also analyzing the meaning and impressions of an image. For example, we have patented analysis technology that estimates how people perceive a particular image or video, such as a television commercial. This tech is now being used commercially by companies.

HIGUCHI: It’s amazing that this technology is already being applied.

FBS Staff: Could you explain a bit about your research into practical communication without language?

Prof. NISHIMOTO: It goes a little beyond that. To put it in simpler terms, "language" is an important tool for quantitatively dealing with people’s higher cognitive functions. Most animals don’t have a complex language system to explain how they think and behave, but people do. Being able to utilize this is an important part of learning how the brain works. That being said, language is only a small part of conveying our thoughts. Information that cannot be verbalized, such as sensations from smells and images to higher-level content like ideas and concepts, are hidden deep within the brain. If that information can be accessed directly, a richer form of communication without language could be in our future. Wouldn’t it be great to convey things exactly as we imagine them?

FBS Staff: Could you elaborate?

Prof. NISHIMOTO: Expressing such using language can be challenging, but it is true that the information we think and feel exists in our minds. Currently, there are limited ways of getting that information out, but in the future, I think powerful communication tools could be getting more direct input and output by recording and stimulating brain activity.

HIGUCHI: You mean to have communication facilitated by a new device?

Prof. NISHIMOTO: Precisely. I think we might be moving in the direction of applicable possibilities; right now, we are working on basic research that could be a prerequisite for such. Another example could be our research into decoding models for smells similar to the one used to decode vision.

FBS Staff: There are many types of smells—has there been any progress in understanding the molecular mechanisms involved?

Prof. NISHIMOTO: Yes—though there are hundreds of different types of smell receptors, we are making progress in analyzing genes and protein expressions of human olfactory receptors. As the differences between individuals and fluctuations within each person are greater when comparing smell to sight, dealing with the sense of smell presents quite a challenge; however, we are conducting joint research with professors specializing in olfaction at the University of Tokyo and other institutions.

FBS Staff: It would be fantastic to be able to recreate what can be envisioned, but I have the impression that since the sense of smell is affected by so many conditions it will have a lot of parameters, such as a person’s physical condition and individual differences.

Prof. NISHIMOTO: That’s what makes this research so interesting to conduct. For example, manga artists and novelists must have a rich imagination. Wouldn’t it be amazing to study their brain activity when they are working?

FBS Staff: I’ve heard that many people in that kind of work see their stories in their minds as they write them down, so it probably would be interesting to study them as special cases from a different perspective and use a completely different approach from that in molecular biology.

Prof. NISHIMOTO: The special conditions of each subject are actually easy to deal with because our brain models are created specifically for each individual person. For example, some of my research included examining how the brains of advanced and beginner programmers differ.

HIGUCHI: That’s quite different to molecular biology, where we tend to get caught up in how to show the significant differences between the general control (wild type) and gene-disrupted phenotypes. We have a mind to automatically try and reduce the data variation (scattering) of a control, so actually focusing on individual variations instead is refreshing.

FBS Staff: It seems there are many ways to approach the process and explore which aspects can be cut out. The ideas and potential developments are endless!

Prof. NISHIMOTO: It’s hard to gather manga artists to be test subjects, however.

HIGUCHI: This may be off topic, but I’ve just thought of a social implementation service that can see whether an individual is suited for a job or has room for growth by checking the process in which they acquire skills and expertise. What do you think?

Prof. NISHIMOTO: That’s an idea. Personally, I think we need to be aware of the risks involved, but developing such technology may be possible. There could be potential applications for a device or method that can show an individual’s character or tendencies by looking at their brain, something of personal genomics alongside personal brainomics.

FBS Staff: There could be many social benefits if that kind of big data could be organized.

Prof. NISHIMOTO: It may also reveal idiosyncrasies in learning and mastery as well as the approaches that are best suited for each individual.

FBS Staff: Out of all of your current ongoing research, what do feel is the most important?

Prof. NISHIMOTO: Above all, exciting and fun research. My primary interest is to fundamentally understand how the brain processes information in natural, everyday activities. Most conventional neuroscience and research focus on certain hypotheses and use special stimuli or tasks to test those hypotheses, but our daily lives are a complex combination of many more things. I would like to study that more multilaterally. I don't know if setting up conditions and approaches taken from a perspective focused on a person’s daily life is a shortcut or not, but I believe this method can provide an effective perspective that is complementary to conventional research.

FBS Staff: Switching topics, can you tell us what kind of factors or prior knowledge would make it easier for students interested in research like yours to pursue it?

Prof. NISHIMOTO: I often use programming and machine learning systems, so it would be good to have fundamental knowledge in that as well as in neuroscience. Though we spend altogether about 10 to 20% of our time on experiments, more than 80% is spent on analysis.

HIGUCHI: Do you spend time discussing the results of the analysis with other lab members?

Prof. NISHIMOTO: Yes. The data is complex so discussing how to interpret it is necessary.

HIGUCHI: Though I think it depends on how data is analyzed, in your research field, how do you control clear data obtained through MRI analysis performed under certain conditions?

Prof. NISHIMOTO: The programs used for basic pre-processing share a common language. Though we follow the standard methodology for analysis in the machine learning field, getting the minute details are a matter of trial and error for each research theme.

FBS Staff: Do you have any messages for students interested in your research field? If you don’t mind, could you also tell us your research outlook?

Prof. NISHIMOTO: I’d love to have interested students come and talk to me. There are so many research themes I am still working on. Now, we are mainly conducting research on human subjects utilizing MRI scanning, and in the future, we will be conducting joint research using animal models as a reference.

Under the framework of neuroscience, I felt that Prof. NISHIMOTO’s research is not that far off from my own, but the subjects, the way experiments are constructed, and the daily schedule was completely different. Our type of research tends to be lumped together as “brain science,” but through our talk, I was able to again realize that there are many different approaches depending on the subjects and purposes of experiments. Additionally, I was impressed by his focus on everyday functions as well as the way his experiments were set up while keeping in mind the natural state of things to better understand brain functions. What really left an impression was how different our approaches are. In my research, I use a less-than-natural approach, as is common in many molecular biological methods that use mutants for validation, to discover abnormalities in the neural networks of knockout mice and try to prove the involvement of cyclic formations created from specific molecules (in the Yagi lab, these are clustered protocadherins). Today, there were so many things I didn’t know, and I gained new perspectives and ideas speaking with Prof. NISHIMOTO. I would like to find ways to use all the things I learned today in the future.

HIGUCHI Ryuon

CiNet, a joint research center supported by NICT, Osaka University, and ATR, houses two 3T (tesla) magnets and one 7T magnet for human structural, functional, and spectroscopic MRI scans. The MRI machines are operated by specialized technical staff, allowing researchers to concentrate on setting up experiments. In addition to acquiring cross-sectional (structural) images of the brain, the MRI devices are also capable of capturing the minute details of brain activity in response to stimuli (functional MRI = fMRI). According to test subjects, after a very loud sound from inside the tube, they feel a strange sensation when the machine scans them. Subjects are strapped into the device, and they can receive visual stimuli and communicate with the lab technicians through a microphone. If you’re claustrophobic, this might not be the machine for you. As the device uses a strong magnet, wearing metal is strictly prohibited. Special clothing must be used when being scanned, and subjects must be extremely careful they aren’t wearing underwear woven with metal, as is common in recent years. Magnetic materials have been found in unexpected places, such as in heat-retaining underwear and even sunscreen. In a field where delicate and precise equipment is handled, safety precautions and instructions are provided.