Yokota Lab

AboutAbout the lab

Powering large-scale learning with supercomputers.

The Yokota Lab studies how large-scale distributed parallel computing on supercomputers can be applied to a wide range of fields.
The neural networks used in deep learning grow larger every year, and are reaching a point where only supercomputers can run the computation.
Running existing frameworks as they are does not improve speed or accuracy, so the lab solves, one by one, the challenges specific to large-scale computing.
Solving today's supercomputer challenges also means solving in advance the challenges that will arise on ordinary local compute servers ten years from now.

ResearchResearch themes

The Yokota Lab uses one of the world's largest supercomputers to study large-scale deep learning and large-scale matrix computation. The lab also works on algorithms that reduce computation, memory use, and I/O.

*Illustrative image

Pretraining large language models↗

Together with the Okazaki Lab and the National Institute of Advanced Industrial Science and Technology (AIST), the lab develops the large language model Swallow. Because training from scratch requires enormous computing resources, the lab takes the approach of continual pretraining centered on Japanese, starting from publicly available models trained mainly on English.

*Illustrative image

Sparsity and reasoning in Mixture-of-Experts models↗

The lab investigated how the sparsity of a Mixture-of-Experts (MoE) model affects its memorization ability and its reasoning ability. By training a family of models under fixed computing resources, the work shows that for reasoning, the balance between active computation and tokens per parameter matters.

Linking images and words in fine detail↗

This is a pretraining framework that comprehensively optimizes the correspondence between image regions and phrases in a sentence, using a loss defined between the power set of the regions and the syntax tree. The lab also introduced a method to curb the growth in computation caused by combinatorial explosion.

*Illustrative image

Making language model computation lighter↗

The gated linear units used in large language models have separate weight matrices for the gate and the value, which doubles memory reads. The lab developed a new scheme that splits shared weights between the two using masks, along with fast kernels for it.

VoicesStudent voices

Ratings and reviews reflect the experiences of individual current or former students, not the lab's official views. Whether a lab has reviews, or how many, says nothing about the lab itself.

Students who take part in an interview may receive a thank-you gift worth ¥1,000, regardless of what they say.