Editorial diagram of the ideas or workflow discussed in this article. It is a schematic, not measured data or a product screenshot.
Sheng Zha, in conversation with Yuzheng Sun. Edited account of the 2026-03-04 video, focusing on 02:00–10:42. This is a thematic article, not a verbatim transcript.
A team can run a series of experiments that mostly succeed and still learn too slowly. In this conversation about training large models, Sheng Zha explains why experimental success and research progress are not the same measure.
Model training involves enough interacting choices that money, compute, and a familiar recipe do not settle the problem. Teams must decide what to test, what to measure, which small experiments can inform larger runs, and how much confidence a published result should earn. These are questions of experimental method as much as model architecture.
Ask what an experiment will teach
An experiment expected to work with very high probability often confirms something the team already believes. An experiment expected almost certainly to fail may be similarly uninformative. Zha argues for exploring valuable directions where the outcome is meaningfully uncertain.
He uses a roughly even chance of success or failure to illustrate the idea. Either result can substantially change the team’s understanding. The point is not to impose a fifty-percent success quota on every project. It is to choose experiments for the information they can produce, including useful information from failure.
This creates a tension with organizations that demand a steady sequence of low-risk wins. Such a team may look reliable while covering very little of the unknown design space. Research needs a way to value learning before it can turn that learning into a dependable training process.
A scaling curve is an instrument, not a guarantee
Large training runs are expensive, so teams use smaller experiments to anticipate what will happen at greater scale. Scaling laws offer an empirical framework for that extrapolation. But using the framework well requires more than fitting a curve and adding compute.
Zha emphasizes interactions between choices and the possibility that a trend will fail to continue. People notice capabilities that appear at larger scale; there can also be negative surprises, where a model stops following the expected trajectory. A small run that looks healthy on one metric may hide a problem that emerges later.
This makes measurement design part of the experiment. Teams need to decide which signals besides loss are worth tracking and which assumptions the scale-up depends on. The small run is valuable because it helps reduce uncertainty about the larger one, not because small-scale success automatically validates the full design.
Read the setup before accepting the conclusion
The same discipline applies when learning from other researchers. An idea working in one setting does not establish that it works everywhere. An idea failing once does not establish that it has no useful setting.
Zha recommends looking closely at the experiments behind a paper: the setup, the conditions, and whether they resemble the problem at hand. A broad claim may be a compressed description of a narrower observation. Reading the underlying evidence gives a practitioner more room to judge where the result transfers.
The discussion also mentions basic protections such as keeping benchmark information out of training data. Without sound experimental practice, an apparent improvement can provide the wrong feedback to the team. Faster iteration then compounds a measurement error rather than a useful discovery.
The cluster belongs in the research picture
Training at scale introduces physical and systems problems as well. Zha describes synchronized computation creating sharp power demand, nodes failing, and hardware errors that can corrupt results without a clean warning. A cloud environment that works well for other workloads does not automatically remove these challenges.
This brings the conversation back to the positive iteration loop. A model team needs trustworthy experiments; trustworthy experiments depend on reliable infrastructure and meaningful measurements. The work of training a better model includes building a system in which the team can tell what it has actually learned.
The practical question for a research review is consequently more demanding than whether the latest run succeeded: what belief did the run change, how reliable is that change, and what should it make us try next?