Why LLM Inference Is Inconsistent: Non-determinism

Have you ever wondered why ChatGPT gives different answers even when you ask the same question multiple times? This is not just randomness from "sampling" - even with temperature set to 0, theoretically deterministic LLM APIs are still non-deterministic in practice! Many believe this is due to the "concurrency + floating-point" hypothesis caused by non-associativity of floating-point operations and parallel execution, but this is not the full picture. In this video, we will dive deep into the real causes of non-determinism in Large Language Model (LLM) inference and share how to solve this problem to achieve reproducible results!