AMD Instinct MI300X: 192 GB HBM3 and AMD's Bet on Large AI Models

The key features of AMD Instinct MI300X are 192 GB of HBM3 memory and a memory bandwidth of 5.3 TB/s. This is a server accelerator designed for large AI models and HPC, where memory volume and speed, matrix computations, and high FP64 performance are crucial.

The MI300X has become the first AMD accelerator to seriously compete with the NVIDIA H100 in large AI systems. However, its advantages do not manifest equally across all workloads.

CDNA 3: Architecture for AI and HPC

At the core of the MI300X is the CDNA 3 architecture - AMD's compute architecture for data centers. It is not designed for gaming graphics and is optimized for matrix operations, FP64, FP8/BF16, HBM, and scaling multiple accelerators.

The MI300X features 304 Compute Units, 1216 Matrix Cores, and 19,456 stream processors. The peak frequency reaches 2.1 GHz. Eight compute XCDs are combined into a single module, with a total transistor count of 153 billion.

For different computation formats, AMD specifies the following peak performance metrics:

It's not appropriate to directly compare these figures with gaming GPUs: the MI300X is aimed at AI and HPC rather than rasterization and gaming workloads.

192 GB HBM3 - The Main Advantage of MI300X

The accelerator features 192 GB of HBM3 on an 8192-bit bus with a bandwidth of 5.3 TB/s. The volume of the Infinity Cache is 256 MB.

In terms of memory volume and bandwidth, the MI300X surpasses the H100 and H200:

For large language models, the amount of HBM affects how many accelerators are needed to store the model weights and intermediate data. The fewer GPUs needed to accommodate the model, the less data must be transferred between accelerators during inference.

In MLPerf Inference v4.1, a single MI300X was able to host the Llama 2 70B entirely in the memory of one accelerator. In a configuration with eight GPUs, the MI300X results in Server and Offline modes were close to those of a system with eight H100s.

Performance on Large LLMs

For smaller-sized models, a large memory volume alone does not guarantee superiority.

In AMD's published tests using Llama 3.1, systems with eight MI300X and eight H100 were compared:

For Llama 3.1 70B, the H100 is slightly faster or shows nearly equivalent results. For Llama 3.1 405B, the MI300X takes the lead, particularly with longer generations.

These results were published by AMD, so they should be viewed as comparisons of well-optimized configurations rather than as an independent universal ranking.

The advantage of the MI300X depends on the workload type. The more performance is constrained by memory, the more pronounced the advantage of the 192 GB HBM3 and 5.3 TB/s bandwidth becomes.

High FP64 Performance

The MI300X is designed not only for AI but also for HPC.

The peak FP64 performance reaches 81.7 TFLOPS for vector operations and 163.4 TFLOPS for matrix operations.

Such performance is required in computational fluid dynamics, climate and physical modeling, molecular dynamics, geophysics, and other scientific calculations.

For HPC, high FP64 speed is one of the architectural priorities of the MI300X, rather than a secondary feature.

Eight MI300X Equals 1.5 TB HBM3

In a system with eight MI300X, 1,536 GB of HBM3 is available, which is approximately 1.5 TB of memory directly on the accelerators.

This volume allows for large models to fit inside a single server instead of being distributed across several nodes.

The MI300X is produced in the OAM format and is installed on Universal Base Board server platforms. Infinity Fabric is used between the accelerators.

Each MI300X is equipped with seven Infinity Fabric channels, each with a bandwidth of up to 128 GB/s. The system connects to the processor via PCIe 5.0 x16.

Up to 750 W per Accelerator

The maximum TBP of the MI300X is 750 W.

The OAM form factor and this level of power consumption rule out using MI300X in typical workstations. The module does not have its own active cooling - the required airflow is provided by the server platform.

Eight MI300X can lead to 6 kW of consumption just for the accelerators, excluding the CPU, memory, storage, and network equipment.

ROCm - The Main Limitation of MI300X

While the hardware of the MI300X is competitive, the software ecosystem is equally important for a server accelerator.

AMD is developing ROCm, which supports PyTorch, TensorFlow, JAX, Triton, and other tools. With the arrival of the MI300X, the company has expanded support for modern AI frameworks and optimizations for large models.

CUDA remains one of NVIDIA's strongest advantages. Many libraries and corporate computing stacks were originally built for it.

Therefore, choosing between the MI300X and the H100 cannot be reduced to just TFLOPS or memory volume. If the workload is already running on ROCm, hardware differences become more significant - primarily the volume of HBM and its bandwidth.

The Hidden Context of the Lineup

With the MI300 generation, the Instinct lineup has noticeably shifted from predominantly HPC workloads to generative AI.

The MI100 was the first AMD accelerator on CDNA. The MI200 expanded the HPC direction and was used in exaflops systems.

In the MI300, the CDNA 3 architecture received FP8, more powerful Matrix Cores, and a stronger focus on neural networks.

The next MI325X retained CDNA 3 but received more memory and higher bandwidth. The shift to a new architecture occurred with the MI350 Series using CDNA 4.

The MI300X became the first Instinct that AMD began to position directly against NVIDIA's leading AI accelerators.

Conclusion

The main advantage of the MI300X is 192 GB of HBM3 with a bandwidth of 5.3 TB/s. Together with high FP64 and matrix computation performance, this makes the accelerator particularly suitable for large LLMs and HPC.

For medium-sized LLMs, the H100 may be faster or provide comparable results. On larger models, the MI300X's advantages in memory volume and bandwidth become more noticeable.

The main limitation remains in the software: CUDA still gives NVIDIA a significant advantage due to the maturity of its ecosystem.

The MI300X is no longer the newest accelerator from AMD, but it was the first model that could be seriously placed next to the H100 in large AI systems.