DeepSeek V4 Pro achieves competitive frontier performance without Nvidia hardware by combining sparse Mixture of Experts activation, Multi-Head Latent Attention KV compression, and FP8 mixed-precision training on AMD Radeon Instinct and Huawei Ascend chips.
DeepSeek V4 Pro is running on hardware that Nvidia's competitors make, and it is not struggling. That is the story worth paying attention to.
For years, the working assumption in AI development was simple: serious models require Nvidia H100s or A100s. CUDA is the compute standard, and everything else is a workaround that costs performance. DeepSeek V4 Pro broke that assumption by shipping a frontier-class large language model trained and deployed primarily on AMD Radeon Instinct and Huawei Ascend hardware. This is not just a supply-chain story or a geopolitical footnote. It is a deeply architectural achievement.
Understanding how DeepSeek V4 Pro does this requires looking at three core design choices: its Mixture of Experts (MoE) topology, its Multi-Head Latent Attention (MLA) mechanism, and its FP8 mixed-precision training strategy. Together, these architectural decisions removed the hard dependency on Nvidia's proprietary CUDA software stack while dramatically cutting the memory bandwidth and compute budget required for both training and inference.
The US government's export controls on advanced semiconductors, particularly Nvidia's H100 and A100 GPUs, placed Chinese AI research institutions under significant hardware constraints. The obvious responses would have been to wait, lobby for exceptions, or source chips through third-party channels. DeepSeek's response was architectural: design a model that does not need those chips to perform at a competitive level.
Executing that strategy required rethinking assumptions at every layer of the stack. The CUDA toolkit, which Nvidia controls, is the de facto standard compute layer for essentially every major LLM training run outside China. Moving away from CUDA means writing and optimizing training kernels for AMD's ROCm stack or Huawei's CANN (Compute Architecture for Neural Networks) framework. Both frameworks are capable, but neither has the ecosystem maturity that CUDA has accumulated over 15 years of community and commercial investment.
The engineering challenge was not just porting code to a different API. It was proving that the resulting model, trained with less ideal tooling and on chips with lower peak specifications on paper, could be competitive with models trained on the best available hardware. DeepSeek V4 Pro cleared that bar and, in doing so, demonstrated that the relationship between hardware quality and model quality is less deterministic than the industry had assumed.
DeepSeek V4 Pro is a Mixture of Experts (MoE) model with approximately 671 billion total parameters, but only around 37 billion activated per forward pass. That gap between total and active parameters is the single most important number in this architecture.
| Specification | Value |
|---|---|
| Total Parameters | ~671 billion |
| Active Parameters Per Token | ~37 billion |
| Architecture Type | Sparse MoE |
| Training Precision | FP8 |
| Attention Mechanism | Multi-Head Latent Attention (MLA) |
| Context Window | 128,000 tokens |
The massive gap between total and active parameters is what lets DeepSeek V4 Pro run on hardware that would buckle under a dense 671B model. Each forward pass only engages a carefully selected subset of the model's capacity, dramatically reducing memory bandwidth requirements and compute intensity per generated token.
💡 Worth noting: Memory bandwidth, not raw FLOPS, is the primary bottleneck for LLM inference at scale. By activating only 5.5% of its parameters per token, DeepSeek V4 Pro makes every hardware platform a more viable host.
In a standard dense transformer, every parameter participates in every token prediction. A 70B dense model processes every token through all 70B parameters, every single time. In a MoE model, the network is divided into many specialized "expert" sub-networks, and a learned router selects which experts handle each token based on that token's content and context.
DeepSeek V4 Pro uses a fine-grained MoE design with both shared and routed experts. The shared experts always activate regardless of token content, handling general knowledge and common linguistic patterns. The routed experts compete for activation through a lightweight router network that adds negligible compute overhead to each forward pass. Only the winning experts actually process each token.
Nvidia's H100 dominates AI training partly because training dense large models requires extremely tight communication bandwidth between GPUs during the backward pass, and NVLink (Nvidia's proprietary GPU interconnect) outperforms alternative interconnects by a significant margin. MoE models reduce this communication volume substantially during training: only activated experts receive gradient updates, and only the activated experts' parameters need to move across the interconnect at each step.
This makes training over InfiniBand, which is hardware-agnostic, far more competitive relative to NVLink. AMD's MI300X clusters and Huawei's Ascend 910B clusters both use InfiniBand or equivalent interconnects, and both handle the MoE communication pattern adequately without paying the NVLink-absence penalty that dense models incur.
During inference, MoE models distribute expert shards across GPUs. Each GPU hosts a subset of the expert network and receives only the tokens routed to its experts. This is a naturally hardware-agnostic parallelism pattern that maps cleanly onto AMD and Ascend hardware with no fundamental disadvantage.
Standard multi-head attention (MHA) stores a key-value (KV) cache that grows proportionally with sequence length. For a 128,000-token context window, this cache becomes enormous, consuming memory that directly limits how many concurrent requests can be served on a given set of GPUs.
Nvidia's hardware edge in long-context inference is partly raw HBM capacity, with H100s shipping 80GB of HBM3. Many AMD and Ascend configurations have comparable or greater capacity per module in the latest generations, but KV cache growth at 128K contexts can push the limits of any hardware when running standard MHA at scale.
Multi-Head Latent Attention (MLA) compresses the KV cache by projecting keys and values into a low-dimensional latent space before caching them, then decompressing the latent representations on the fly during attention computation. The full-rank key and value matrices are never stored in memory at rest, only the compressed latent vectors.
The result is a KV cache that is 5 to 13 times smaller than standard MHA at equivalent context lengths. This is not a minor optimization. It is the difference between serving 128K context requests on hardware with 64GB of HBM versus requiring 320GB-plus configurations for the same workload.
💡 Practical impact: A model serving 128K context windows with MLA uses roughly the same KV cache memory as a standard model serving 10,000 to 25,000 token contexts. AMD and Ascend hardware can serve long-context requests that would otherwise require H100-class VRAM budgets.
DeepSeek's MLA implementation maintains output quality through careful design of the projection matrices. The compression is learned during training rather than applied post-hoc, so the model's representations adapt to work effectively within the compressed latent space from the ground up.
Training large models has traditionally used FP32 (32-bit floating point) or FP16 (16-bit). Lower precision reduces memory footprint and increases throughput because more numbers fit in the same memory, and operations complete faster on dedicated low-precision hardware units. FP8 (8-bit) pushes this further, using only 8 bits per number.
The challenge with lower precision is numerical stability. During training, very small gradients can underflow to zero, and very large activations can overflow to infinity. Both pathologies corrupt training. Getting FP8 stable across hundreds of billions of parameters is a genuinely difficult engineering problem, one that most research teams avoided for years.
DeepSeek's FP8 training strategy uses a mixed-precision scheme in which sensitive operations, specifically gradient accumulation and master weight storage, remain in higher precision (BF16 or FP32), while the computationally dominant matrix multiplications run in FP8. The team developed custom dynamic scaling strategies that adapt based on observed gradient statistics throughout training, preventing the numerical instabilities that plagued earlier FP8 attempts at large scale.
This is significant for hardware independence for three distinct reasons:
The compounding effect across all three innovations is what matters: MoE sparse activation reduces compute per token by 18x, MLA compresses the memory footprint for long contexts by 5 to 13 times, and FP8 training halves the memory and compute cost of each training step. Together, they shift the efficient operating point of the model to hardware that would otherwise be wholly inadequate for a 671B parameter frontier model.
The AMD Radeon Instinct MI300X is not a second-tier chip. It ships with 192GB of HBM3 per module, more than double the H100's 80GB. For inference workloads where KV cache memory is the primary limiting factor, the MI300X's memory capacity is an advantage over H100 in specific configurations.
What AMD lacks is CUDA. ROCm, AMD's compute stack, is mature enough for inference and increasingly capable for training, but has historically required additional engineering effort to match Nvidia's turnkey developer experience. DeepSeek's team invested in that work directly, building training kernels and optimized attention operators specifically for ROCm's programming model.
The Huawei Ascend 910B delivers approximately 256 TFLOPS at FP16, significantly below the H100's 989 TFLOPS on paper. It runs on Huawei's CANN framework rather than CUDA or ROCm, meaning the toolchain requires separate optimization work. DeepSeek trained substantial portions of earlier versions and V4 Pro's lineage on Ascend 910B clusters.
The MoE plus MLA plus FP8 stack directly mitigates the Ascend's hardware limitations. A chip with lower memory bandwidth and peak FLOPS becomes far more competitive when the model it runs only activates 5.5% of its parameters per token and compresses its attention cache by 5 to 13 times. The hardware gap on paper is much larger than the performance gap in practice.
| Hardware | VRAM | Peak FP16 TFLOPS | Compute Stack |
|---|---|---|---|
| Nvidia H100 | 80GB HBM3 | 989 | CUDA (mature) |
| AMD MI300X | 192GB HBM3 | 1,307 | ROCm (developing) |
| Huawei Ascend 910B | 64GB HBM2e | 256 | CANN (proprietary) |
H100 clusters rent at $2.00 to $4.50 per GPU-hour depending on provider and region. AMD MI300X clusters typically run 20 to 40 percent cheaper in equivalent configurations. Ascend-based cloud compute in China is lower still. When training runs execute competitively on these platforms, the cost per training token drops in a meaningful way.
DeepSeek reported training the V3 predecessor model for approximately $5.6 million in compute costs, a fraction of what comparable dense models from Western labs cost at the same capability tier. The architectural choices described above are the primary explanation.
DeepSeek releases its model weights publicly. This means any organization with access to AMD or Ascend hardware, or even consumer GPU clusters running quantized versions, can benefit from frontier-class model capability without Nvidia allocation, export licensing, or expensive proprietary cloud quotas.
The combined effect of architectural efficiency and open release is a meaningful shift in who can access and deploy competitive AI. Organizations that previously could not afford H100 access are running DeepSeek models on hardware they already own, at inference quality that was simply not accessible to them a year earlier.
The export control strategy surrounding Nvidia chips assumed that cutting off access to advanced semiconductors would function as a hard ceiling on AI development outside targeted countries. DeepSeek V4 Pro is concrete evidence that architectural innovation can partially offset hardware constraints in ways that alter the effective ceiling substantially. The model is not identical to what unlimited H100 access would produce, but it is competitive enough to matter across a wide range of real workloads.
This changes the calculus for chip policy, for AI competition strategy, and for what "access to frontier AI" practically means to organizations without massive cloud budgets or preferred vendor relationships with Nvidia.
PicassoIA hosts multiple DeepSeek models in its Large Language Models collection, giving you immediate access without managing infrastructure or provisioning hardware:
PicassoIA also provides access to Qwen3 235B A22B Instruct, Llama 4 Maverick Instruct, Claude Opus 4.7, GPT 5, and Kimi K2 Instruct in the same interface, letting you compare outputs across the frontier without managing multiple accounts or API keys.
The architectural story behind DeepSeek V4 Pro is that efficient design can substitute for brute-force hardware investment. That same principle applies to how you work with AI: you do not always need the largest model on the most expensive hardware. You need the right model, applied to the right problem, with the right prompt.
Pull up DeepSeek R1 or DeepSeek V3.1 on PicassoIA, give it a task that matters to your work, and see what hardware-independent frontier AI actually delivers. The architecture that made it possible without Nvidia GPUs is, in this case, also what makes it broadly accessible to you.
