roobriefIT news for developers

Hardware

Local LLM Hardware From NVIDIA, AMD and Intel: Look Past Memory Size to Bandwidth

When choosing hardware for local LLMs, look at memory bandwidth, not just memory capacity.

Cloud API bills and data-security concerns are pushing more developers to run AI models on their own machines, and NVIDIA, AMD and Intel are each offering developer hardware in a different form. Put the specs side by side and one thing stands out: two products can both have 128 GB of memory, yet differ a lot in how fast they can read it (bandwidth), and that speed largely decides how fast the AI produces text.

A few terms first. VRAM is memory dedicated to a graphics card. Unified memory means the CPU and GPU share one pool of memory. Bandwidth is how fast memory can move data (GB/s). Inference is using a finished model to produce answers. An LLM (large language model) is the kind of AI behind chatbots like ChatGPT.

At a glance

Product Memory Bandwidth Reported price
NVIDIA DGX Spark (GB10) 128 GB unified 273 GB/s $4,699 (raised in February)
NVIDIA RTX PRO 6000 Blackwell 96 GB VRAM 1,792 GB/s $13,250 on NVIDIA’s marketplace (out of stock; listing date unknown); some reports say $16,000 recently
AMD Ryzen AI Halo (Ryzen AI Max+ 395) 128 GB unified 256 GB/s $3,999 (sold exclusively at Micro Center)
Intel Arc Pro B60 / B70 24 GB / 32 GB (GDDR6) 456 GB/s / 608 GB/s B70 about $1,099 (partner post), B60 not found

Prices move fast and differ by source. Check the official site and retailers for current prices and stock before buying.

NVIDIA: two very different options

  • DGX Spark puts 128 GB of unified memory in a small box. It is good for loading big models whole, but its 273 GB/s of bandwidth is criticized as lower than Apple’s M4 Max (about 400 GB/s). Its price reportedly rose from $3,999 to $4,699 in February because of memory supply constraints.
  • RTX PRO 6000 Blackwell is a 96 GB workstation GPU that plugs into a regular PC. Its bandwidth is more than six times the Spark’s, so the same model runs much faster, but it costs far more. This class of card has no fixed list price, so prices vary by seller and have climbed sharply recently.

AMD: 128 GB unified memory, and setup is the catch

  • At CES (the big consumer tech show held each January in Las Vegas) this year, AMD introduced Ryzen AI Halo, a developer mini PC with 128 GB of unified memory and ROCm (software that runs AI calculations on AMD GPUs) preinstalled. AMD says it is sold exclusively at Micro Center, with pre-orders starting in June.
  • PCs from other makers use the same chip (Ryzen AI Max+ 395). AMD says the memory given to the GPU (Variable Graphics Memory) comes preconfigured, but you have to raise it yourself to run larger models.
  • Community reports often say Vulkan (a standard way to use graphics cards for general computing) works better than ROCm for models of 70B (70 billion parameters, the values a model learns; more usually means smarter but bigger) and up. That comes from informal blog benchmarks, so it is best to compare on your own workload.

Intel: big memory is for servers, developer parts are experimental

  • Intel’s large-memory part, Crescent Island, is a data-center inference GPU with 160 GB of LPDDR5X (a fast, low-power memory standard). Intel expects to send test units to customers in the second half of this year and to sell it widely in 2027, so it is not something an individual developer will buy.
  • What individuals can use is the Arc Pro B60 (24 GB) and B70 (32 GB) cards. Users report running them with local AI programs such as llama.cpp and vLLM (through SYCL, Intel’s GPU computing method, or Vulkan), along with driver errors and out-of-memory problems.
  • Some coverage says Intel is targeting developers with large unified-memory chips, but that isn’t true today. What developers can get now are Arc Pro cards, and on the software side OpenVINO (Intel’s AI runtime) has only a preview connection to llama.cpp.

What could change for developers

What follows is our estimate and advice based on the reports.

  • Model size first, then speed. If the model doesn’t fit in memory, speed is irrelevant. Check capacity against the model you want to run, then compare bandwidth.
  • Same memory size, different speed. Even among 128 GB unified-memory products, and against a 96 GB VRAM card, the speed at which text appears can differ a lot.
  • Check software support. CUDA (NVIDIA’s GPU computing toolkit) is the easiest; reports for ROCm and Intel (SYCL, OpenVINO) often mention setup effort. Make sure the programs you use to run AI locally (Ollama, llama.cpp, vLLM) support the hardware.
  • Prices swing. There are reports that memory shortages pushed prices up. When comparing with cloud API costs, recalculate with current prices and your real usage.

What to watch

The actual release and price of Ryzen AI Halo, successors to the Spark and price trends, Intel’s Crescent Island schedule, and the state of ROCm and OpenVINO support.

Sources

Source: www.servethehome.com

← Back