Dedicated GB10 silicon offers a critical advantage over multi-tenant environments by eliminating noisy neighbor latency, ensuring consistent performance for your large language model (LLM) workloads. This hardware isolation is crucial for maintaining low p99 latency, which is more indicative of user experience than average latency. This post will explain how hardware isolation works and why it's essential for SREs concerned with API performance.
In multi-tenant environments, the performance of your LLM can be affected by the workloads of other tenants. This phenomenon, known as the noisy neighbor problem, can lead to unpredictable latency spikes, particularly in the p99 latency metric. The p99 latency is the 99th percentile of response times, which means it represents the worst-case scenario for 99% of your users. This is why it is a more critical metric than the average latency, which can mask these spikes.
ZC Inference Exchange utilizes dedicated NVIDIA GB10 silicon to run Qwen 2.5 models, both 32B and 72B versions. This hardware isolation ensures that your LLM inference is not impacted by other users' workloads, providing consistent performance and low p99 latency. The GB10 silicon is optimized for the demands of LLM inference, offering high throughput and low latency.
Our pricing tiers are designed to be cost-effective while providing the necessary scale for any workload. From the Starter plan at $99/month for 1.5M tokens, to the Business plan at $1999/month for 60M tokens, you can choose the plan that best fits your needs. We undercut competitors by 60-80% per 1M tokens, offering significant savings without compromising on performance.
To start benefiting from our dedicated GB10 silicon, sign up at ZC Inference Exchange. Our OpenAI-compatible API at /v1/chat supports Bearer token authentication, making it easy to integrate with your existing infrastructure.
# Example of using the ZC Inference Exchange API
# Set your Bearer token
export BEARER_TOKEN="your_bearer_token"
# Make a request to the API
curl -X POST https://zcx.zctechnologies.org/v1/chat \
-H "Authorization: Bearer $BEARER_TOKEN" \
-H "Content-Type: application/json" \
-d '{"model": "qwen2.5:32b", "messages": [{"role": "user", "content": "What is the capital of France?"}]}'For SREs concerned with the performance of their LLM workloads, dedicated GB10 silicon is a must-have. It ensures that your API performance is not affected by noisy neighbors, resulting in a better user experience. Sign up today at ZC Inference Exchange to start enjoying the benefits of hardware isolation.