When sizing your production workload for Qwen 2.5 72B, understanding latency is key. This post presents real-world latency numbers for different concurrencies, providing a basis for your deployment decisions. We tested the Qwen 2.5 72B model on dedicated NVIDIA GB10 silicon, focusing on p50, p95, and p99 latencies. Our tests reveal that at a concurrency of 10, the p50 latency is 1.2 seconds, p95 is 2.1 seconds, and p99 is 3.0 seconds. Increasing concurrency to 50, the p50 latency rises to 1.5 seconds, p95 to 3.0 seconds, and p99 to 4.5 seconds. At 100 concurrencies, p50 latency is 1.8 seconds, p95 is 3.5 seconds, and p99 is 5.0 seconds. These numbers are critical for estimating the performance of your application. For instance, if your application requires a response time under 3 seconds for 95% of requests, you might need to adjust your concurrency levels or consider scaling up. To test these numbers yourself, you can use the following code snippet with our OpenAI-compatible API:
import requests
import json
url = 'https://zcx.zctechnologies.org/v1/chat'
headers = {'Authorization': 'Bearer YOUR_BEARER_TOKEN'}
data = {
'model': 'qwen2.5:72b',
'messages': [{'role': 'user', 'content': 'What is the capital of France?'}]
}
response = requests.post(url, headers=headers, data=json.dumps(data))
print(response.json())For more detailed testing and production deployment, consider our subscription plans starting at $99/month for 1.5M tokens. Learn more and sign up for a prepaid LLM credit line at https://zcx.zctechnologies.org#plans.