python3simple-bench-to-api.py\--urlhttp://10.96.0.188:11434/v1\--modeldeepseek-r1:32b\--api_key"any-string-if-no-apikey-on-sever"\--concurrency1\--prompt"Tellmeastory"\--max_tokens100\--duration_seconds30
--url 推理服务的基础地址，路径以 /v1 结束
--model 推理服务的模型名称
--concurrency 并发数，如果这个设置了，则忽略 --concurrencys 参数内容
--prompt 每个请求的用户问题
--max_tokens 最大 token 数
--duration_seconds 持续时间，单位秒
--concurrencys 逗号分隔的并发数列表，如 1,5,10,15,20,30；不设置 concurrency 时才生效。会分别按设置的并发数发起压力，两个并发批次中间间隔5秒
压测结果：并发数:1总请求数:9成功率:100.00%平均延迟:3.3685s最大延迟:3.4171s最小延迟:3.3369s平均首字延迟:0.0767sP90延迟:3.3893sP95延迟:3.4032sP99延迟:3.4143s总生成tokens数:918单并发最小吞吐量:30.71tokens/s单并发最大吞吐量:31.26tokens/s单并发平均吞吐量:30.99tokens/s总体吞吐量:30.24tokens/s
”延迟“：从发出请求，到接收到最后一个token或字符的时间
“P90延迟”：分位数90的延迟，计算方法为延迟从小到大排序，前90%的最大延迟值，和下一个延迟值，基于线性插值计算的一个介于2者之间的值。表示90%的请求都可以在这个时间之内完成。P95、P99延迟以此类推。
“首字延迟”：从发出请求，到接收到第一个返回字符(或token)的时间。
“单并发吞吐量”：这个名字是我为了和其他指标区分想出来的一个名字，一时没找到通用的对应名字。意思是指站在每个并发用户/通道的角度看，从首token返回后，token的生成速度。统计时间不包含首字延迟。即一个通道的吞吐量 = 该通道生成的token数/除首token延迟外的生成时间。个人觉得，这个指标加上平均首字延迟，能反映真实的用户体感。
平均首字延迟：所有通道的首字延迟的平均值
单并发最小吞吐量： 所有并发通道中，吞吐量最小的通道的吞吐量
单并发最大吞吐量： 所有并发通道中，吞吐量最大的通道的吞吐量
单并发平均吞吐量：所有并发通道的吞吐量的平均值
总体吞吐量：在压测期间所有通道生成的tokens总数/压测开始到结束的时间
P90延迟: 3.3893s：表示有90%的请求延迟低于这个数值
P95延迟: 3.4032s：表示有95%的请求延迟低于这个数值
P99延迟: 3.4143s：表示有95%的请求延迟低于这个数值
模型量化：无
部署方式：k8s + vLLM 0.7.3
压测发起地：与模型推理同一台机器，这样就消除了远程网络开销的影响。
服务端：
--gpu_memory_utilization0.95\--max-num-seqs512\--max-model-len65536
客户端压测命令：
python3simple-bench-to-api.py\--urlhttp://10.96.2.221:7869/v1\--modelDeepSeek-R1-Distill-Qwen-7B\--concurrencys1,10,50,100,150,200\--prompt"Tellmeastory"\--max_tokens100\--api_key服务端配置的APIKey\--duration_seconds30
压测结果（脚本会汇总每个并发的结果，生成Markdown表格）：
从这个结果汇总看，当并发从1到200时
平均首字延迟(avg_ttft)从0.0363s增长到了0.6999s，仍然很快。
单并发平均吞吐量(avg)从 60.14 tokens/s 下降到了 23.27 tokens/s； 说明在200个并发时，单用户的延迟会变慢1倍多，符合预期。
总体吞吐量从 58.76 tokens/s 上升到了 3730.98 tokens/s，并越大，总体吞吐量增长越缓慢，符合一般规律。
平均延迟(avg_latency)从1.6825s 增长到了4.9803s， 还算不错的。
总体说明，单卡 4090 跑 R1-7B 模型，在200并发之内都是很流畅的。
服务端日志：
# 1并发INFO03-0303:59:21metrics.py:455] Avg prompt throughput:1.3tokens/s, Avg generation throughput:0.1tokens/s, Running:1reqs, Swapped:0reqs, Pending:0reqs, GPU KV cache usage:0.0%, CPU KV cache usage:0.0%.INFO03-0303:59:26metrics.py:455] Avg prompt throughput:3.6tokens/s, Avg generation throughput:59.7tokens/s, Running:0reqs, Swapped:0reqs, Pending:0reqs, GPU KV cache usage:0.0%, CPU KV cache usage:0.0%.INFO03-0303:59:31metrics.py:455] Avg prompt throughput:5.4tokens/s, Avg generation throughput:59.5tokens/s, Running:1reqs, Swapped:0reqs, Pending:0reqs, GPU KV cache usage:0.1%, CPU KV cache usage:0.0%.INFO03-0303:59:36metrics.py:455] Avg prompt throughput:5.4tokens/s, Avg generation throughput:59.5tokens/s, Running:1reqs, Swapped:0reqs, Pending:0reqs, GPU KV cache usage:0.1%, CPU KV cache usage:0.0%.INFO03-0303:59:41metrics.py:455] Avg prompt throughput:5.4tokens/s, Avg generation throughput:59.4tokens/s, Running:1reqs, Swapped:0reqs, Pending:0reqs, GPU KV cache usage:0.1%, CPU KV cache usage:0.0%.INFO03-0303:59:46metrics.py:455] Avg prompt throughput:5.4tokens/s, Avg generation throughput:59.5tokens/s, Running:1reqs, Swapped:0reqs, Pending:0reqs, GPU KV cache usage:0.1%, CPU KV cache usage:0.0%.INFO03-0303:59:51metrics.py:455] Avg prompt throughput:5.4tokens/s, Avg generation throughput:59.5tokens/s, Running:1reqs, Swapped:0reqs, Pending:0reqs, GPU KV cache usage:0.1%, CPU KV cache usage:0.0%.# 10并发INFO03-0303:59:57metrics.py:455] Avg prompt throughput:3.4tokens/s, Avg generation throughput:2.4tokens/s, Running:10reqs, Swapped:0reqs, Pending:0reqs, GPU KV cache usage:0.2%, CPU KV cache usage:0.0%.INFO03-0304:00:02metrics.py:455] Avg prompt throughput:50.3tokens/s, Avg generation throughput:561.9tokens/s, Running:10reqs, Swapped:0reqs, Pending:0reqs, GPU KV cache usage:1.0%, CPU KV cache usage:0.0%.INFO03-0304:00:07metrics.py:455] Avg prompt throughput:53.9tokens/s, Avg generation throughput:553.2tokens/s, Running:10reqs, Swapped:0reqs, Pending:0reqs, GPU KV cache usage:0.8%, CPU KV cache usage:0.0%.INFO03-0304:00:12metrics.py:455] Avg prompt throughput:54.0tokens/s, Avg generation throughput:553.3tokens/s, Running:10reqs, Swapped:0reqs, Pending:0reqs, GPU KV cache usage:0.5%, CPU KV cache usage:0.0%.INFO03-0304:00:17metrics.py:455] Avg prompt throughput:54.0tokens/s, Avg generation throughput:554.3tokens/s, Running:10reqs, Swapped:0reqs, Pending:0reqs, GPU KV cache usage:0.3%, CPU KV cache usage:0.0%.INFO03-0304:00:22metrics.py:455] Avg prompt throughput:39.6tokens/s, Avg generation throughput:557.7tokens/s, Running:10reqs, Swapped:0reqs, Pending:0reqs, GPU KV cache usage:0.8%, CPU KV cache usage:0.0%.INFO03-0304:00:27metrics.py:455] Avg prompt throughput:50.3tokens/s, Avg generation throughput:556.0tokens/s, Running:10reqs, Swapped:0reqs, Pending:0reqs, GPU KV cache usage:0.8%, CPU KV cache usage:0.0%.INFO03-0304:00:32metrics.py:455] Avg prompt throughput:6.1tokens/s, Avg generation throughput:50.9tokens/s, Running:50reqs, Swapped:0reqs, Pending:0reqs, GPU KV cache usage:0.8%, CPU KV cache usage:0.0%.# 50并发INFO03-0304:00:37metrics.py:455] Avg prompt throughput:172.2tokens/s, Avg generation throughput:1967.8tokens/s, Running:49reqs, Swapped:0reqs, Pending:0reqs, GPU KV cache usage:5.5%, CPU KV cache usage:0.0%.INFO03-0304:00:42metrics.py:455] Avg prompt throughput:180.0tokens/s, Avg generation throughput:1775.5tokens/s, Running:50reqs, Swapped:0reqs, Pending:0reqs, GPU KV cache usage:4.9%, CPU KV cache usage:0.0%.INFO03-0304:00:47metrics.py:455] Avg prompt throughput:178.7tokens/s, Avg generation throughput:1923.5tokens/s, Running:50reqs, Swapped:0reqs, Pending:0reqs, GPU KV cache usage:4.2%, CPU KV cache usage:0.0%.INFO03-0304:00:52metrics.py:455] Avg prompt throughput:181.2tokens/s, Avg generation throughput:1939.9tokens/s, Running:47reqs, Swapped:0reqs, Pending:1reqs, GPU KV cache usage:3.3%, CPU KV cache usage:0.0%.INFO03-0304:00:58metrics.py:455] Avg prompt throughput:186.5tokens/s, Avg generation throughput:1942.9tokens/s, Running:50reqs, Swapped:0reqs, Pending:0reqs, GPU KV cache usage:3.3%, CPU KV cache usage:0.0%.INFO03-0304:01:03metrics.py:455] Avg prompt throughput:172.2tokens/s, Avg generation throughput:1946.6tokens/s, Running:45reqs, Swapped:0reqs, Pending:0reqs, GPU KV cache usage:2.8%, CPU KV cache usage:0.0%.# 100并发INFO03-0304:01:10metrics.py:455] Avg prompt throughput:3.6tokens/s, Avg generation throughput:321.9tokens/s, Running:62reqs, Swapped:0reqs, Pending:38reqs, GPU KV cache usage:1.0%, CPU KV cache usage:0.0%.INFO03-0304:01:15metrics.py:455] Avg prompt throughput:352.8tokens/s, Avg generation throughput:3194.2tokens/s, Running:100reqs, Swapped:0reqs, Pending:0reqs, GPU KV cache usage:8.0%, CPU KV cache usage:0.0%.INFO03-0304:01:20metrics.py:455] Avg prompt throughput:190.3tokens/s, Avg generation throughput:2817.9tokens/s, Running:6reqs, Swapped:0reqs, Pending:0reqs, GPU KV cache usage:0.3%, CPU KV cache usage:0.0%.INFO03-0304:01:25metrics.py:455] Avg prompt throughput:352.9tokens/s, Avg generation throughput:3163.5tokens/s, Running:100reqs, Swapped:0reqs, Pending:0reqs, GPU KV cache usage:8.1%, CPU KV cache usage:0.0%.INFO03-0304:01:30metrics.py:455] Avg prompt throughput:197.2tokens/s, Avg generation throughput:2960.1tokens/s, Running:13reqs, Swapped:0reqs, Pending:2reqs, GPU KV cache usage:0.8%, CPU KV cache usage:0.0%.INFO03-0304:01:35metrics.py:455] Avg prompt throughput:358.0tokens/s, Avg generation throughput:3230.8tokens/s, Running:100reqs, Swapped:0reqs, Pending:0reqs, GPU KV cache usage:7.8%, CPU KV cache usage:0.0%.INFO03-0304:01:40metrics.py:455] Avg prompt throughput:185.1tokens/s, Avg generation throughput:2847.4tokens/s, Running:3reqs, Swapped:0reqs, Pending:0reqs, GPU KV cache usage:0.2%, CPU KV cache usage:0.0%.# 150并发INFO03-0304:01:47metrics.py:455] Avg prompt throughput:2.7tokens/s, Avg generation throughput:25.5tokens/s, Running:39reqs, Swapped:0reqs, Pending:22reqs, GPU KV cache usage:0.6%, CPU KV cache usage:0.0%.INFO03-0304:01:52metrics.py:455] Avg prompt throughput:535.9tokens/s, Avg generation throughput:3599.7tokens/s, Running:150reqs, Swapped:0reqs, Pending:0reqs, GPU KV cache usage:4.8%, CPU KV cache usage:0.0%.INFO03-0304:01:57metrics.py:455] Avg prompt throughput:268.1tokens/s, Avg generation throughput:3613.7tokens/s, Running:150reqs, Swapped:0reqs, Pending:0reqs, GPU KV cache usage:9.7%, CPU KV cache usage:0.0%.INFO03-0304:02:02metrics.py:455] Avg prompt throughput:273.3tokens/s, Avg generation throughput:3520.0tokens/s, Running:150reqs, Swapped:0reqs, Pending:0reqs, GPU KV cache usage:11.7%, CPU KV cache usage:0.0%.INFO03-0304:02:07metrics.py:455] Avg prompt throughput:280.3tokens/s, Avg generation throughput:3703.8tokens/s, Running:150reqs, Swapped:0reqs, Pending:0reqs, GPU KV cache usage:14.1%, CPU KV cache usage:0.0%.INFO03-0304:02:12metrics.py:455] Avg prompt throughput:278.9tokens/s, Avg generation throughput:3685.7tokens/s, Running:11reqs, Swapped:0reqs, Pending:0reqs, GPU KV cache usage:0.8%, CPU KV cache usage:0.0%.INFO03-0304:02:17metrics.py:455] Avg prompt throughput:406.6tokens/s, Avg generation throughput:3176.2tokens/s, Running:81reqs, Swapped:0reqs, Pending:0reqs, GPU KV cache usage:2.0%, CPU KV cache usage:0.0%.# 200并发INFO03-0304:02:25metrics.py:455] Avg prompt throughput:13.0tokens/s, Avg generation throughput:867.2tokens/s, Running:58reqs, Swapped:0reqs, Pending:44reqs, GPU KV cache usage:0.9%, CPU KV cache usage:0.0%.INFO03-0304:02:30metrics.py:455] Avg prompt throughput:472.7tokens/s, Avg generation throughput:4012.6tokens/s, Running:157reqs, Swapped:0reqs, Pending:29reqs, GPU KV cache usage:2.7%, CPU KV cache usage:0.0%.INFO03-0304:02:35metrics.py:455] Avg prompt throughput:244.4tokens/s, Avg generation throughput:3969.8tokens/s, Running:12reqs, Swapped:0reqs, Pending:0reqs, GPU KV cache usage:0.2%, CPU KV cache usage:0.0%.INFO03-0304:02:40metrics.py:455] Avg prompt throughput:439.9tokens/s, Avg generation throughput:4022.8tokens/s, Running:93reqs, Swapped:0reqs, Pending:32reqs, GPU KV cache usage:1.6%, CPU KV cache usage:0.0%.INFO03-0304:02:45metrics.py:455] Avg prompt throughput:258.5tokens/s, Avg generation throughput:3944.5tokens/s, Running:6reqs, Swapped:0reqs, Pending:20reqs, GPU KV cache usage:0.1%, CPU KV cache usage:0.0%.INFO03-0304:02:50metrics.py:455] Avg prompt throughput:405.5tokens/s, Avg generation throughput:4000.0tokens/s, Running:68reqs, Swapped:0reqs, Pending:22reqs, GPU KV cache usage:1.1%, CPU KV cache usage:0.0%.INFO03-0304:02:55metrics.py:455] Avg prompt throughput:320.2tokens/s, Avg generation throughput:4016.7tokens/s, Running:5reqs, Swapped:0reqs, Pending:0reqs, GPU KV cache usage:0.2%, CPU KV cache usage:0.0%.
GPU KV cache usage在1%以内，说明缓存还没有充分利用，系统还有潜力
资源稳态消耗：
# 1并发|=========================================+======================+======================|| 7 NVIDIA GeForce RTX 4090 Off | 00000000:E1:00.0 Off | Off || 30% 50C P2 291W / 450W | 22736MiB / 24564MiB | 94% Default || | | N/A |+-----------------------------------------+----------------------+----------------------+# 10并发|=========================================+======================+======================|| 7 NVIDIA GeForce RTX 4090 Off | 00000000:E1:00.0 Off | Off || 30% 56C P2 297W / 450W | 22736MiB / 24564MiB | 89% Default || | | N/A |+-----------------------------------------+----------------------+----------------------+# 50并发|=========================================+======================+======================|| 7 NVIDIA GeForce RTX 4090 Off | 00000000:E1:00.0 Off | Off || 30% 57C P2 298W / 450W | 22736MiB / 24564MiB | 84% Default || | | N/A |+-----------------------------------------+----------------------+----------------------+# 150并发|=========================================+======================+======================|| 7 NVIDIA GeForce RTX 4090 Off | 00000000:E1:00.0 Off | Off || 46% 61C P2 342W / 450W | 22736MiB / 24564MiB | 83% Default || | | N/A |+-----------------------------------------+----------------------+----------------------+# 200并发|=========================================+======================+======================|| 7 NVIDIA GeForce RTX 4090 Off | 00000000:E1:00.0 Off | Off || 35% 60C P2 323W / 450W | 22736MiB / 24564MiB | 78% Default || | | N/A |+-----------------------------------------+----------------------+----------------------+
符合预期，并发越大，系统调度的上下文切换消耗就越多，显卡的算力使用就会下降
服务端：
--gpu_memory_utilization0.95\--max-num-seqs512\--max-model-len65536
客户端：
python3simple-bench-to-api.py\--urlhttp://10.96.2.221:7869/v1\--modelDeepSeek-R1-Distill-Qwen-7B\--concurrencys1,10,50,100,150,200\--prompt"Tellmeastory"\--max_tokens1024\--api_key服务端配置的APIKey\--duration_seconds30
服务不变，客户端 max_tokens 从参数组合1的100，变成了 1024，压测结果：
可以看到平均延迟在100个token的时候1个并发只有1.7秒，当上下文增加了10倍到1k后，1个并发下的平均延迟到了16到17秒，正好也变为原来的10倍，符合预期。
当设置为1k上下文后，从1个并发到200个并发，变化规律和100个token的上下文是基本一样的：
平均首字延迟(avg_ttft)从0.0404s增长到了0.7275s，仍然很快。
单并发平均吞吐量(avg)从 60.18 tokens/s 下降到了 20.89 tokens/s； 说明在200个并发时，单用户的延迟会变慢1倍多，符合预期。
总体吞吐量从 59.95 tokens/s 上升到了 3157.15 tokens/s，并发越大，总体吞吐量增长越缓慢，符合一般规律。
平均延迟(avg_latency)从16.3665s 增长到了49.7491s，符合预期
服务端日志：略
并发从1到200的过程中，GPU KV cache usage 从个位数增长到了99.9%
资源稳态情况：略
服务端：
--gpu_memory_utilization0.95\--max-num-seqs512\--max-model-len65536
客户端：
#python3simple-bench-to-api.py\--urlhttp://10.96.2.221:7869/v1\--modelDeepSeek-R1-Distill-Qwen-7B\--concurrencys1,10,50,100,150,200\--prompt"Tellmeastory"\--max_tokens16384\--api_key服务端配置的APIKey\--duration_seconds30
服务端不变， 客户端 max_tokens 改为 16k，压测结果如下：
可以看到，在16k上下文时，200个并发和150个并发比较，吞吐量上升已经不明显了。平均延迟66秒，P99的延迟已经比较高。16k下，7B还是在100个并发内质量比较稳定。另外我测试的时候，服务同时有个别小伙伴有调用，可能有些结果并不准确。
服务端日志：略
资源情况：略
服务端：
--gpu_memory_utilization0.95\--max-num-seqs256\--max-model-len103632
客户端：
python3simple-bench-to-api.py\--urlhttp://10.96.2.221:7869/v1\--modelDeepSeek-R1-Distill-Qwen-7B\--concurrencys1,10,50,100,150,200\--prompt"IntroducethehistoryofChina"\--max_tokens100\--api_key服务端配置的APIKEY\--duration_seconds30
服务端 max-num-seqs 设为 256，max-model-len 设为103632， 客户端 max_tokens 设为 100
压测结果：
服务端日志：略
资源情况：略
服务端：
--gpu_memory_utilization0.95\--max-num-seqs256\--max-model-len103632
客户端：
python3simple-bench-to-api.py--urlhttp://10.96.2.221:7869/v1\--modelDeepSeek-R1-Distill-Qwen-7B\--concurrencys1,10,50,100,150,200\--prompt"IntroducethehistoryofChina"\--max_tokens1024\--api_key服务端配置的APIKEY\--duration_seconds30
服务端不变， 客户端 max_tokens 设为 1024，压测结果
服务端日志：略
资源稳态消耗：略
服务端：
--gpu_memory_utilization0.95\--max-num-seqs256\--max-model-len103632
客户端：
python3simple-bench-to-api.py--urlhttp://10.96.2.221:7869/v1\--modelDeepSeek-R1-Distill-Qwen-7B\--concurrencys1,10,50,100,150,200\--prompt"IntroducethehistoryofChina"\--max_tokens16384\--api_key服务端配置的APIKEY\--duration_seconds30
服务端不变， 客户端 max_tokens 设为 16384（16k上下文），压测结果：
可以看到，同样是16k的max_tokens，服务端参数调整后，200个并发下的吞吐量934，比调整前的794增加了 17% 多，且P99延迟下降了近一半。这个原因还在分析，也不排除是两次压测期间，受到的干扰不同造成的。
服务端日志：略
资源情况：略
模型地址：https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B
部署方式： k8s + ollama，ollama版本： 0.5.10
模型量化：ollama 官方模型 deepseek-r1:32b， 基于 deepseek-ai/DeepSeek-R1-Distill-Qwen-32B 做的 4-bit （Q4_K_M） 量化的 gguf 版本；
压测发起地：与模型推理同一台机器，这样就消除了远程网络开销的影响。
python3simple-bench-to-api.py\--urlhttp://10.96.0.188:11434/v1\--modeldeepseek-r1:32b\--concurrencys1,5,10,15,20,30\--prompt"IntroducethehistoryofChina"\--max_tokens100\--api_key服务端设置的APIKEY，没有随便填一个\--duration_seconds30
资源情况：
# 1并发|=========================================+======================+======================|| 2 NVIDIA GeForce RTX 4090 Off | 00000000:41:00.0 Off | Off || 30% 49C P2 340W / 450W | 23012MiB / 24564MiB | 79% Default || | | N/A |+-----------------------------------------+----------------------+----------------------+# 5并发|=========================================+======================+======================|| 2 NVIDIA GeForce RTX 4090 Off | 00000000:41:00.0 Off | Off || 30% 53C P2 379W / 450W | 23012MiB / 24564MiB | 75% Default || | | N/A |+-----------------------------------------+----------------------+----------------------+# 10并发|=========================================+======================+======================|| 2 NVIDIA GeForce RTX 4090 Off | 00000000:41:00.0 Off | Off || 42% 61C P2 372W / 450W | 23012MiB / 24564MiB | 76% Default || | | N/A |+-----------------------------------------+----------------------+----------------------+# 15并发|=========================================+======================+======================|| 2 NVIDIA GeForce RTX 4090 Off | 00000000:41:00.0 Off | Off || 53% 58C P2 356W / 450W | 23012MiB / 24564MiB | 72% Default || | | N/A |+-----------------------------------------+----------------------+----------------------+# 20并发|=========================================+======================+======================|| 2 NVIDIA GeForce RTX 4090 Off | 00000000:41:00.0 Off | Off || 53% 61C P2 378W / 450W | 23012MiB / 24564MiB | 76% Default || | | N/A |+-----------------------------------------+----------------------+----------------------+# 30并发|=========================================+======================+======================|| 2 NVIDIA GeForce RTX 4090 Off | 00000000:41:00.0 Off | Off || 55% 62C P2 382W / 450W | 23012MiB / 24564MiB | 76% Default || | | N/A |+-----------------------------------------+----------------------+----------------------+
python3simple-bench-to-api.py\--urlhttp://10.96.0.188:11434/v1\--modeldeepseek-r1:32b\--concurrencys1,5,10,15,20,30\--prompt"IntroducethehistoryofChina"\--max_tokens1024\--api_key服务端设置的APIKEY，没有随便填一个\--duration_seconds30
资源情况：
# 1并发|=========================================+======================+======================|| 2 NVIDIA GeForce RTX 4090 Off | 00000000:41:00.0 Off | Off || 30% 52C P2 333W / 450W | 23012MiB / 24564MiB | 87% Default || | | N/A |+-----------------------------------------+----------------------+----------------------+# 5并发|=========================================+======================+======================|| 2 NVIDIA GeForce RTX 4090 Off | 00000000:41:00.0 Off | Off || 33% 57C P2 370W / 450W | 23012MiB / 24564MiB | 79% Default || | | N/A |+-----------------------------------------+----------------------+----------------------+# 10并发|=========================================+======================+======================|| 2 NVIDIA GeForce RTX 4090 Off | 00000000:41:00.0 Off | Off || 53% 60C P2 376W / 450W | 23012MiB / 24564MiB | 77% Default || | | N/A |+-----------------------------------------+----------------------+----------------------+# 15并发|=========================================+======================+======================|| 2 NVIDIA GeForce RTX 4090 Off | 00000000:41:00.0 Off | Off || 54% 63C P2 385W / 450W | 23012MiB / 24564MiB | 78% Default || | | N/A |+-----------------------------------------+----------------------+----------------------+# 20并发|=========================================+======================+======================|| 2 NVIDIA GeForce RTX 4090 Off | 00000000:41:00.0 Off | Off || 65% 62C P2 376W / 450W | 23012MiB / 24564MiB | 77% Default || | | N/A |+-----------------------------------------+----------------------+----------------------+# 30并发|=========================================+======================+======================|| 2 NVIDIA GeForce RTX 4090 Off | 00000000:41:00.0 Off | Off || 63% 59C P2 371W / 450W | 23012MiB / 24564MiB | 76% Default || | | N/A |+-----------------------------------------+----------------------+----------------------+
python3simple-bench-to-api.py\--urlhttp://10.96.0.188:11434/v1\--modeldeepseek-r1:32b\--concurrencys1,5,10,15,20,30\--prompt"IntroducethehistoryofChina"\--max_tokens16384\--api_key服务端设置的APIKEY，没有随便填一个\--duration_seconds30
资源情况：略
| 欢迎光临 链载Ai (https://www.lianzai.com/) | Powered by Discuz! X3.5 |