Xinference简化各种 AI 模型的运行和集成，用于本地环境部署开源 LLM、嵌入模型和多模态模型运行推理。condacreate-nxinferencepython=3.11condaactivatexinference pip3installtorchtorchvisiontorchaudio--index-urlhttps://download.pytorch.org/whl/cu126 #环境变量配置exportHF_ENDPOINT="https://hf-mirror.com"exportUSE_MODELSCOPE_HUB=1exportXINFERENCE_HOME=/home/jovyan/dev/xinferenceexportXINFERENCE_MODEL_SRC=modelscope#exportXINFERENCE_ENDPOINT=http://0.0.0.0:9997
gitclonehttps://github.com/xorbitsai/inference.gitcdinferencepipinstall-e. pipinstall"xinference[all]" pipinstallsentence-transformers#pipinstallflash_attn pipinstallsentencepiecepipinstallprotobuf 1，Transformers 引擎（PyTorch）#支持几乎有所的最新模型，Pytorch模型默认使用的引擎pipinstall"xinference[transformers]" #支持高并发，使用vllm引擎能获取更高的吞吐量pipinstall"xinference[vllm]"#FlashInferisoptionalbutrequiredforspecificfunctionalitiessuchasslidingwindowattentionwithGemma2.#ForCUDA12.4&torch2.4tosupportslidingwindowattentionforgemma2andllama3.1styleropepipinstallflashinfer-ihttps://flashinfer.ai/whl/cu124/torch2.4#ForotherCUDA&torchversions,pleasecheckhttps://docs.flashinfer.ai/installation.html # 说明：当满足以下条件时，自动选择 vllm 作为引擎：模型格式为pytorch，gptq或者awq。当模型格式为pytorch时，量化选项需为none。当模型格式为awq时，量化选项需为Int4。当模型格式为gptq时，量化选项需为Int3、Int4或者Int8。操作系统为Linux并且至少有一个支持CUDA的设备自定义模型的model_family字段和内置模型的model_name字段在vLLM的支持列表中。 说明：自 v1.5.0 起，xllamacpp 成为 llama.cpp 后端的默认选项。如需启用 llama-cpp-python，请设置环境变量 USE_XLLAMACPP=0#CPU或MacMetal：pipinstall-Uxllamacpp#Cuda:pipinstallxllamacpp--force-reinstall--index-urlhttps://xorbitsai.github.io/xllamacpp/whl/cu124#HIP:pipinstallxllamacpp--force-reinstall--index-urlhttps://xorbitsai.github.io/xllamacpp/whl/rocm-6.0.2 #llama-cpp-python 不同硬件的安装方式：#AppleM系列CMAKE_ARGS="-DLLAMA_METAL=on"pipinstallllama-cpp-python#英伟达显卡：CMAKE_ARGS="-DLLAMA_CUBLAS=on"pipinstallllama-cpp-python#AMD显卡：CMAKE_ARGS="-DLLAMA_HIPBLAS=on"pipinstallllama-cpp-python Xinference通过xllamacpp或llama-cpp-python支持gguf格式的模型。xllamacpp由Xinference团队开发，并将在未来成为llama.cpp的唯一后端。llama-cpp-python是llama.cpp后端的默认选项。要启用xllamacpp，请添加环境变量USE_XLLAMACPP=1。在即将发布的Xinferencev1.5.0中，xllamacpp将成为llama.cpp的默认选项，而llama-cpp-python将被弃用。在Xinferencev1.6.0中，llama-cpp-python将被移除。 pipinstall"xinference[sglang]"#ForCUDA12.4&torch2.4tosupportslidingwindowattentionforgemma2andllama3.1styleropepipinstallflashinfer-ihttps://flashinfer.ai/whl/cu124/torch2.4#ForotherCUDA&torchversions,pleasecheckhttps://docs.flashinfer.ai/installation.html SGLang具有基于RadixAttention的高性能推理运行时。SGLang通过在多个调用之间自动重用KV缓存，显著加速了复杂LLM程序的执行。SGLang还支持其他常见推理技术，如连续批处理和张量并行处理。 pipinstall"xinference[mlx]" MLX-lm用来在苹果silicon芯片上提供高效的LLM推理。 1，启动（注意：默认从huggingface拉模型）XINFERENCE_MODEL_SRC=modelscopexinference-local--host0.0.0.0--port9997 说明1：xinference-local默认会在本地启动一个worker，端点为：http://127.0.0.1:9997，端口默认为9997，仅支持本机本地访问。说明2：默认使用<HOME>/.xinference作为主目录来存储一些必要的信息（日志文件、模型文件等），配置环境变量XINFERENCE_HOME修改主目录：XINFERENCE_HOME=/tmp/xinferencexinference-local--host0.0.0.0--port9997说明3：默认从huggingface拉模型，可配置XINFERENCE_MODEL_SRC=modelscope指定拉取模型hub http://127.0.0.1:9997/docs XINFERENCE_MODEL_SRC=modelscopexinference-local--host0.0.0.0--port9997#后台启动nohupXINFERENCE_MODEL_SRC=modelscopexinference-local--host0.0.0.0--port9997>xinference-local.log2>&1& curl-X' OST'\'http://127.0.0.1:9997/v1/chat/completions'\-H'accept:application/json'\-H'Content-Type:application/json'\-d'{"model":"qwen2.5-instruct","messages":[{"role":"system","content":"Youareahelpfulassistant."},{"role":"user","content":"你是谁?"}]}' # 测试qwen2.5-vl模型（下载运行同qwen2.5）curlhttp://127.0.0.1:9997/v1/embeddings\-H"Content-Type:application/json"\-d'{"input":"北京景点推荐","model":"jina-embeddings-v3"}'curl-X'OST''http://127.0.0.1:9997/v1/rerank'\-H'Content-Type:application/json'\-d'{"model":"bge-reranker-v2-m3","query":"一个男人正在吃意大利面。","documents":["一个男人在吃东西。","一个男人正在吃一块面包。","这个女孩怀着一个婴儿。","一个人在骑马。","一个女人在拉小提琴。"]}' #测试propmtAdigitalillustrationofamoviepostertitled[‘SadSax:FuryToad’],[MadMax]parodyposter,featuring[asaxophone-playingtoadinapost-apocalypticdesert,withacustomizedcarmadeofmusicalinstruments],inthebackground,[awastelandwithothermusicalvehiclechases],movietitlein[agritty,boldfont,dustyandintensecolorpalette]. #测试propmtAdigitalillustrationofamoviepostertitled['Mulan'],featuring[afiercewarriorwomanwithlongflowingblackhair,dressedintraditionalChinesearmorwithredaccents,holdingaswordreadyforbattle].Sheisposedinadynamicactionstanceagainst[abackdropofruggedsnow-coveredmountainswithdarkstormyskies].Themovietitle['Mulan']iswrittenin[boldredcalligraphy-styletext,prominentlydisplayedatthebottom],alongwith[areleasedateinsmallerfont].Thesceneconveys[intensity,bravery,andanepicadventure]. curl-X'OST'\'http://localhost:9997/v1/audio/speech'\-H'accept:application/json'\-H'Content-Type:application/json'\-d'{"model":"CosyVoice2-0.5B","input":"hello","voice":"中文女"}'-ohello1.mp3 curl-X'OST'\'http://127.0.0.1:9997/v1/video/generations'\-H'accept:application/json'\-H'Content-Type:application/json'\-d'{"model":"CogVideoX-5b","prompt":"anapple"}'-oapple.mp4 # 取b64_json进行base64解码即可生成视频说明：pytorch模型格式，可以选择vllm启动，速度会快很多curl-X'OST'\'http://127.0.0.1:9997/v1/chat/completions'\-H'accept:application/json'\-H'Content-Type:application/json'\-d'{"model":"my-Qwen2.5-32B-Instruct-GPTQ-Int4","messages":[{"role":"system","content":"Youareahelpfulassistant."},{"role":"user","content":"你是谁?"}]}' pip3installtorchtorchvisiontorchaudio--index-urlhttps://download.pytorch.org/whl/cu126 pipinstallsentence-transformers | 
