|
一、服务器信息
二、安装NVIDIA 显卡驱动和 CUDA
2.1 安装内核头文件等依赖包
yuminstall-ykernel-uek-develgccmakedkmselfutils-libelf-devel
安装完成后确保kernel 与 kernel-devel 对应
如图所示：
2.2 下载并安装驱动 https://www.nvidia.com/en-us/drivers/results/ NVIDIA-Linux-x86_64-570.133.07
chmod+xNVIDIA-Linux-x86_64-570.133.07.run./NVIDIA-Linux-x86_64-570.133.07.run https://developer.nvidia.com/cuda-12-4-0-download-archive?target_os=Linux&target_arch=x86_64&Distribution=RHEL&target_version=8&target_type=runfile_local CUDA 12.4 runfile (local)
下载后执行以下命令： chmod+xcuda_12.4.0_550.54.14_linux.run./cuda_12.4.0_550.54.14_linux.run
2.3 验证驱动是否安装成功
若能正确显示显卡信息，则安装成功。如图所示：
三、安装Conda
3.1 安装 Miniconda
wgethttps://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.shbashMiniconda3-latest-Linux-x86_64.sh 按提示安装后，执行以下命令使conda 生效：
3.2 创建并激活虚拟环境
condacreate-nxinferencepython=3.10-ycondaactivatexinference
四、安装Xinference 及相关依赖
4.1 安装 xinference（vllm 版本）
pipinstall"xinference[vllm]"
4.2 安装 flashinfer
pipinstallflashinfer-ihttps://flashinfer.ai/whl/cu124/torch2.4
参考文档： https://inference.readthedocs.io/zh-cn/latest/getting_started/installation.html#vllm-backend
4.3 安装 sentence-transformers
pipinstallsentence-transformers
五、启动Xinference 服务
1、确保xinference环境已激活，如图所示，如未激活，执行下面命令激活。
2、启动xinference服务
nohupbash-c"HF_ENDPOINT=https://hf-mirror.comXINFERENCE_HOME=/data/xinferenceNewCUDA_VISIBLE_DEVICES=0,1xinference-local--host0.0.0.0--port9997"&
启动命令参数说明： nohup：命令在后台运行。 bash -c "..."：通过 bash 执行一系列设置和命令。 HF_ENDPOINT=https://hf-mirror.com：指定 Hugging Face国内镜像地址，用于加速模型下载。 XINFERENCE_HOME=/data/xinference：xinference 的工作路径，用于保存模型文件和缓存。 CUDA_VISIBLE_DEVICES=0,1：指定使用GPU 的编号（2块显卡）。 xinference-local：启动本地模式。 --host 0.0.0.0：监听所有 IP，外部网络可访问。 --port 9997：指定 Web 服务监听端口。
六、模型加载示例
6.1 启动重排序模型
6.2 启动嵌入模型
6.3 启动语言模型
参数说明： gpu_memory_utilization: 指定每张 GPU 最多使用其 80% 的显存（0~1 之间），防止显存占满。 max_num_seqs: 最大并行推理序列数量，此处设置最多同时处理30 个用户请求。 max_model_len: 模型输入 token 的最大长度，最多 40960 个 token。
ingFang SC", -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Ubuntu, "Helvetica Neue", Helvetica, Arial, "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", "Source Han Sans CN", sans-serif;font-optical-sizing: inherit;font-size-adjust: inherit;font-kerning: inherit;font-feature-settings: inherit;font-variation-settings: inherit;margin: 16px 0px 0px;padding: 0px;vertical-align: baseline;color: rgba(0, 0, 0, 0.9);letter-spacing: -0.16px;orphans: 2;text-align: start;text-indent: 0px;text-transform: none;widows: 2;word-spacing: 0px;-webkit-text-stroke-width: 0px;white-space: normal;background-color: rgb(255, 255, 255);text-decoration-thickness: initial;text-decoration-style: initial;text-decoration-color: initial;">6.4正在运行的模型
如图所示，已成功运行语言模型qwq32b，嵌入模型bge-m3，重排序模型bge-reranker-large
七、Conda 的一些常用命令及其功能介绍
如果有conda 常用命令不是很清楚的，可细看Conda 的一些常用命令及其功能介绍：
ingFang SC", -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Ubuntu, "Helvetica Neue", Helvetica, Arial, "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", "Source Han Sans CN", sans-serif;font-optical-sizing: inherit;font-size-adjust: inherit;font-kerning: inherit;font-feature-settings: inherit;font-variation-settings: inherit;margin: 16px 0px 0px;padding: 0px;vertical-align: baseline;color: rgba(0, 0, 0, 0.9);letter-spacing: -0.16px;orphans: 2;text-align: start;text-indent: 0px;text-transform: none;widows: 2;word-spacing: 0px;-webkit-text-stroke-width: 0px;white-space: normal;background-color: rgb(255, 255, 255);text-decoration-thickness: initial;text-decoration-style: initial;text-decoration-color: initial;">7.1、查看所有环境ingFang SC", -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Ubuntu, "Helvetica Neue", Helvetica, Arial, "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", "Source Han Sans CN", sans-serif;font-optical-sizing: inherit;font-size-adjust: inherit;font-kerning: inherit;font-feature-settings: inherit;font-variation-settings: inherit;margin: 16px 0px 0px;padding: 0px;vertical-align: baseline;color: rgba(0, 0, 0, 0.9);letter-spacing: -0.16px;orphans: 2;text-align: start;text-indent: 0px;text-transform: none;widows: 2;word-spacing: 0px;-webkit-text-stroke-width: 0px;white-space: normal;background-color: rgb(255, 255, 255);text-decoration-thickness: initial;text-decoration-style: initial;text-decoration-color: initial;">显示所有已创建的 Conda 环境，当前使用的环境前会标有一个星号*ingFang SC", -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Ubuntu, "Helvetica Neue", Helvetica, Arial, "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", "Source Han Sans CN", sans-serif;font-optical-sizing: inherit;font-size-adjust: inherit;font-kerning: inherit;font-feature-settings: inherit;font-variation-settings: inherit;margin: 16px 0px 0px;padding: 0px;vertical-align: baseline;color: rgba(0, 0, 0, 0.9);letter-spacing: -0.16px;orphans: 2;text-align: start;text-indent: 0px;text-transform: none;widows: 2;word-spacing: 0px;-webkit-text-stroke-width: 0px;white-space: normal;background-color: rgb(255, 255, 255);text-decoration-thickness: initial;text-decoration-style: initial;text-decoration-color: initial;">ingFang SC", -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Ubuntu, "Helvetica Neue", Helvetica, Arial, "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", "Source Han Sans CN", sans-serif;font-optical-sizing: inherit;font-size-adjust: inherit;font-kerning: inherit;font-feature-settings: inherit;font-variation-settings: inherit;margin: 16px 0px 0px;padding: 0px;vertical-align: baseline;color: rgba(0, 0, 0, 0.9);letter-spacing: -0.16px;orphans: 2;text-align: start;text-indent: 0px;text-transform: none;widows: 2;word-spacing: 0px;-webkit-text-stroke-width: 0px;white-space: normal;background-color: rgb(255, 255, 255);text-decoration-thickness: initial;text-decoration-style: initial;text-decoration-color: initial;">7.2、创建新环境condacreate--namemyenvpython=3.10 ingFang SC", -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Ubuntu, "Helvetica Neue", Helvetica, Arial, "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", "Source Han Sans CN", sans-serif;font-optical-sizing: inherit;font-size-adjust: inherit;font-kerning: inherit;font-feature-settings: inherit;font-variation-settings: inherit;margin: 16px 0px 0px;padding: 0px;vertical-align: baseline;color: rgba(0, 0, 0, 0.9);letter-spacing: -0.16px;orphans: 2;text-align: start;text-indent: 0px;text-transform: none;widows: 2;word-spacing: 0px;-webkit-text-stroke-width: 0px;white-space: normal;background-color: rgb(255, 255, 255);text-decoration-thickness: initial;text-decoration-style: initial;text-decoration-color: initial;">创建一个名为myenv的新环境，并指定 Python 版本为 3.10。你可以根据需要安装其他包。ingFang SC", -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Ubuntu, "Helvetica Neue", Helvetica, Arial, "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", "Source Han Sans CN", sans-serif;font-optical-sizing: inherit;font-size-adjust: inherit;font-kerning: inherit;font-feature-settings: inherit;font-variation-settings: inherit;margin: 16px 0px 0px;padding: 0px;vertical-align: baseline;color: rgba(0, 0, 0, 0.9);letter-spacing: -0.16px;orphans: 2;text-align: start;text-indent: 0px;text-transform: none;widows: 2;word-spacing: 0px;-webkit-text-stroke-width: 0px;white-space: normal;background-color: rgb(255, 255, 255);text-decoration-thickness: initial;text-decoration-style: initial;text-decoration-color: initial;">ingFang SC", -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Ubuntu, "Helvetica Neue", Helvetica, Arial, "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", "Source Han Sans CN", sans-serif;font-optical-sizing: inherit;font-size-adjust: inherit;font-kerning: inherit;font-feature-settings: inherit;font-variation-settings: inherit;margin: 16px 0px 0px;padding: 0px;vertical-align: baseline;color: rgba(0, 0, 0, 0.9);letter-spacing: -0.16px;orphans: 2;text-align: start;text-indent: 0px;text-transform: none;widows: 2;word-spacing: 0px;-webkit-text-stroke-width: 0px;white-space: normal;background-color: rgb(255, 255, 255);text-decoration-thickness: initial;text-decoration-style: initial;text-decoration-color: initial;">7.3、激活环境ingFang SC", -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Ubuntu, "Helvetica Neue", Helvetica, Arial, "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", "Source Han Sans CN", sans-serif;font-optical-sizing: inherit;font-size-adjust: inherit;font-kerning: inherit;font-feature-settings: inherit;font-variation-settings: inherit;margin: 16px 0px 0px;padding: 0px;vertical-align: baseline;color: rgba(0, 0, 0, 0.9);letter-spacing: -0.16px;orphans: 2;text-align: start;text-indent: 0px;text-transform: none;widows: 2;word-spacing: 0px;-webkit-text-stroke-width: 0px;white-space: normal;background-color: rgb(255, 255, 255);text-decoration-thickness: initial;text-decoration-style: initial;text-decoration-color: initial;">激活名为myenv的环境，之后安装的包将只在该环境中生效。ingFang SC", -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Ubuntu, "Helvetica Neue", Helvetica, Arial, "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", "Source Han Sans CN", sans-serif;font-optical-sizing: inherit;font-size-adjust: inherit;font-kerning: inherit;font-feature-settings: inherit;font-variation-settings: inherit;margin: 16px 0px 0px;padding: 0px;vertical-align: baseline;color: rgba(0, 0, 0, 0.9);letter-spacing: -0.16px;orphans: 2;text-align: start;text-indent: 0px;text-transform: none;widows: 2;word-spacing: 0px;-webkit-text-stroke-width: 0px;white-space: normal;background-color: rgb(255, 255, 255);text-decoration-thickness: initial;text-decoration-style: initial;text-decoration-color: initial;">7.4、退出/停用当前环境退出当前环境，回到 base（基础）环境或系统环境。7.5、删除环境condaremove--namemyenv--all 删除名为myenv的环境及其所有包。7.6、安装包
condainstallnumpycondainstallnumpypandasmatplotlib 7.7、更新包
更新当前环境中的numpy包到最新版本。
|
