|
ingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;letter-spacing: 0.544px;text-wrap: wrap;border-style: none none solid;border-width: 1px 1px 4px;border-color: rgb(0, 0, 0) rgb(0, 0, 0) rgb(160, 249, 176);border-radius: 4px 4px 0px 0px;background: none 0% 0% / auto no-repeat scroll padding-box border-box rgb(24, 26, 33);width: auto;height: auto;align-items: unset;box-shadow: none;flex-direction: unset;justify-content: unset;line-height: 1.5em;overflow: unset;text-shadow: none;transform: none;-webkit-box-reflect: unset;visibility: visible;">作者：lucas大叔原文地址：https://zhuanlan.zhihu.com/p/693407124 当地时间4月18日，Meta在官网上公布了旗下最新大模型Llama 3。目前，Llama 3已经开放了 8B和 70B 两个小参数版本，上下文窗口为8k。Meta表示，通过使用更高质量的训练数据和指令微调，Llama 3比Llama 2有了“显著提升”。 Llama 3 instruction-tuned 模型针对对话/聊天场景进行了微调和优化，在通用基准测试上优于许多开源聊天模型。
开源社区对此反应迅速，ollama和LlamaIndex第一时间宣称完成了对Llama 3的支持，langchain也宣称可在LangSmith Playground中试用最新的Llama 3 8B 和 70B模型。下面我们展示如何在ollama和LlamaIndex中使用Llama 3模型。
Ollama使用指南CLI打开终端执行 ollama run llama3 即可 APIcurl使用示例: curl -X POST http://localhost:11434/api/generate -d '{
"model": "llama3",
"prompt":"Why is the sky blue?"
}'
其他接口示例参见API 文档。 Model variantsInstruct是针对聊天/对话场景微调的版本 Example:ollama run llama3ollama run llama3:70b Pre-trained是基础模型 Example:ollama run llama3:textollama run llama3:70b-text LlamaIndex使用指南本指南以Llama-3-8B-Instruct为例，说明如何基于llamaindex使用Llama3。 安装包!pip install llama-index
!pip install llama-index-llms-huggingface
!pip install llama-index-embeddings-huggingface
为了使用官方仓库的llama3，你需要授权你的huggingface账号并使用你的huggingface token。 hf_token="hf_..." 设置 Tokenizer 和 Stopping idsfrom transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(
"meta-llama/Meta-Llama-3-8B-Instruct",
token=hf_token,
)
stopping_ids = [
tokenizer.eos_token_id,
tokenizer.convert_tokens_to_ids("<|eot_id|>"),
]
用HuggingFaceLLM设置LLM可选择加载全精度或4bit量化版本。 # generate_kwargs parameters are taken from https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct
import torch
from llama_index.llms.huggingface import HuggingFaceLLM
# Optional quantization to 4bit
# import torch
# from transformers import BitsAndBytesConfig
# quantization_config = BitsAndBytesConfig(
# load_in_4bit=True,
# bnb_4bit_compute_dtype=torch.float16,
# bnb_4bit_quant_type="nf4",
# bnb_4bit_use_double_quant=True,
# )
llm = HuggingFaceLLM(
model_name="meta-llama/Meta-Llama-3-8B-Instruct",
model_kwargs={
"token": hf_token,
"torch_dtype": torch.bfloat16,# comment this line and uncomment below to use 4bit
# "quantization_config": quantization_config
},
generate_kwargs={
"do_sample": True,
"temperature": 0.6,
"top_p": 0.9,
},
tokenizer_name="meta-llama/Meta-Llama-3-8B-Instruct",
tokenizer_kwargs={"token": hf_token},
stopping_ids=stopping_ids,
)
## You can deploy the model on HF Inference Endpoint and use it
# from llama_index.llms.huggingface import HuggingFaceInferenceAPI
# llm = HuggingFaceInferenceAPI(
# model_name="",
# token=''
# )
用prompt调用completeresponse = llm.complete("Who is Paul Graham?")
print(response)
用messages列表调用chatfrom llama_index.core.llms import ChatMessage
messages = [
ChatMessage(role="system", content="You are CEO of MetaAI"),
ChatMessage(role="user", content="Introduce Llama3 to the world."),
]
response = llm.chat(messages)
print(response)
用Llama3创建RAG pipeline!wget"https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt""paul_graham_essay.txt" from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
documents = SimpleDirectoryReader(
input_files=["paul_graham_essay.txt"]
).load_data()
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-small-en-v1.5")
from llama_index.core import Settings
# bge embedding model
Settings.embed_model = embed_model
# Llama-3-8B-Instruct model
Settings.llm = llm
index = VectorStoreIndex.from_documents(
documents,
)
query_engine=index.as_query_engine(similarity_top_k=3) response=query_engine.query("Whatdidpaulgrahamdogrowingup?")参考文献llama3https://ollama.com/library/llama3 https://docs.llamaindex.ai/en/latest/examples/cookbooks/llama3_cookbook/ | 
