Level 1:Character Splitting-简单的字符长度切分
Level 2:Recursive Character Text Splitting-通过分隔符切分，然后递归合并
Level 3
ocument Specific Splitting- 针对不同文档格式切分(PDF, Python, Markdown)
Level 4:Semantic Splitting-语义切分
Level 5:Agentic Splitting-使用代理实现自动切分
Level 1:Character Splitting-简单的字符长度切分
Level 2:Recursive Character Text Splitting-通过分隔符切分，然后递归合并
Level 3ocument Specific Splitting- 针对不同文档格式切分(PDF, Python, Markdown)
Level 4:Semantic Splitting-语义切分
Level 5:Agentic Splitting-使用代理实现自动切分
分块通常旨在将具有共同上下文的文本放在一起。考虑到这一点，我们可能希望特别尊重文档本身的结构。例如，markdown 文件按标题组织。在特定标题组中创建块是一种直观的想法。为了解决这一挑战，我们可以使用MarkdownHeaderTextSplitter。这将按指定的一组标题拆分 markdown 文件。
本文用到的安装包如下：
pipinstalllangchain-text-splitters
markdown_document = "# Foo\n\n## Bar\n\nHi this is Jim\n\nHi this is Joe\n\n ### Boo \n\n Hi this is Lance \n\n ## Baz\n\n Hi this is Molly"headers_to_split_on = [("#", "Header 1"),("##", "Header 2"),("###", "Header 3"),]markdown_splitter = MarkdownHeaderTextSplitter(headers_to_split_on)md_header_splits = markdown_splitter.split_text(markdown_document)print(md_header_splits)
结果如下：
[Document(page_content='HithisisJim\nHithisisJoe',metadata={'Header1':'Foo','Header2':'Bar'}),Document(page_content='HithisisLance',metadata={'Header1':'Foo','Header2':'Bar','Header3':'Boo'}),Document(page_content='HithisisMolly',metadata={'Header1':'Foo','Header2':'Baz'})]markdown_splitter=MarkdownHeaderTextSplitter(headers_to_split_on,strip_headers=False)md_header_splits=markdown_splitter.split_text(markdown_document)print(md_header_splits)
[Document(page_content='#Foo\n##Bar\nHithisisJim\nHithisisJoe',metadata={'Header1':'Foo','Header2':'Bar'}),Document(page_content='###Boo\nHithisisLance',metadata={'Header1':'Foo','Header2':'Bar','Header3':'Boo'}),Document(page_content='##Baz\nHithisisMolly',metadata={'Header1':'Foo','Header2':'Baz'})]如何将 Markdown 行返回为单独的文档
默认情况下，MarkdownHeaderTextSplitter根据headers_to_split_on中指定的标题聚合行。我们可以通过指定return_each_line来禁用此功能，使得一行就是一条内容：
markdown_splitter=MarkdownHeaderTextSplitter(headers_to_split_on,return_each_line=True,)md_header_splits=markdown_splitter.split_text(markdown_document)print(md_header_splits)
[Document(page_content='HithisisJim',metadata={'Header1':'Foo','Header2':'Bar'}),Document(page_content='HithisisJoe',metadata={'Header1':'Foo','Header2':'Bar'}),Document(page_content='HithisisLance',metadata={'Header1':'Foo','Header2':'Bar','Header3':'Boo'}),Document(page_content='HithisisMolly',metadata={'Header1':'Foo','Header2':'Baz'})]如何限制块大小：
然后，我们可以在每个 markdown 组中应用任何我们想要的文本分割器，例如RecursiveCharacterTextSplitter，它允许进一步控制块大小。
markdown_document = "# Intro \n\n## History \n\n Markdown[9] is a lightweight markup language for creating formatted text using a plain-text editor. John Gruber created Markdown in 2004 as a markup language that is appealing to human readers in its source code form.[9] \n\n Markdown is widely used in blogging, instant messaging, online forums, collaborative software, documentation pages, and readme files. \n\n ## Rise and divergence \n\n As Markdown popularity grew rapidly, many Markdown implementations appeared, driven mostly by the need for \n\n additional features such as tables, footnotes, definition lists,[note 1] and Markdown inside HTML blocks. \n\n #### Standardization \n\n From 2012, a group of people, including Jeff Atwood and John MacFarlane, launched what Atwood characterised as a standardisation effort. \n\n ## Implementations \n\n Implementations of Markdown are available for over a dozen programming languages."headers_to_split_on = [("#", "Header 1"),("##", "Header 2"),]# MD splitsmarkdown_splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers_to_split_on, strip_headers=False)md_header_splits = markdown_splitter.split_text(markdown_document)# Char-level splitsfrom langchain_text_splitters import RecursiveCharacterTextSplitterchunk_size = 250chunk_overlap = 30text_splitter = RecursiveCharacterTextSplitter(chunk_size=chunk_size, chunk_overlap=chunk_overlap)# Splitsplits = text_splitter.split_documents(md_header_splits)splits
| 欢迎光临 链载Ai (https://www.lianzai.com/) | Powered by Discuz! X3.5 |