CLI · Python

simple-markdown-crawler

In simple terms

Turns an entire website into Markdown files, one per page, keeping tables and images. Ideal for preparing material to use with AI for search or summaries.

Technical details

RAG fork of markdown_crawler in Python with BeautifulSoup and markdownify. Configurable multithreading, depth control, path/domain filters and CSS selectors (target_content/target_links) with resume. Dual use: CLI and md_crawl() library for OpenWebUI/autogen pipelines.

PythonuvBeautifulSoupmarkdownify

Highlights

Multithreaded with resume and depth control
Domain/path filters and CSS selectors
CLI + md_crawl() library ready for chunking

Install

pip install simple-markdown-crawler

Lines of code

1,970

Python

100%

Last recorded activity 5 commits · 52 sem

Last recorded activity in the project.