Deepcrawl: What It Is and How It Works
Deepcrawl explained simply: discover what it does, which version you need, and how to use it without the usual confusion.
Deepcrawl currently refers to two different products. DeepCrawl was the former name of Lumar, an enterprise website-crawling and website-intelligence platform; separately, Deepcrawl.dev is a newer open-source toolkit that turns web pages into clean, structured data for AI agents and applications.
If you’re trying to audit a large website, you’re probably looking for Lumar. If you’re a developer trying to fetch webpages, extract links, or convert web content into agent-friendly Markdown, you’re probably looking for Deepcrawl.dev.
Why Deepcrawl Is Confusing in 2026
Type “Deepcrawl” into a browser today and you can easily end up researching the wrong product.
The confusion comes from a name change and a newer project using the same name. The original DeepCrawl changed its name to Lumar in September 2022, while Deepcrawl.dev now describes itself as a free, open-source toolkit for making website data easier for AI agents and applications to consume.
That distinction matters because the two tools solve very different problems.
One is built around large-scale website analysis, monitoring, and automated quality assurance. The other is built around extracting useful web content and structure through developer-friendly APIs and SDKs.
DeepCrawl is now Lumar; Deepcrawl.dev is a separate, newer open-source project.
What Was DeepCrawl?
DeepCrawl was a cloud-based website crawler designed to help teams understand large websites and identify technical problems across many URLs.
In 2022, its company announced that DeepCrawl was becoming Lumar. The rebrand reflected an expansion from a primarily technical website-crawling product into a broader website-intelligence platform. The underlying crawler remained a central part of the product.
Today, Lumar’s platform includes Analyze, Monitor, Protect, and Impact. Analyze focuses on crawling and identifying website issues; Monitor tracks changes; Protect provides automated testing; and Impact helps teams communicate and assess the effect of website work.
So if an older tutorial tells you to “log into DeepCrawl,” don’t assume the service disappeared. In most cases, you’re looking at documentation for what is now Lumar.
What Does Lumar Do?
Lumar is designed for organizations that need to inspect websites at considerable scale rather than manually check individual pages.
Its crawler can examine technical website data and produce reports covering areas such as technical website health, site speed, accessibility, and custom metrics. Current Lumar documentation also describes functionality for AI-search-related analysis.
Analyze Large Websites
The Analyze product is the closest modern equivalent to what people traditionally associated with DeepCrawl.
It can crawl websites, segment areas of a site, identify technical problems, and provide reporting designed to help teams prioritize issues. Lumar says Analyze supports hundreds of built-in reports and custom data analysis.
This becomes particularly useful on websites with hundreds of thousands or millions of URLs, where manually checking pages isn’t realistic.
Monitor Changes Over Time
A single website audit only tells you what is happening at one point in time.
Lumar Monitor adds an ongoing layer by tracking selected domains or sections and generating alerts when specified problems or changes appear. That makes it more useful for teams managing websites that change frequently.
Protect Websites Before Release
One particularly practical distinction is Lumar Protect.
Instead of waiting for a broken page, redirect, or other website issue to reach production, teams can test changes before deployment. Lumar says Protect can integrate with CI/CD pipelines and use thresholds that either warn developers or stop a build.
Think of it as a spell-checker for website releases: finding mistakes before everyone else has to deal with them.
What Is Deepcrawl.dev?
Deepcrawl.dev is a completely different project.
Its documentation describes it as an open-source, agent-oriented web page context extraction platform. Rather than primarily auditing a website for technical problems, it focuses on retrieving web content and structure in forms that applications and AI systems can process efficiently.
Its core capabilities include:
- Converting webpages into clean Markdown
- Reading URLs and returning richer page data
- Extracting links into a structured site tree
- Providing APIs and a Node.js SDK
- Supporting self-hosting
- Using caching to reduce repeated work
The project is also explicitly described as being in active development, meaning its APIs and SDKs can change.
Deepcrawl.dev is designed to turn messy web pages into structured context that software can actually use.
How Deepcrawl.dev Works
The easiest way to understand Deepcrawl.dev is to imagine giving an application a webpage and asking it to remove everything that isn’t useful to the application.
A normal webpage contains navigation, styling, scripts, advertisements, layout elements, tracking code, and other material surrounding the actual information. Deepcrawl.dev’s tools can transform that page into cleaner representations.
Get Markdown
The getMarkdown endpoint takes a URL and returns cleaned Markdown.
That’s useful when an application needs the actual text of a webpage rather than its full HTML structure. The official documentation specifically positions this endpoint for prompt-ready snippets, cached retrieval, and applications that need a lightweight representation of page content.
For example, a developer building a research assistant could retrieve an article as Markdown and pass the resulting text into another processing step without first writing a custom HTML cleaner.
Read a URL
The readUrl operation is intended for situations where Markdown alone isn’t enough.
According to the documentation, the richer response can include metadata, cleaned HTML, Markdown, robots information, sitemap data, and metrics.
That makes it more suitable when an application needs both the content and context surrounding that content.
Extract Links
The extractLinks endpoint takes a deeper approach by building a structured link tree from a page.
Importantly, the documentation says this process works by parsing the actual HTML rather than depending on sitemap.xml or robots.txt to discover links. That can be valuable when an application needs to understand how pages are connected.
Imagine giving an agent the homepage of a documentation site and asking it to understand the site’s structure. A link tree can provide a much more useful starting point than one isolated page.
Deepcrawl vs. Lumar at a Glance
| Feature | Deepcrawl.dev | Lumar, formerly DeepCrawl |
| Primary purpose | Web data extraction | Website intelligence and analysis |
| Main audience | Developers and AI application builders | Enterprise digital, development, and website teams |
| Open source | Yes | No |
| Main output | Markdown, page data, link trees | Audits, reports, monitoring, QA, dashboards |
| API/SDK focus | Central to the product | API and integrations support the wider platform |
| Website monitoring | Not its primary purpose | Yes |
| CI/CD website QA | Not its primary focus | Yes |
| Large-scale website auditing | Not its primary purpose | Yes |
| Self-hosting | Supported | Cloud platform |
| Current status | Active early-stage development | Established commercial platform |
The key point is that these aren’t really competing versions of the same product. They sit at different layers of the web technology stack.
Which Deepcrawl Should You Use?
If your question is, “How do I find technical problems across a huge website?” look at Lumar, the product formerly known as DeepCrawl.
If your question is, “How do I turn webpages into clean information that my application or AI agent can consume?” look at Deepcrawl.dev.
For a developer building an AI-powered research workflow, Deepcrawl.dev’s Markdown and link-extraction APIs are particularly relevant. Its Node.js documentation provides examples for getMarkdown(), readUrl(), and extractLinks(), along with examples for frameworks such as Next.js and Hono.
For an enterprise team responsible for a large commercial website, Lumar’s Analyze, Monitor, and Protect products address a broader operational problem: finding issues, tracking them, and preventing them from returning.
What Deepcrawl Cannot Tell You
A crawler is powerful, but it isn’t a crystal ball.
For example, finding a page in a crawl does not automatically mean that Google will treat that page exactly as your crawler did. Google processes websites through separate crawling, rendering, and indexing stages, and JavaScript can change what becomes visible during rendering.
This is especially important for JavaScript-heavy applications.
A crawler may retrieve an initial HTML response while a browser later generates additional content, links, or elements through JavaScript. Google explains that its systems render JavaScript using an evergreen version of Chromium, but also notes that JavaScript implementations can still create limitations.
That means crawler data should be treated as evidence about a website’s behavior, not as an unquestionable replica of every external system’s behavior.
Common Deepcrawl Mistakes to Avoid
Confusing DeepCrawl With Deepcrawl.dev
This is the biggest mistake in 2026.
A tutorial published before 2022 may be talking about DeepCrawl, while a current developer tutorial using the lowercase spelling may be referring to Deepcrawl.dev. Always check the domain and documentation before following installation or pricing instructions.
Assuming a Crawl Equals a Diagnosis
A crawl produces observations. Someone still needs to interpret them.
For example, discovering hundreds of duplicate URLs doesn’t automatically tell you which URLs should change. You need to understand why the duplicates exist, whether they serve a legitimate purpose, and what the desired architecture should be.
Treating JavaScript Rendering as a Perfect Browser Simulation
JavaScript rendering is useful, but different crawlers and platforms can process pages differently.
Google itself describes a pipeline involving crawling, rendering, and indexing, and notes that resources can fail to load or behave differently depending on how a page is implemented.
Ignoring the Server
Sometimes the apparent website problem isn’t in the HTML at all.
Slow responses, intermittent 5xx errors, DNS failures, blocked resources, or restrictive robots rules can prevent crawlers from seeing what you expect. Google specifically identifies server, network, and robots.txt issues as factors affecting crawling.
A Practical Way to Choose
Use this simple decision path:
- Need enterprise-scale website auditing? Start with Lumar.
- Need continuous website monitoring? Look at Lumar Monitor.
- Need automated checks before code reaches production? Look at Lumar Protect.
- Need webpage-to-Markdown conversion? Look at Deepcrawl.dev.
- Need a structured map of links? Look at Deepcrawl.dev’s link extraction.
- Need both content extraction and metadata? Use Deepcrawl.dev’s richer URL-reading functionality.
The distinction becomes much clearer once you stop treating “Deepcrawl” as one product.
FAQ
Is Deepcrawl the same as DeepCrawl?
No. DeepCrawl, the former website-crawling brand, became Lumar in 2022. Deepcrawl.dev is a separate open-source project focused on extracting and structuring web content for applications and AI agents.
What happened to DeepCrawl?
DeepCrawl rebranded as Lumar in September 2022. Its website-crawling technology remained central to the company’s platform, which expanded into website intelligence, monitoring, automated QA, accessibility, and related capabilities.
Is Deepcrawl.dev open source?
Yes. Deepcrawl.dev describes itself as a free, open-source, open-code toolkit. Its documentation also warns that the project is under active development and that APIs and SDKs may change.
What can Deepcrawl.dev extract?
Its documented functionality includes clean Markdown, page information, metadata, links, and structured link trees. The platform provides endpoints such as getMarkdown, readUrl, and extractLinks.
Can a crawler replace Google’s own testing tools?
No. A third-party crawler can reveal valuable technical information, but Google’s own documentation recommends tools such as URL Inspection and the Rich Results Test when you need to understand how Google processes a particular page.
Key Takeaways
- DeepCrawl is now Lumar, following the company’s 2022 rebrand.
- Deepcrawl.dev is a separate open-source project, focused on turning web content into structured, application-friendly data.
- Lumar is aimed at large-scale website analysis, monitoring, and automated quality assurance.
- Deepcrawl.dev provides tools such as Markdown extraction, URL reading, and structured link extraction.
- JavaScript rendering can reveal content that isn’t present in the initial HTML, but different crawlers can still observe different things.
- A crawl gives you evidence and data, not automatic answers; human interpretation remains important.
- When researching “Deepcrawl,” check the domain first. That one habit prevents most of the confusion surrounding the name.
Additional Resources
- Lumar official rebrand announcement: Explains exactly why DeepCrawl became Lumar and what happened to its original crawling platform.
