hatang.dev
Products
Customer SupportWeb, app, email, Zalo OA, personal Zalo, Messenger, Instagram, WhatsApp, Telegram, X, LINE, TikTok and TikTok Shop in one inboxEmailA Resend-compatible email API, sending from your own domains
PricingDocsBlogHelpAboutGet the app
Sign in
VI

HatangBot

HatangBot is the crawler of Ha Tang (hatang.dev). It reads a website only when a business on Ha Tang adds that site's address as a knowledge source for its own AI support assistant.

Last updated 11 November 2026

1. What it does

A business that uses Ha Tang's AI assistant can add web addresses (its own help pages, product pages, policies) as knowledge sources. HatangBot fetches those pages and turns them into text that the assistant cites when it answers that business's customers. The content is used as reference for answers; it is not used to train AI models.

2. How it crawls

Every crawl starts from an address a person added. HatangBot stays on that host and under the starting path, and is gentle by design:

  • At most 50 pages per source, following links up to two levels deep, one page at a time.
  • Sitemaps listed in robots.txt (or /sitemap.xml) are used to find pages.
  • A source is read again when its refresh period elapses (7 days by default) or when the business asks for it.
  • Pages that fill in their content with JavaScript may be loaded once in a headless browser with the same User-Agent.
  • Each page fetch times out after 15 seconds.

3. Identifying HatangBot

Requests carry the User-Agent HatangBot/1.0 (+https://hatang.dev/bot). They are also signed with Web Bot Auth (HTTP Message Signatures, Ed25519): each request has Signature-Agent: "https://hatang.dev", Signature-Input and Signature headers over @authority and signature-agent, valid for one minute. The public keys are published at https://hatang.dev/.well-known/http-message-signatures-directory. A request that claims to be HatangBot without a valid signature may not be ours.

4. Controlling access

HatangBot obeys robots.txt. Rules for the HatangBot user agent apply first, otherwise the rules for *. Crawl-delay is honoured up to 2 seconds. Pages marked noindex (meta robots or X-Robots-Tag) are skipped, and links on nofollow pages are not followed.

  • Block it everywhere: User-agent: HatangBot then Disallow: /
  • Allow it through a firewall: let requests with a valid HatangBot signature, or with HatangBot in the User-Agent, skip bot challenges.

5. Contact

Questions or a problem with HatangBot's traffic: [email protected]. Include your domain and the time of the requests.