KapiwayBot

KapiwayBot reads public web pages for Kapiway, a tool that helps businesses show up on Google and in AI answers. It reads a business's own website for that business, and the public pages of sites that business works with: the competitors it names, the guides and publishers in its field, directories that list it, and the sites it may ask for a link. It never logs in, never fills in forms and never posts anything. We never read Google search pages ourselves, and for a Google Maps short link we only read the address it redirects to, never the Maps page.

How it identifies itself

Every Kapiway request names Kapiway in its user agent, as one of three names, and never retries pretending to be a browser:

• KapiwayBot: competitor discovery (the homepage only), the contact finder (the homepage, which may be read twice, 15 contact-style paths and 1 page an AI model picks), the robots.txt and llms.txt files the AI readiness check looks at, and the broken-link checker (a HEAD request, and a GET only when the server doesn't accept HEAD).
• KapiwayRadar: the weekly competitor price watch, community threads that AI answers cite, and guide and publisher pages in the business's field (up to 24 pages a week; a page we read is read again at most once a month, and a page we couldn't read is tried again after a week).
• KapiwayAudit: reads of a website its owner added to Kapiway: the site check (up to 400 pages, about once a month), reads that learn what it offers and where its locations are, and the website read on kapiway.com/start (about 30 pages: robots.txt, the sitemaps and homepage, up to 14 pages found there, up to 7 usual pages such as /about and /contact, up to 5 address pages, and the homepage once more; to find locations it may read robots.txt and the sitemaps a second time, and the homepage can be read a few times). The AI readiness check also reads as KapiwayAudit: robots.txt, the sitemaps, up to 6 pages, the homepage and the 7 usual pages. Google PageSpeed loads up to 3 of its pages.

We never ask for a picture of a page that refused our reader. When a page can't be read and didn't refuse us, we may ask DataForSEO for a picture of the public page, which its own browser loads.

Allow Kapiway

If Kapiway told you your site blocks bots, a firewall or bot protection is usually turning our reader away. Let these two user agents in: KapiwayAudit (reads of your own site in Kapiway: at set up, what you offer, where your locations are, the monthly site check and most of the AI readiness check) and KapiwayBot (the robots.txt and llms.txt files the AI readiness check looks at).

Cloudflare. Kapiway is not on Cloudflare's verified bots list, so add your own rule:

1. In the Cloudflare dashboard, open your domain, then Security, WAF, Custom rules, and choose Create rule.
2. Name it "Allow Kapiway". Set Field to User Agent, Operator to contains, Value to KapiwayAudit. Choose Or and add a second line: User Agent contains KapiwayBot.
3. Set the action to Skip, tick every option it offers (the remaining custom rules, rate limiting, managed rules and Super Bot Fight Mode where shown), and deploy.
4. Bot Fight Mode on the free plan cannot be skipped for one bot. If it is on, turn it off while we read your site, or write to us.

Other firewalls and hosts (Sucuri, Wordfence, your host's bot protection): add an allow rule for user agents that contain KapiwayAudit or KapiwayBot. robots.txt: KapiwayBot and KapiwayRadar follow it, so make sure no rule blocks them:

Then run the read again from Kapiway. Still blocked? Write to support@kapiway.com with your domain.

Keeping it off your site

Add rules for KapiwayBot to your robots.txt. These readers check robots.txt before they fetch a page: competitor discovery and the contact finder (KapiwayBot), and the weekly competitor watch and reads of community threads (KapiwayBot or KapiwayRadar rules). If your robots.txt answers with a server error (5xx), they treat it as allowing them. We review each directory by hand before we compare a listing there.

The other readers do not follow robots.txt: the site check and the /start read run on a site its owner gave us, the AI readiness check reads a site someone typed in to have checked, the broken-link checker only asks whether a link works, and a publisher page is read again at most once a month. To stop every Kapiway reader for your domain, write to support@kapiway.com with the domain, and we stop reading it by hand. More on the data we handle is in our privacy policy.