PorticanBot
PorticanBot is the crawler of the Portican platform. Stores on Portican use it to track the prices of similar products at competitors they add themselves in the admin.
What it reads
Only public product pages that the website allows robots to access in robots.txt, in the PorticanBot or * group, following RFC 9309 and the longest match rule. From those pages it takes only the product name, price, availability and dimensions from JSON-LD, microdata and OpenGraph structured data.
What it does not do
It does not log in, fill in forms, get around CAPTCHA or bot protection, or run JavaScript.
How often it visits
It sends one request at a time to a website, waits at least 5 seconds between requests and visits at night. It respects Crawl-delay and skips a website with a value above 60 seconds. On a 429 or 503 response it stops and comes back later.
How to stop it
Add a group for PorticanBot to the robots.txt file on your website that disallows the whole site:
User-agent: PorticanBot
Disallow: /Text and data mining reservation
PorticanBot respects a reservation made in the HTTP header or in the meta tag tdm-reservation: 1 (TDMRep), under § 51c of the Slovak Copyright Act and Article 4 of Directive 2019/790.
Send questions about the crawler to info@portican.com.