About the INSIGHTEX web crawler
How the INSIGHTEX research crawler identifies itself, which sites it reads, how it follows robots.txt and how to opt out.
Who we are
INSIGHTEX is a global directory and news service for startups, funding, events, programs and deals. Our research crawler, WebResearchCrawler, reads public web pages so the information we publish can be checked against the official source.
How it identifies itself
Every request carries this user agent: Mozilla/5.0 (compatible; WebResearchCrawler/1.0; +https://www.insightex.si/p/crawler). The product token to use in robots.txt is WebResearchCrawler.
What it reads and why
- Public pages of companies, programs, events, funding providers and deal offers, to confirm details and find the official page to link to.
- Public news pages and feeds, to credit and link the original source.
robots.txt and politeness
Before reading a page the crawler fetches the site's robots.txt and follows it, including a Crawl-delay when one is set. If robots.txt cannot be read or does not allow the page, the page is not fetched. Requests to the same site are spaced out so the site is not loaded heavily.
How to opt out
Add the lines below to your robots.txt and the crawler stops reading your site:
User-agent: WebResearchCrawler
Disallow: /
You can also write to hello@insightex.si with the address of your site and we will exclude it.
Contact
Questions about the crawler, or a request that it behave differently on your site: hello@insightex.si.