About the INSIGHTEX web crawler

How the INSIGHTEX research crawler identifies itself, which sites it reads, how it follows robots.txt and how to opt out.

Updated November 21, 2026

Who we are

INSIGHTEX is a global directory and news service for startups, funding, events, programs and deals. Our research crawler, WebResearchCrawler, reads public web pages so the information we publish can be checked against the official source.

How it identifies itself

Every request carries this user agent: Mozilla/5.0 (compatible; WebResearchCrawler/1.0; +https://www.insightex.si/p/crawler). The product token to use in robots.txt is WebResearchCrawler.

What it reads and why

  • Public pages of companies, programs, events, funding providers and deal offers, to confirm details and find the official page to link to.
  • Public news pages and feeds, to credit and link the original source.

robots.txt and politeness

Before reading a page the crawler fetches the site's robots.txt and follows it, including a Crawl-delay when one is set. If robots.txt cannot be read or does not allow the page, the page is not fetched. Requests to the same site are spaced out so the site is not loaded heavily.

How to opt out

Add the lines below to your robots.txt and the crawler stops reading your site:

User-agent: WebResearchCrawler

Disallow: /

You can also write to hello@insightex.si with the address of your site and we will exclude it.

Contact

Questions about the crawler, or a request that it behave differently on your site: hello@insightex.si.