Skip to main content

Crawler reference

Allow VisualSentinelBot through your firewall

If a Visual Sentinel customer monitors your website, these are the requests you will see. Allow the addresses below in your firewall or bot protection so the checks get through, including screenshot checks, which look like an ordinary Chrome browser.

At a glance

Addresses
  • 157.180.59.176EU region (Helsinki)
  • 65.108.39.15EU additional IPv4 (provisioned)
  • 2a01:4f9:3100:46d8::/64EU region (Helsinki), IPv6 range
  • 5.78.139.24US region
  • 2a01:4ff:1f0:b102::1US region, IPv6
  • 187.77.118.182Asia region (Jakarta)
  • 2a02:4780:59:7809::1Asia region (Jakarta), IPv6
Uptime and API checks
VisualSentinelBot/2.0 (+https://visualsentinel.com/bot)
Content checks
VisualSentinel/1.0 ContentMonitor

The additional EU IPv4 address 65.108.39.15 is provisioned and listed for allowlisting before use. Outbound checks from this address have not yet been verified.

Apart from performance checks, which Google runs, every request we send to your website comes from these addresses, so allow the addresses. The User-Agents help you recognise our requests in your logs, but anyone can copy a User-Agent, and screenshot checks send a normal Chrome one.

What each check looks like

A customer can turn on several kinds of check for the same site. Each one reaches you differently.

How each Visual Sentinel check identifies itself
CheckSent fromIdentifies asAllow it by
Uptime and APIOur addressesVisualSentinelBot/2.0. An API check sends any User-Agent its owner sets instead.Address
ContentOur addressesVisualSentinel/1.0 ContentMonitorAddress
ScreenshotsOur addressesA normal Chrome browserAddress
SSL certificateOur addressesA secure connection only, with no page requestAddress
PerformanceGoogle, daily and when the owner runs a testGoogle PageSpeed Insights (Lighthouse)Your rules for Google's tools
DNSPublic DNS resolversDNS lookups. Nothing reaches your web server.Nothing to allow

How often we check

  • Each monitor runs on the schedule its owner picks, from once a minute to once a day, from each region they turn on.
  • Scheduled screenshots run at most once every 15 minutes per monitor. The owner can also take one by hand, and a large change is confirmed with a second screenshot straight away.
  • When an uptime check fails, we check again about a minute later, and keep doing so until the failure is confirmed or clears.
  • Uptime and content checks follow redirects, up to five by default, including to another domain. A screenshot check follows redirects the way a browser does.
  • A screenshot check loads the page the way a browser does, including its images, scripts and styles from other services.
  • Checks request only the address they were set up for and don't follow links. A screenshot check fills in a form only when its owner sets up a login for it, with credentials the owner provides. An API check sends exactly the request its owner set up, which can include a method, headers and a body.
  • Checks don't read robots.txt. Each one was set up by a customer who wants that address watched, usually the site's owner or their agency.

Cloudflare

Which rule works depends on your Cloudflare plan.

Free plan (Bot Fight Mode)

Bot Fight Mode ignores custom rules, including Skip rules. It does not run when an IP access rule matches first, so allow each address:

  1. Go to Security rules and select Create rule, then IP access rules.
  2. Enter one of our addresses (or the IPv6 range), set the action to Allow, and apply it to this website.
  3. Repeat for each address and range.

Pro plan and above

Create a custom rule with the action Skip and choose All Super Bot Fight Mode rules, plus the managed rules and rate limiting rules if you use them. Paste this expression, which matches our addresses:

Cloudflare rule expression

(ip.src in {157.180.59.176 65.108.39.15 2a01:4f9:3100:46d8::/64 5.78.139.24 2a01:4ff:1f0:b102::1 187.77.118.182 2a02:4780:59:7809::1})

Once the rule is live, screenshot checks should start succeeding within a few minutes. To confirm, open Security, then Events, and filter by our addresses.

AWS WAF

AWS WAF keeps IPv4 and IPv6 addresses in separate IP sets, so create one of each with our addresses:

IPv4 IP set

157.180.59.176/32
65.108.39.15/32
5.78.139.24/32
187.77.118.182/32

IPv6 IP set

2a01:4f9:3100:46d8::/64
2a01:4ff:1f0:b102::1/128
2a02:4780:59:7809::1/128

Then add this rule to your web ACL in the JSON rule editor. Replace the two ARNs with your IP sets' ARNs, and give it a priority that runs before your bot control and rate-based rules. Allow stops the rest of the web ACL for these requests, which is why it matches only our addresses.

AWS WAF allow rule

{
  "Name": "allow-visual-sentinel",
  "Priority": 0,
  "Action": {
    "Allow": {}
  },
  "Statement": {
    "OrStatement": {
      "Statements": [
        {
          "IPSetReferenceStatement": {
            "ARN": "YOUR_VISUAL_SENTINEL_IPV4_SET_ARN"
          }
        },
        {
          "IPSetReferenceStatement": {
            "ARN": "YOUR_VISUAL_SENTINEL_IPV6_SET_ARN"
          }
        }
      ]
    }
  },
  "VisibilityConfig": {
    "SampledRequestsEnabled": true,
    "CloudWatchMetricsEnabled": true,
    "MetricName": "allow-visual-sentinel"
  }
}

Nginx

To keep our checks out of a request rate limit, give our addresses an empty limit key. Nginx does not count requests whose key is empty. Put this in the http block and use zone=per_ip in your limit_req line.

Nginx rate limit exception

geo $visual_sentinel {
    default                 0;
    157.180.59.176          1;
    65.108.39.15            1;
    2a01:4f9:3100:46d8::/64 1;
    5.78.139.24             1;
    2a01:4ff:1f0:b102::1    1;
    187.77.118.182          1;
    2a02:4780:59:7809::1    1;
}

map $visual_sentinel $limit_key {
    1 "";
    0 $binary_remote_addr;
}

limit_req_zone $limit_key zone=per_ip:10m rate=10r/s;

If you block by address instead, add allow lines for 157.180.59.176, 65.108.39.15, 2a01:4f9:3100:46d8::/64, 5.78.139.24, 2a01:4ff:1f0:b102::1, 187.77.118.182 and 2a02:4780:59:7809::1 above your deny rules.

Report unexpected traffic

If you think VisualSentinelBot is checking your site without anyone having asked for it, email reportabuse@visualsentinel.com with the domain, a few sample requests and roughly how often they arrive. We will look into it within one business day.

Monitor your own site

Add your site and see its first check. Then try a paid plan for 7 days (card required) or subscribe, from $15/month.

Start 7-day trial