My Canary settings changes are getting stuck

You are also welcome to reach out to the support team to help identify and correct this.

What's happening?

Your Canaries chat to your Console using DNS tunneling. This channel covers settings changes, updates and alert notifications. If DNS traffic in your environment is filtered (by a firewall, DNS filter, or security product), the Canary might show as online (traffic can leave) but won't be able to apply new settings or update. (Receive DNS responses.)

Sometimes only part of the traffic gets through, enough for the device to check in, but not enough to fetch and apply new config. That's when you'll see the Settings Push getting stuck.

A healthy Canary will have settings pushes complete in a couple seconds, and it's "last seen" timer will reliably reset every 30 seconds indicating it's heartbeats are received successfully.

Signs you're running into this

A Canary with a degraded tunnel or that has it's traffic filtered will see delays in it's settings changes, or updates that take days to complete. It's "last seen" timer will also exceed the 30 second reset marker.

What causes it

Your Canary's DNS query traffic will flow to your preferred DNS server, where it will recursively look up the DNS chain. Through this, it may traverse one or more firewalls. While the traffic leaves the network, responses from your Console don't make their way back.

Firewalls or security products filtering DNS like Cisco Umbrella, Palo Alto, OpenDNS, Zscaler etc can all filter this traffic, causing the trouble.

What to do?

Check for DNS filtering/Blocking

• Check your firewall logs for blocked DNS requests to the <your-console-hash>.cnr.io domain.
  • Your Console hash is the subdomain of you Console URL
    • e.g. For https://abc1234.canary.tools — abc1234 will be your Console hash.
• We have some helpful guides on exempting your Canary domain for Cisco Umbrella here, Palo Alto here and OpenDNS here.

Test DNS Communication

Run these commands from a machine on the same subnet as the Canary. Replace the DNS server IP and <your-console-hash> as needed:

nslookup -q=TXT test.250.prb.<your-console-hash>.cnr.io <your-DNS-server>

nslookup -q=TXT FORBOOYXFDC7KS72P6RW4ZWALRNBEOHFDTSNKR7TIL7C2UQT2ZTMPUBHHUDMOC.3H5UAZEYXFDC7KS72P6RW4ZWALRNBEOHFDTSNKR7TIL7C2UQT2ZTMPUBHHUDMOC.3H5UAZEYXFDC7KS72P6RW4ZWALRNBEOHFDTSN.250.prb.<your-console-hash>.cnr.io <your-DNS-server>

A good response will return a large text block back:

% nslookup -q=TXT test.250.prb.6b42426d.cnr.io 1.1.1.1
Server:		1.1.1.1
Address:	1.1.1.1#53

Non-authoritative answer:
test.250.prb.6b42426d.cnr.io	text = "test.250.prb.6b42426d.cnr.iotest.250.prb.6b42426d.cnr.iotest.250.prb.6b42426d.cnr.iotest.250.prb.6b42426d.cnr.iotest.250.prb.6b42426d.cnr.iotest.250.prb.6b42426d.cnr.iotest.250.prb.6b42426d.cnr.iotest.250.prb.6b42426d.cnr.iotest.250.prb.6b42426d.cnr."

Some common bad responses can be found below and indicate the server is not suitable for Canary DNS queries:

** server can't find test.250.prb.6b42426d.cnr.io: NXDOMAIN
** server can't find test.250.prb.6b42426d.cnr.io: SERVFAIL
** server can't find test.250.prb.6b42426d.cnr.io: REFUSED
*** No TXT record found for test.250.prb.6b42426d.cnr.io

Try a different DNS temporarily

In some cases where the DNS filter doesn't affect public DNS servers (like Cloudflare's 1.1.1.1), you can temporarily set your Canary to use another server.

Because we can't get responses to the Bird, we won't be able to do this via the Canary Console and instead need to make these changes locally.

We offer guides on doing this for Hardware, Virtual appliances and Cloud Canaries.