Network Connectivity Status Indicator (NCSI) Shows Offline

Posted by James F. Prudente on July 21, 2015
Posted in: Uncategorized. Leave a comment

ncsiNCSI, or the Network Connectivity Status Indicator, is that annoying icon in the system tray that insists on telling you that you have no internet access despite your being perfectly able to surf the web.

Normally this discrepancy doesn’t matter, but if you’re using Office 2013 and trying to connect to Office 365 it poses a problem, as the Office suite will not connect to cloud services if the PC thinks it’s offline.

Working out why NCSI is wrong requires understanding how it works – or at least is supposed to work. This Technet article explains its behavior in Vista, which is where NCSI was first introduced. Then we have this blog post, which is backed up by the author’s own packet traces. The blog post is quite good and does a better (and seemingly more accurate) job explaining NCSI than the Technet article, but there are a couple of points I’d like to clarify based on what we saw in our environment.

To very quickly summarize how NCSI determines connectivity status:

  • It first performs an HTTP request for http://www.msftncsi.com/ncsi.txt, then
  • Performs a DNS lookup for dns.msftncsi.com to confirm the proper IP address is returned.

This should typically work well in a simple home environment, but in an enterprise there are any number of points where the HTTP request can be intercepted: perimeter firewall, IDS/IPS, proxy, content filter, endpoint firewall, antivirus, etc. And if that request fails, Windows considers you to be disconnected from the Internet.

What kept throwing us off during our own troubleshooting of this problem was that while we saw the DNS queries for www.msftncsi.com as the system tried to resolve the webserver IP, we never saw the HTTP request go out. The Technet article indicates this should happen the first time a user logs on after a reboot, and both articles refer to it happening the first time a computer joins a new network.

That latter point seems to be correct, but it’s important to note the “new” part of that statement: a network connection coming online is not sufficient, rather the network detected has to be different. It’s not clear what constitutes different, but changing proxy settings appears to be sufficient; disconnecting the network cable is not.

As to the former statement: Perhaps in Vista the HTTP request happened when a user logged on, but on 7 and 8.1, this is not what we saw. Instead we only saw the HTTP request go out once, as the computer started up. To be clear, a user login was not triggering network detection, and we never saw that HTTP request go out again after the computer booted. That means – and this is the important part – that if that initial HTTP request during boot fails, the system will think you are disconnected from the Internet and will never discover otherwise.

Remember all those possible applications that can interfere with the HTTP request? If any of them are keyed to only permit traffic based on the logged on user, you may have an issue, since the HTTP request will almost always be sent before anyone is logged on. That was our problem; we needed to allow all traffic to www.msftncsi.com even if nobody was logged on to a PC.

There’s one other caveat to be aware of: the HTTP request comes through winhttp, which does not inherit IE’s proxy settings. So if you’re using an explicit proxy without WPAD, you will need to manually configure winhttp with your proxy settings.

Other than the clarification those two points, the blog post linked earlier seems correct and is a good reference if you’re looking to change any other NCSI settings. Hopefully you won’t have to.

Update 9-27-16: While working on an unrelated problem, I found our Win 10 machines are reaching out to http://www.msftconnecttest.com. I don’t have the time right now to test this but on the surface it seems this is yet another online check that MS has started using. I recommend allowing traffic to this site as well.

Overloading an IP Address with PAT – Cisco ASA 8.4

Posted by James F. Prudente on April 20, 2015
Posted in: ASA, Cisco. Tagged: , . Leave a comment

I recently ran into the need to overload a public IP address, using port address translation (PAT) to allow multiple internal devices to share one public IP. Surprisingly, it was a lot harder finding the info needed to do this than I expected, largely because Cisco changed the syntax for this in ASA version 8.4, and as such many older posts are no longer valid.

The basic NAT syntax in ASA 8.4 is as follows:

nat (insideinterfacename,outsideinterfacename) static publicIPaddress service tcp_or_udp internalport externalport

The following example overloads a single public IP address as shown below:

Description Internal IP Internal Port External IP External Port
Server01 10.20.20.1 80 1.1.1.1 81
Server02 10.20.20.2 80 1.1.1.1 82

Note we’re using two different internal IPs, but only one external IP, with different external ports mapping to different internal addresses.

object network Server01
host 10.20.20.1
nat (inside,Outside) static 1.1.1.1 service tcp 80 81

object network Server02
host 10.20.20.2
nat (inside,Outside) static 1.1.1.1 service tcp 80 82

If you need to translate multiple ports for the same destination server, you will need to create an object for each port:

Description Internal IP Internal Port External IP External Port
Server01_http 10.20.20.1 80 1.1.1.1 81
Server01_https 10.20.20.1 443 1.1.1.1 8081

object network Server01_http
host 10.20.20.1
nat (inside,Outside) static 1.1.1.1 service tcp 80 81

object network Server01_https
host 10.20.20.1
nat (inside,Outside) static 1.1.1.1 service tcp 443 8081

Hope this helps!

Revisiting PXE Booting for UEFI Devices

Posted by James F. Prudente on March 9, 2015
Posted in: Deployment. Tagged: , , . 2 Comments

One of my earliest posts pertained to troubleshooting we did to fix PXE booting for some UEFI Windows tablets we were deploying. The issue there stemmed from what turned out to be an incorrect PXE configuration that wasn’t allowing the deployment server to dynamically serve the proper boot file.

Since those changes, we’ve been working with a few models of 32-bit UEFI tablets for about a year without issue and could successfully image the devices as many times as needed. So when we started bringing in some different tablets we were surprised to find we were once again having problems.

This time around, the devices (specifically the Acer SW5-012 and ASUS T100) would PXE boot once and attempt to image. Many times things would work and we’d have a properly imaged tablet. Other times imaging would fail. When that happened, or if for some reason we needed to reimage, we ran into very strange behavior. The device would attempt to PXE boot and then fail, claiming it could not find the device’s internal storage. And sure enough, while a unit fresh out of the box showed internal storage in the BIOS, once the imaging process wiped the storage, the BIOS no longer showed any internal storage. We could still install Windows directly from a UEFI bootable USB drive, but PXE would not work.

The fact that other 32-bit UEFI tablets did work really threw us off. Packet captures even showed that both the working and non-working devices reported the same “IA32 EFI” architecture during the DHCP process.

We ultimately stumbled on a piece of MS documentation (which I can’t find as I type this up) that indicated the problem came down to the OS version of the WDS deployment server in use. According to this link, support for 32-bit UEFI devices is only present in Server 2012, but the original documentation we found specified 2012 R2. What I can say with certainty is we now have 32-bit UEFI working with WDS on 2012 R2.

We really would like to understand why some 32-bit UEFI devices work while others don’t. Our best guess is it has something to do with the eMMC storage in the problem devices, but we can’t explain it. At least it’s fixed. Hopefully this solves the problem for someone else, as it was quite frustrating and took up a lot of our time.

Websense Content Gateway Crashes After a Failed Upgrade and Rollback

Posted by James F. Prudente on February 23, 2015
Posted in: Web Filtering. Tagged: . Leave a comment

I haven’t had a chance to post anything lately but wanted to get this up quickly in the hopes it saves someone else the marathon support session I recently endured.

We attempted to upgrade our Websense appliance from version 7.8.1 to 7.8.4, which should not have been a big deal. Unfortunately the upgrade failed; the system automatically rolled-back to 7.8.1, but we then found the content gateway (proxy) module would continually crash shortly after starting up.

As I said, it took quite a long time to get to the bottom of it – and there’s nothing you can do as an end-user to fix it – but the problem turned out to be that our content gateway was joined to our domain in order to make use of Integrated Windows Authentication (IWA). When the rollback occurred, something got out of sync with the domain and caused the proxy to crash. It’s a known problem but evidently a rather obscure one, and you will likely need to get someone reasonably high up in Websense’s tech support to do the following if you find yourself in this situation:

  • Reboot the appliance.
  • As soon as the content gateway module is up, SSH to it and shut down all the Websense services. This stops it from crashing.
  • Manually edit the module’s configuration to clear out the IWA settings.
  • Restart the services and/or module.
  • Using the UI, disjoin from the domain if it lets you; it may already be disjoined.
  • Using the UI, rejoin to the domain.

Going forward, it was suggested to us that if you use IWA, your upgrade process ought to be the below:

  • Perform a full appliance backup.
  • Disjoin the content gateway from the domain.
  • Perform another full backup.
  • Upgrade/Patch
  • Rejoin to the domain.

This should allow the system to survive a failed upgrade and rollback; you would then just need to rejoin the domain.

With any luck this helps someone. Hopefully I can post some other topics soon.

Revisiting Java Configuration

Posted by James F. Prudente on January 16, 2015
Posted in: Scripting. Tagged: , , . Leave a comment

I haven’t had much time to post lately but wanted to quickly update my earlier post about scripting changes to Java’s deployment.properties file.

We had to create a security exception for a particular website on a large number of PCs. These exceptions are stored in plaintext in the “exception.sites” file, which can be found at c:\Users\profilename\AppData\LocalLow\Sun\Java\Deployment\security\ – note this is one level below the “deployment.properties” file itself.

I updated my original script to modify the “exception.sites” file as well. All you need to do is add the sites you need to create exceptions for as comma separated URLs in the line shown below:

SiteExceptions = Array ("http://sitetounblock.com", "https://site2.com")

I also made a few changes to prevent infinite loops that occasionally occur as a result of what seems to be an odd set of circumstances in checking the AtEndOfStream property.

As always, use at your own risk and please test thoroughly before putting into production.

Here’s the script.