Deployment

MDT/ADK Issues – Path Not Found

Posted by James F. Prudente on October 12, 2022
Posted in: Deployment, Windows 10, Windows 11, Windows Server. Tagged: , , . Leave a comment

Brief but important warning here about Windows 11 ADK/MDT.

We recently ran into a corruption issue with our imaging server that led us to reinstall some of Microsoft’s imaging components including the Assessment and Deployment Kit and the additional WinPE extension. Naturally we went for the latest version, which as of this writing corresponds to Win 11 22H2 build 22621.

After the reinstall, we ran into an issue where we could not run scripts from within WinPE. I added the missing scripting packages to the deployment share, but every attempt to rebuild the WinPE media resulted in errors.

Deployment Image Servicing and Management tool
Version: 10.0.22621.1
Image Version: 10.0.22621.1
Processing 1 of 1 - Adding package WinPE-Scripting-Package~31bf3856ad364e35~amd64~~10.0.22621.1
[======                     11.0%                          ]
[===========                20.0%                          ]
[==============             25.0%                          ]
[=================          30.0%                          ]
[====================       35.0%                          ]
[=======================    40.0%                          ]
[========================== 45.0%                          ]
[==========================100.0%==========================]
An error occurred - WinPE-Scripting-Package Error: 0x80070003
Error: 3
The system cannot find the path specified.
The DISM log file can be found at C:\Windows\Logs\DISM\dism.log
Exit code = 3

I spent way too long trying to figure out why DISM was erroring trying to find a package file that was clearly available. Interestingly, Sysinternals Process Monitor did not show any file not found errors that seemed relevant.

Even more interesting was that the x86 WinPE image would build successfully with the scripting components. It was only x64 WinPE that failed while adding the additional packages.

I started looking at any obvious differences between the WinPE logs for x86 and x64, and one thing stood out: the version number for DISM and the WinPE image. The latest ADK shows x64 build 10.0.22621 but x86 build 10.0.22000.

Some quick research turned up this page which, in true Microsoft fashion, doesn’t quite tell the whole story. Per Microsoft, “The 32-bit versions of Windows PE are no longer included in the Windows PE add-ons starting with the ADK for Windows 11, version 22H2.” That doesn’t seem to be the case; although I didn’t install on a clean server to confirm, it looks like the 22H2 ADK still includes x86 WinPE components, just an older version.

Microsoft goes on to say that “The last supported version of 32-bit Windows PE is available in the Windows PE add-on for Windows 10, version 2004.” But that doesn’t seem to be accurate either. Win 10 version 2004 corresponds to build 19041, yet the x86 DISM included with the latest 22H2 ADK is 22000, which corresponds to Win 11 build 21H2.

At this point I had a path not found error that made no sense, and a version discrepancy that I also couldn’t explain. I wanted to try the 21H2 version of the ADK and see what happened, but Microsoft doesn’t list it for download on the page I linked above. Fortunately. I found a post by Prajwal Desai that included a link to the 21H2 ADK. Note that the download is direct from Microsoft.

I removed the 22H2 ADK and installed the 21H2 ADK. This gave me version 22000 for both x86 and x64 architectures. And perhaps not surprisingly at this point, the 22000 x64 build installs the scripting packages without issue.

So long story short, the 22H2 ADK is apparently bugged. Use the 21H2 ADK instead.

Revisiting PXE Booting for UEFI Devices

Posted by James F. Prudente on March 9, 2015
Posted in: Deployment. Tagged: , , . 2 Comments

One of my earliest posts pertained to troubleshooting we did to fix PXE booting for some UEFI Windows tablets we were deploying. The issue there stemmed from what turned out to be an incorrect PXE configuration that wasn’t allowing the deployment server to dynamically serve the proper boot file.

Since those changes, we’ve been working with a few models of 32-bit UEFI tablets for about a year without issue and could successfully image the devices as many times as needed. So when we started bringing in some different tablets we were surprised to find we were once again having problems.

This time around, the devices (specifically the Acer SW5-012 and ASUS T100) would PXE boot once and attempt to image. Many times things would work and we’d have a properly imaged tablet. Other times imaging would fail. When that happened, or if for some reason we needed to reimage, we ran into very strange behavior. The device would attempt to PXE boot and then fail, claiming it could not find the device’s internal storage. And sure enough, while a unit fresh out of the box showed internal storage in the BIOS, once the imaging process wiped the storage, the BIOS no longer showed any internal storage. We could still install Windows directly from a UEFI bootable USB drive, but PXE would not work.

The fact that other 32-bit UEFI tablets did work really threw us off. Packet captures even showed that both the working and non-working devices reported the same “IA32 EFI” architecture during the DHCP process.

We ultimately stumbled on a piece of MS documentation (which I can’t find as I type this up) that indicated the problem came down to the OS version of the WDS deployment server in use. According to this link, support for 32-bit UEFI devices is only present in Server 2012, but the original documentation we found specified 2012 R2. What I can say with certainty is we now have 32-bit UEFI working with WDS on 2012 R2.

We really would like to understand why some 32-bit UEFI devices work while others don’t. Our best guess is it has something to do with the eMMC storage in the problem devices, but we can’t explain it. At least it’s fixed. Hopefully this solves the problem for someone else, as it was quite frustrating and took up a lot of our time.

Adding Roles and Features During Deployment

Posted by James F. Prudente on February 27, 2014
Posted in: Deployment. Tagged: , , , , . 5 Comments

More on our continuing saga to deploy Windows 8.1 via MDT…

I thought we were done, with a fully-functional MDT deployment that installed Windows 8.1 exactly the way we wanted. And in a sense, we were, until we started pushing a few applications. One of these applications failed during a silent install with a generic error, and when I manually attempted the install it quickly became obvious what the problem was…the app required the .NET 3.5 Framework.

Now, the .NET Framework is typically delivered as a redistributable runtime along with apps that require it, however on Windows 8 / 8.1 it has been moved to a “feature” that must be installed through “Programs and Features” in the control panel. If you interactively try to install an app that requires the runtime, Windows is smart enough to catch that and prompt you to add the feature. That may or may not work without additional intervention, depending on if your install has access to a full Windows source or can find what it needs from Windows Update or WSUS, but it certainly does not work during a silent install.

There’s a few ways of dealing with this, but I really wanted to add the feature as part of our deployment process. That way, we know it’s installed everywhere. I was pleased to find out MDT 2013 (and I believe 2012 as well) offers “Install Roles and Features” as a pre-canned task step. I added it within the “State Restore” section, just before Windows Update runs.

When you add this step, you can then choose the target OS and select the roles and features you want added.

Easy, right? It never is though, is it?

Once I added this step and re-tested the deployment, we started getting prompted to select roles and features during what should have been a silent deployment. This happened very early in the deployment process, not even where one would have expected based on its location in the task.

The fix for this was pretty straightforward. In our customsettings.ini file, I added one line to the [Default] section:

    SkipRoles=YES

Note that YES must be capitalized. Once this was done the prompt went away and the deployment was back to being silent.

Of course, that still didn’t fix things. Before I go any further I should point out we use SpecOps Deploy which acts as a wrapper around MDT, and it’s possible the error we ran into is a result of something the SpecOps people are doing. I will touch base with them and if they have any feedback I’ll update this post accordingly. And just to be clear, I’m a big fan of SpecOps Deploy; you can see a case-study they did on us here.

Anyway, I started getting a “Path Not Found” error from ZTIOSRoles.wsf, which is the script that gets called when you add the “Install Roles and Features” task step. Based on the BDD.LOG file that MDT generates it was clear ZTIOSRoles.wsf was generating this error within its GetSource function.

Function GetSource
   ' By default, assume any needed files (e.g. NetFx3) can be loaded from WU or WSUS.
   GetSource = ""
   ' If the user explicitly set WindowsSource, copy the files locally,
   ' pass the value along via the source switch, and limit access to
   ' the internet.
   If oEnvironment.Item("WindowsSource")  "" then
	oUtility.ValidateConnection oEnvironment.Item("WindowsSource")
	If not oFSO.FolderExists(oUtility.LocalRootPath & "\sources\" & oEnvironment.Item("Architecture")) then
	   oLogging.CreateEntry "Copying source files locally from " & oEnvironment.Item("WindowsSource"), LogTypeInfo
	   oUtility.VerifyPathExists oUtility.LocalRootPath & "\sources\" & oEnvironment.Item("Architecture")
	   oFSO.CopyFolder oEnvironment.Item("WindowsSource"), oUtility.LocalRootPath & "\sources\" & oEnvironment.Item("Architecture"), true
	End if
	GetSource = oUtility.LocalRootPath & "\sources\" & oEnvironment.Item("Architecture")
	' If the SourcePath value was set (typically in LTI via ZTIUtility.vbs),
	' copy the files locally, pass that path along via the source switch,
	' and limit access to the internet.
   ElseIf oEnvironment.Item("SourcePath")  "" then
      oUtility.ValidateConnection oEnvironment.Item("SourcePath")
      If not oFSO.FolderExists(oUtility.LocalRootPath & "\sources\" & oEnvironment.Item("Architecture")) then
	 If oFSO.FolderExists(oEnvironment.Item("SourcePath") & "\sources\sxs") then
	    oLogging.CreateEntry "Copying source files locally from " & oEnvironment.Item("SourcePath") & "\sources\sxs", LogTypeInfo
	    oUtility.VerifyPathExists oUtility.LocalRootPath & "\sources\" & oEnvironment.Item("Architecture")
	    oFSO.CopyFolder oEnvironment.Item("SourcePath") & "\sources\sxs", oUtility.LocalRootPath & "\sources\" & oEnvironment.Item("Architecture"), true				GetSource = oUtility.LocalRootPath & "\sources\" & oEnvironment.Item("Architecture")
	 Else
            oLogging.CreateEntry "SourcePath was set, but " & oEnvironment.Item("SourcePath") & "\sources\sxs does not exist, not using local source.", LogTypeInfo
	 End if
      End if
   End if
End Function

Sorry the code block looks horrible…I’m still learning how to handle code with WordPress.

After adding some debug logging of my own (not shown here) I established the error was being generated by the oFSO.CopyFolder line bolded above. That line attempts to copy the “sxs” folder that contains the .NET binaries from the deployment share to the local PC. Frustratingly, after more debugging it seemed both the source and destination path did exist, so I was at a loss as to why the error was being generated. I’ve run into similar problems before with this function in vbScript, and there doesn’t seem to be a good way of troubleshooting what’s actually happening within CopyFolder.

I ended up with a bit of a hack as a workaround. I replaced the entire ElseIf block with the below:

ElseIf oEnvironment.Item("SourcePath") = "" then
   oUtility.ValidateConnection oEnvironment.Item("SourcePath")
   GetSource = oEnvironment.Item("SourcePath") & "\sources\sxs"
End if

Basically, instead of trying to copy the binaries locally, we’re just telling the script to use the networked copy. I have no idea why this isn’t done by default, unless there’s some side effect that I’m not thinking of. For now at least, this is working fine.

Hope this helps someone!

Connected Standby and OS Deployment

Posted by James F. Prudente on February 20, 2014
Posted in: Deployment. Tagged: , , , . 6 Comments

I make my living primarily by supporting Microsoft products, and I genuinely think their technology is quite good. But every now and then they make decisions so monumentally stupid that they need to be called out for them; “Connected Standby” is one of those instances.

Connected Standby is a new power mode in Windows 8 or 8.1 designed for tablets which essentially allows them to enter sleep mode but keep just enough functionality active so they can receive e-mails, keep their DHCP leases, etc. In other words, you’re in standby but still “connected.” This is of course to help battery life and create a more smartphone-like experience. Fine, I’ve got no arguments there.

The problem is what causes the system to go into this mode, or more accurately what prevents it from doing so. Apparently there’s an entire sub-system that manages this, the Desktop Activity Moderator. Whatever is happening under the hood though, the only thing that seems to keep the system from sleeping is user activity: mouse, touch, or keyboard input. In theory that’s fine, but try doing an OS deployment to a system that goes to sleep after ten minutes and see how well that works. Just to be clear, a “connected standby”-capable tablet will go to sleep during the OS deployment process while using Microsoft Deployment Toolkit (MDT) 2013.

Compounding this problem there’s no obvious way to disable connected standby. It’s supposed to be transparent to applications and thus there shouldn’t be a need to turn it off. Yeah, anyway. Fortunately, after a lot of research and a lot of trial and error, I was able to disable this “feature” while imaging a system.

A “connected standby”-capable device should by default have only a “balanced” power scheme available. We need to make changes to this scheme during the imaging process, taking note of the fact that the setting which controls connected standby is actually the display timeout, not the sleep setting. We can do this by calling a batch file during the deployment task sequence, which itself will use powercfg.exe to change the power settings.

First, save the below to the Scripts folder of your Deployment Share(s) as CHANGEPOWERSETTINGS.BAT.

REM Set the display timeout

powercfg /setacvalueindex 381b4222-f694-41f0-9685-ff5bb260df2e 7516b95f-f776-4464-8c53-06167f40cc99 3c0bc021-c8a8-4e07-a973-6b14cbcb2b7e 0

REM Set the sleep timeout

powercfg /setacvalueindex 381b4222-f694-41f0-9685-ff5bb260df2e 238c9fa8-0aad-41ed-83f4-97be242c8f20 29f6c1db-86da-48c5-9fdb-f2b67b1f44da 0

REM Activate this power scheme

powercfg /s 381b4222-f694-41f0-9685-ff5bb260df2e

By using /setacvalueindex we are changing the timeouts when the device is on AC (line) power for the specific settings referenced by the GUIDs. Setting the timeout to “0” disables that particular power setting. Note that despite the “balanced” scheme already being active (by virtue of it being the only scheme), we still need to “activate” it for the new settings to take effect. Also, since we’re making changes to the AC values only, this should have no effect on a device running on battery power.

Next we need to add a task to our deployment task sequence in Deployment Workbench. Here, I’ve added “Set Power Options,” which is a “Run Command Line” task that simply calls the batch file we created above. I chose to put this between “Tattoo” and “Windows Update (Pre-Application Install)” because it was during the initial Windows Update task that the systems were going to sleep. On the options tab, you can choose if the task should continue on error; at this point I am not doing so since the task itself will fail when the system goes to sleep anyway. Once we start testing 8.1 on the desktop or devices that do not use connected standby, I may need to revisit this option.

Depending on your preferences once the deployment is complete, you may want to create another task near the end of the sequence to re-enable connected standby on AC. We use 3rd party power management software however, so I’m not too concerned about how this is set once the system is up and running.
Hopefully this helps some of you out. And Microsoft, please, when you add new features like this – give us administrators an easier way to decide how we want the device to work. Stopping a system from going to sleep should not require digging around for obscure and illogical settings.

Troubleshooting PXE Booting

Posted by James F. Prudente on February 18, 2014
Posted in: Deployment. Tagged: , , . 12 Comments

PXE is a great example of a topic that turns up a ton of search results but very little helpful content. Search for “PXE configuration” or “PXE troubleshooting” and you’ll find the majority of posts focus on the same thing, specifically a few DHCP options that “must” be set in order for PXE to work. Admittedly that’s how we had PXE setup until recently, but an upgrade to our imaging software forced us to revisit this configuration, make changes, and learn quite a bit along the way.

We use Specops Deploy to image PCs and recently upgraded to their latest version to support the Windows 8.1 tablets we’re beginning to test. Deploy builds on standard Microsoft deployment tools, so pretty much everything in this post should apply regardless of the imaging solution you’re using. Besides, PXE is primarily BIOS-dependent as we’ll see later.

First a little background…

If enabled in the BIOS, PXE is part of the PC’s startup sequence. It uses the DHCP process by which devices are dynamically assigned IP addresses to obtain information about booting from a network location.

The basic DHCP process takes four steps:

  1. The client sends out a DHCP Discover broadcast.
  2. The DHCP server (or servers) respond with a DHCP Offer, which includes the IP address the DHCP server will provide the client.
  3. The client sends a DHCP Request broadcast, indicating the IP address it has selected.
  4. The DHCP server selected sends a DHCP ACK to the client, acknowledging the client has accepted the IP address.

Important Note: As mentioned, DHCP relies on broadcasts, which by definition do not traverse VLANs or subnets. If your DHCP server(s) is on a different VLAN from your clients, your router will need to be configured as a DHCP relay; on Cisco equipment this is done through the ip helper-address command. This will come into play for PXE as well.

Anyway, that’s the basic DHCP traffic flow, but it’s important to recognize that the initial DHCP Discover includes requests for quite a few parameters, as shown below via a Wireshark capture. Note options 60, 66, and 67, Vendor Class Identifier, TFTP server name and Bootfile Name.

The typical “recommended” configuration for PXE requires explicitly setting these three options (again: 60, 66, and 67) so that your clients receive this information directly from your DHCP server. And sure enough, if they are set a certain way, PXE will work in some (perhaps most) cases. But this is not the proper way to setup PXE, and doing so is not supported by Microsoft as per KB #259670. There are a few reasons this is not the right way to do things; first of all, by explicitly setting the PXE server in this fashion, you eliminate the option of having redundant PXE servers. Additionally, in some cases if your PXE server is down, your client PCs may hang up – either briefly or indefinitely – while looking for it. The biggest problem though – and the one that for us started this whole process – is that by specifying a boot file name in option 67, you eliminate your PXE server’s ability to dynamically determine which boot file it serves to a client.

The first indication we had that anything was wrong with our PXE setup was that the new tablets we were testing (Dell Venue 11 5130s, which are Atom-based devices) would not PXE boot. Turns out they need a UEFI boot file, which is different than everything else on our network. Explicitly setting option 67 caused the tablets to look for the standard boot file we’d been using, and since that was not a valid boot file for the tablets, they could not find a boot device through PXE. My first attempted fix was just to remove option 67, but evidently 60, 66, and 67 all need to be set if you’re going to use the explicit method.

The good news is that fixing all this is simple – sorta. You effectively just need to do two things:

  • Remove options 60, 66, and 67 from your DHCP server(s). You may find reference elsewhere to removing option 43 (Vendor-Specific Information) as well but we have this option set for other purposes and this has not caused any issues.
  • If your PXE server is on a different subnet/VLAN from your clients, configure your router to forward broadcasts to it, exactly as you do for your DHCP servers.

Two things will then happen when a client sends out a DHCP Discover broadcast:

  • Your DHCP server(s) will respond with IP address(es) and related info.
  • Your PXE server will respond with option 60, identifying itself as a boot server.

Note that second bullet-point…the PXE server should be replying with option 60, but the DHCP servers should not.

Once we realized we shouldn’t be setting these options explicitly and configured the ip helper-addresses, the Venue tablets sure enough would successfully PXE boot, getting the proper UEFI boot file in the process. But it’s never that simple, right?

Making these changes broke PXE for all (or as it turns out, almost all) of our desktops and laptops, which had been working fine for years. We have five different models of Dell OptiPlex desktops, plus a few models of Latitude laptops….a pretty good mix of their corporate product line for the last six years.

Most of the desktops would error out during the PXE part of the boot process with a “PXE-E55 ProxyDHCP service did not respond on port 4011″ error. Some of the laptops were worse, completely hanging during PXE without getting an IP address. Just to be clear, these PCs were not actually trying to boot from the network; the errors were happening during the part of the system’s start-up where it first tries to connect to the boot server and determine if it should be booting from the network.

Research into the PXE-E55 error always came back to the same supposed cause: having option 60 set on your DHCP server (without options 66 and 67) when your PXE server was on a separate server. Essentially in this configuration, your client would see the DHCP server respond with option 60 and because of that try to connect to it on port 4011 to network boot; since your DHCP server was not configured for PXE, port 4011 would obviously not respond.

I double and triple checked our DHCP servers and confirmed they should not be sending out option 60, so I decided to do some packet captures during the boot process to make sure they weren’t. The results were interesting; on a machine that generated the PXE-E55 error, I could see the DHCP Discover request go out and likewise see the DHCP servers and PXE server all send their DHCP Offer in response. The client would display an IP address (and respond to pings on this IP) but no further DHCP packets were sent out. The client never sent a DHCP Request packet and the PXE-E55 error occurred immediately after the client acquired an IP.

My instinct at this point was that something was wrong on the client side; after all the next packet that should have been sent was from the client. But with PXE having worked for years and all of our recent changes having been made on the server and network side, logic seemingly dictated the client couldn’t be the problem.

The breakthrough was one of my techs identifying a machine that was properly getting through the PXE process without an error. I packet captured that system and saw the four-step DHCP exchange as expected.

So now I had two test PCs on the same VLAN, connected to the same switch, and connecting to the same set of servers; one worked, one didn’t.

While they were different model PCs, they both used the same Intel NIC. However the PXE firmware on the NIC was slightly newer on the working system. I downloaded the latest Dell BIOS for the non-working machine, installed it, and low and behold the system started PXE booting as expected.

There’s clearly a bug in the earlier revisions of the Intel PXE firmware. Keep in mind that no traffic at all was being sent from the NIC of the non-working system after the DHCP Offers were received. It seems the NIC saw option 60 set without 66 and 67, and immediately generated the PXE-E55 error without even trying to connect to a PXE server. Interestingly I looked through all the BIOS revision notes I could find for a Dell OptiPlex GX745 – which was one of our affected systems – and cannot find any mention of PXE improvements or fixes. But the version number of the PXE firmware did increment and the behavior changed, so it would seem the release notes are incomplete.

Thankfully Dell provides a way to silently push a BIOS install, so we’re now updating the BIOS on all of our systems, and PXE is back to working as expected.

Your comments or questions are appreciated. Thanks for reading!