Tech Blog

Had this very annoying issue on the WSUS console where updates would just stick.

The fix was really very simple although I tried a varierty of things that didnt work first.

The fix:

Stop the following services (I actually had to disable and reboot as they wouldnt stop):

WSUS Service - WsusService

Windows Update - wuauserv

Background Intelligent Transfer Service - BIT

Then delete this folder:

C:\Windows\SoftwareDistribution

Restart the services and bobs your uncle.

What didnt work but might for you (or me in the future):

Find the failed update:

From the WSUS management console, you can go to Updates -> All Updates. After searching out the approved updates, and then adding the column for File Status, you can see the updates in downloading.

For the updates downloading incompletely, you can decline these updates.

or

Look for these events and decline the update:

Or

Cancel all updates from PowerShell

(Get-WsusServer).CancelAllDownloads()

Then let them download and keep and eye on which is struggling and decline it.

Or

Reset WSUS content:

1) Correct any settings above or disapprove any unneeded updates.

2) Close any open WSUS consoles.

3) Go to Administrative Tools – Services and STOP the Update Services service.

4) In Windows Explorer browse to the WSUSContent folder (typically D:\WSUS\WSUSContent or C:\WSUS\WSUSContent)

5) Delete ALL the files and folders in the WSUSContent folder.

6) Go to Administrative Tools – Services and START the Update Services service.

7) Open a command prompt and navigate to the folder: C:\Program Files\Update Services\Tools.

8) Run the command WSUSUtil.exe RESET

Or

Check the permission of the WSUS Content folder, check if NETWORK SERVICE and WSUS administrators have full permission

Or

Reindex the SQL DB:

https://gallery.technet.microsoft.com/scriptcenter/6f8cde49-5c52-4abd-9820-f1d270ddea61

Below is a SQL script that will show how many updates are missing that are classified as critical or security for servers on WSUS:

SET NOCOUNT ON DECLARE @TargetGroup nvarchar(30) 
DECLARE @Days int SELECT @TargetGroup = 'All Computers' 
SELECT @Days = 7 DECLARE @groups AS TABLE (Id uniqueidentifier NOT NULL) 
DECLARE @groupId uniqueidentifier SET @groupId = (
     SELECT ComputerTargetGroupId
     FROM PUBLIC_VIEWS.vComputerTargetGroup
     WHERE vComputerTargetGroup.Name = @TargetGroup )
WHILE @groupId IS NOT NULL BEGIN
     INSERT INTO @groups SELECT @groupId
     SET @groupId = (
         SELECT ParentTargetGroupId
        FROM PUBLIC_VIEWS.vComputerTargetGroup
         WHERE vComputerTargetGroup.ComputerTargetGroupId = @groupId     )
END 
DECLARE @updates AS TABLE (Id uniqueidentifier NOT NULL PRIMARY KEY) 
INSERT INTO @updates SELECT distinct vUpdate.UpdateId 
FROM     PUBLIC_VIEWS.vUpdate WHERE    vUpdate.MsrcSeverity is NOT NULL
        AND vUpdate.defaultTitle like '%Security%'
		OR vUpdate.defaultTitle like '%critical%'
SELECT vComputerTarget.Name as 'Computer Name', COUNT(*) AS 'Missing Updates' 
FROM PUBLIC_VIEWS.vComputerGroupMembership
     INNER JOIN PUBLIC_VIEWS.vComputerTarget 
on vComputerGroupMembership.ComputerTargetId = vComputerTarget.ComputerTargetId
     INNER JOIN PUBLIC_VIEWS.vComputerTargetGroup 
on vComputerGroupMembership.ComputerTargetGroupId = vComputerTargetGroup.ComputerTargetGroupId
     INNER JOIN PUBLIC_VIEWS.vUpdateInstallationInfoBasic 
on vUpdateInstallationInfoBasic.ComputerTargetId = vComputerTarget.ComputerTargetId
     INNER JOIN @updates GROUPS
on vUpdateInstallationInfoBasic.UpdateId = GROUPS.Id 
WHERE vComputerTarget.ComputerTargetId = vUpdateInstallationInfoBasic.ComputerTargetId
     AND vUpdateInstallationInfoBasic.State in (2, 3, 5, 6)
       AND vComputerTargetGroup.Name = @TargetGroup 
GROUP BY vComputerTarget.Name 
ORDER BY 'Missing Updates' DESC

WSUS console kept crashing with the reset console error:

The WSUS administration console was unable to connect to the WSUS Server via the remote API. 

Verify that the Update Services service, IIS and SQL are running on the server. If the problem persists, try restarting IIS, SQL, and the Update Services Service.

System.Net.WebException -- The operation has timed out

Source
System.Web.Services

Stack Trace:
   at System.Web.Services.Protocols.WebClientProtocol.GetWebResponse(WebRequest request)
   at Microsoft.UpdateServices.Internal.DatabaseAccess.ApiRemotingCompressionProxy.GetWebResponse(WebRequest webRequest)
   at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)
   at Microsoft.UpdateServices.Internal.ApiRemoting.ExecuteSPGetUpdateServerStatus(Int32 updateSources, Boolean includeDownstreamComputers, String updateScopeXml, String computerTargetScopeXml, String preferredCulture, Int32 publicationState, Int32 propertiesToGet)
   at Microsoft.UpdateServices.Internal.DatabaseAccess.AdminDataAccessProxy.ExecuteSPGetUpdateServerStatus(UpdateSources updateSources, Boolean includeDownstreamComputers, String updateScopeXml, String computerTargetScopeXml, String preferredCulture, ExtendedPublicationState publicationState, UpdateServerStatusPropertiesToGet propertiesToGet)
   at Microsoft.UpdateServices.Internal.BaseApi.UpdateServer.GetStatus(UpdateSources updateSources, Boolean includeDownstreamComputers, UpdateScope updatesToInclude, ComputerTargetScope computersToInclude, UpdateServerStatusPropertiesToGet propertiesToGet)
   at Microsoft.UpdateServices.Internal.BaseApi.UpdateServer.GetReplicaStatus(UpdateSources updateSources)
   at Microsoft.UpdateServices.UI.AdminApiAccess.CachedObject.RefreshCache()
   at Microsoft.UpdateServices.UI.AdminApiAccess.CachedObject.GetFromCache()
   at Microsoft.UpdateServices.UI.SnapIn.Pages.ServerSummaryPage.backgroundWorker_DoWork(Object sender, DoWorkEventArgs e)

So after doing all the normal IIS stuff, got Microsoft involved and here’s what they did to fix it:

• We checked and ensured that none of the services crashed or stopped after console crashes,
• We checked and ensured that WSUSPool was also running,
• We checked and ensured that SQL service was also running on the remote server,
• We increased the WSUSPool PrivateMemoryLimit to unlimited by setting it to ‘0’ on the IIS console
• We also increased the Maximum queue length from 1000 to 3000,
• We stopped WSUSPool and Restarted the IIS service,

We ran below mentioned on SQL to check and find if there are Obsoletes update to clean up but found none

exec spGetObsoleteUpdatesToCleanup

• We ran SQL query to Re-index SUSDB and tried to connect to the console https://gallery.technet.microsoft.com/scriptcenter/6f8cde49-5c52-4abd-9820-f1d270ddea61#content
• We changed WSUS to use port 80 using command below

• WSUSUtil.exe usecustomewebsite false

• WSUS was successfully changed to port 80
• We tried to connect to the console but t still fails the same way

• We found that console was failing to connect with below error as well
• We removed the WSUS mmc from folder %appdata%\Microsoft\MMC\
• We tried to connect again to WSUS console but with no avail

• We further investigated and ensured that WSUS s failing to connect with SUSDB as it is timing out
• We checked and found that SUSDB size was more than 6.5 GB
• We restarted the SQL Service for instance hosting SUSDB
• We tried to connect to WSUS console after restarting the service and it connected successfully
• We see that there were more than 13,000 approved updates on WSUS console
• This could cause an issue for WSUS to connect as it would take a long time to connect with SUSDB
• We decided to run PowerShell script to decline superseded updates from WSUS to enhance the performance of WSUS while connecting to SUSDB
• We ran PowerShell script from below mentioned link to decline superseded updates
  https://blogs.technet.microsoft.com/configurationmgr/2015/04/15/support-tip-configmgr-2012-update-scan-fails-and-causes-incorrect-compliance-status/
• Superseded updates were declined successfully

PS C:\Scripts> cd e:
PS E:\> cd .\ms\
PS E:\ms> .\sus.ps1 -updateserver WSUS_SERVER -port 80
Connecting to WSUS server WSUS_Server on Port 80... Connected.
Getting a list of all updates... Done
Parsing the list of updates... Done.
List of superseded updates: E:\SupersededUpdates.csv
Summary:
========
All Updates = 17101
Any except Declined = 15718
All Superseded Updates = 9621
    Superseded Updates (Intermediate) = 6801
    Superseded Updates (Last Level) = 2820

SkipDecline flag is set to False. Continuing with declining updates
 DeclineLastLevel is set to False. Declining all superseded updates.
 Declined 9621 updates.
 Backed up list of superseded updates to E:\SupersededUpdatesBackup.csv

• We changed the WSUS to use customwebsite port 8530 again

• WSUSUtil.exe usecustomewebsite true

   

• WSUS was successfully changed to port 8530
• We tried to open WSUS console and it connected successfully
• We ran synchronization on WSUS server and it completed successfully
• BOOOM WOOP

Recently our WSUS console kept crashing either when opening certain computer folders or All Computers.

Error: Unexpected error

So after lots of frustration I rebuild WSUS on Server 2016 and using a proper SQL instance:

Uninstall WSUS PowerShell:

Uninstall-WindowsFeature -Name UpdateServices,Windows-Internal-Database -Restart

Then re-install with powershell:

Install-WindowsFeature -Name UpdateServices-Services,UpdateServices-DB –IncludeManagementTools

Then set the SQL instance using a elevated CMD:

WsusUtil.exe PostInstall SQL_INSTANCE_NAME="SQLSERVER\INSTANCE" CONTENT_DIR=E:\WSUS

The servers all started populating, great.

Then FAIL again.

The WSUS administration console has encountered an unexpected error. This may be a transient error; try restarting the administration console. If this error persists, 

Try removing the persisted preferences for the console by deleting the wsus file under %appdata%\Microsoft\MMC\.


The WSUS administration console has encountered an unexpected error. This may be a transient error; try restarting the administration console. If this error persists, 

Try removing the persisted preferences for the console by deleting the wsus file under %appdata%\Microsoft\MMC\.


System.Xml.XmlException -- '', hexadecimal value 0x16, is an invalid character. Line 1, position 779977.

Source
System.Xml

Stack Trace:
   at System.Xml.XmlTextReaderImpl.Throw(Exception e)
   at System.Xml.XmlTextReaderImpl.ParseNumericCharRefInline(Int32 startPos, Boolean expand, StringBuilder internalSubsetBuilder, Int32& charCount, EntityType& entityType)
   at System.Xml.XmlTextReaderImpl.ParseCharRefInline(Int32 startPos, Int32& charCount, EntityType& entityType)
   at System.Xml.XmlTextReaderImpl.ParseText(Int32& startPos, Int32& endPos, Int32& outOrChars)
   at System.Xml.XmlTextReaderImpl.ParseText()
   at System.Xml.XmlTextReaderImpl.ParseElementContent()
   at System.Xml.XmlReader.ReadStartElement()
   at System.Xml.Serialization.XmlSerializationReader.ReadStringValue()
   at System.Xml.Serialization.XmlSerializationReader.ReadTypedPrimitive(XmlQualifiedName type, Boolean elementCanBeType)
   at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderApiRemotingCompressionProxy.Read1_Object(Boolean isNullable, Boolean checkType)
   at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderApiRemotingCompressionProxy.Read2_GenericReadableRow(Boolean isNullable, Boolean checkType)
   at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderApiRemotingCompressionProxy.Read249_Item()
   at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle, XmlDeserializationEvents events)
** this exception was nested inside of the following exception **


System.InvalidOperationException -- There is an error in XML document (1, 779977).

Source
System.Xml

Stack Trace:
   at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle, XmlDeserializationEvents events)
   at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)
   at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)
   at Microsoft.UpdateServices.Internal.ApiRemoting.ExecuteSPSearchComputers(String computerTargetScopeXml)
   at Microsoft.UpdateServices.Internal.DatabaseAccess.AdminDataAccessProxy.ExecuteSPSearchComputers(String computerTargetScopeXml)
   at Microsoft.UpdateServices.Internal.BaseApi.ComputerTarget.SearchComputerTargets(ComputerTargetScope searchScope, UpdateServer updateServer)
   at Microsoft.UpdateServices.UI.AdminApiAccess.ComputerTargetManager.GetComputerTargets(ComputerTargetScope searchScope)
   at Microsoft.UpdateServices.UI.AdminApiAccess.BulkComputerPropertiesCache.GetAndCacheComputers(ExtendedUpdateScope updateScope, ComputerTargetScope computerTargetScope)
   at Microsoft.UpdateServices.UI.SnapIn.Pages.ComputersListPage.GetListRows()

After some Googling I found the problem seems to be with a corrupt PC checking in, i.e. the PC checks in, then via WMI all the info is gathered on the PC and if that info is corrupt then WSUS crashes!!

To fix,

Copy the WSUS console error into Notepad++ (don’t use anything else) and look for:

System.Xml.XmlException -- '', hexadecimal value 0x16

Basically this means that the weird square symbol or SYM symbol is present in the WSUS database and causing WSUS to shit itself:

So we need to find it, open SQL Management Studio and open the SUSDB

Note: if you’re not using proper SQL then open the DB via:

\\.\pipe\MICROSOFT##WID\tsql\query

Changed the default view setting by going to the Tools > Options > SQL Server Object Explorer settings, then adjusted the ‘Value for Edit Top <n> Rows command’ from 200 to 0, making the option display all rows.

Navigate down within the SUSDB database, to the dbo.tbComputerTargetDetail table, right-clicked it, and selected ‘Edit All Rows’.

Click the top left corner of the column/rows to select everything, then copy-pasted it into NotePad++

Then copy the SYM from the error and search for it in the info you’ve just copied:

As you can see the ID on the left is 525:

Lets now cross reference this against another table in the SUSDB database via SQL script:

Select * from dbo.tbComputerTarget

Look for the ID:

Thats the PC!!!

Find it and update the BIOS and anything else you can, then get it to report in again to WSUS:

wuauclt /scannow
wuauclt /detectnow
wuauclt /r /ReportNow

Credit to SterlingT, he’s a wonderful human being:

https://social.technet.microsoft.com/Forums/windowsserver/en-US/90dc15d3-c498-42b8-b36a-bd29be35cf99/wsus-console-unexpected-error-when-choosing-all-computers-folder?forum=winserverwsus

So recently we took the plunge to auto patch and reboot all our servers based on the following schedules:

Schedule_1 WSUS Auto Approve – 7 days Deadline – When new updates are downloaded by WSUS they are held for 7 days then rolled out to the servers.  The server will then auto install and reboot (if necessary) the Monday after the 7 day deadline expires at 03:00.  Please note all previous updates that are needed will be installed installed on any Monday at 03:00.

Schedule_2 WSUS Auto Approve – 12 days Deadline – When new updates are downloaded by WSUS they are held for 21 days then rolled out to the servers.  The server will then auto install and reboot (if necessary) the Tuesday after the 21 day deadline expires at 02:00.  Please note all previous updates that are needed will be installed installed on any Tuesday at 02:00.

Schedule_3 WSUS Auto Approve – 31 days Deadline – When new updates are downloaded by WSUS they are held for 31 days then rolled out to the servers.  The server will then auto install and reboot (if necessary) the Wednesday after the 31 day deadline expires at 03:00.  Please note all previous updates that are needed will be installed installed on any Wednesday at 03:00.

So in other words patch schedule 1 first and see if the servers break then 5 days later do install the patches on the servers in schedule 2 and then 24 days later do the rest.  I.e. Test, test and finish.

So here’s how we did it:

• On WSUS setup some Computer Groups i.e. Schedule 1, Schedule 2 and Schedule 3:

• Put your servers in these groups (or if you auto place servers in groups via GPO then I cover that later).
• Create some Auto Approval Rules under Options in WSUS manager:
  • The following example only applies Critical and Security updates to Computer Folder Schedule 1 and the deadline to install (i.e. install after) is 7 days at 03:00
  • 
  • 
  • IMPORTANT – Once you’ve created the rules click RUN RULE or the rule wont run against the existing updates.
  • 
• Finally setup the Group Policies for the Servers (example for schedule 1), I applied a security group to the GPO so only the servers in schedule 1 received these updates:
  • Administrative TemplatesWindows Components/Windows Update
    Allow Automatic Updates immediate installation Enabled
    Always automatically restart at the scheduled time EnabledThe restart timer will give users this much time to save their work (minutes): 15

    Configure Automatic Updates Enabled

    Configure automatic updating: 4 – Auto download and schedule the install
    The following settings are only required and applicable if 4 is selected.
    Install during automatic maintenance Disabled
    Scheduled install day: 3 – Every Monday
    Scheduled install time: 03:00
    Enable client-side targeting Enabled

    Target group name for this computer Schedule_1  (Note – if you use GPOs to place servers in computer groups in WSUS then is the setting)
    

    No auto-restart with logged on users for scheduled automatic updates installations Disabled
    Specify intranet Microsoft update service location Enabled

    Set the intranet update service for detecting updates: http://WSUS01:8530
    Set the intranet statistics server: http://WSUS01:8530

That should be it!!!!

So we auto patch servers using WSUS (Version: 6.2 on Server 2012) which is great as all servers (and we have loads) get patched and we don’t have to do anything (except fix shitty MS updates).

Last week WSUS patched itself and all of a sudden the WSUS Admin console was inaccessible:

Clicking the copy error to clipboard gave this:

The WSUS administration console was unable to connect to the WSUS Server via the remote API. 

Verify that the Update Services service, IIS and SQL are running on the server. If the problem persists, try restarting IIS, SQL, and the Update Services Service.

The WSUS administration console has encountered an unexpected error. This may be a transient error; try restarting the administration console. If this error persists, 

Try removing the persisted preferences for the console by deleting the wsus file under %appdata%\Microsoft\MMC\.


System.IO.IOException -- The handshake failed due to an unexpected packet format.

Source
System

Stack Trace:
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result)
   at System.Net.TlsStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.ConnectStream.WriteHeaders(Boolean async)
** this exception was nested inside of the following exception **


System.Net.WebException -- The underlying connection was closed: An unexpected error occurred on a send.

Source
Microsoft.UpdateServices.Administration

Stack Trace:
   at Microsoft.UpdateServices.Administration.AdminProxy.CreateUpdateServer(Object[] args)
   at Microsoft.UpdateServices.UI.AdminApiAccess.AdminApiTools.GetUpdateServer(String serverName, Boolean useSecureConnection, Int32 portNumber)
   at Microsoft.UpdateServices.UI.SnapIn.Scope.ServerSummaryScopeNode.ConnectToServer()
   at Microsoft.UpdateServices.UI.SnapIn.Scope.ServerSummaryScopeNode.get_ServerTools()

So after much Googling I found KB3148812 is the update that broke it.

To recover your console, run the following in an elevated command prompt (assuming Windows is installed on drive C):

cd C:\Program Files\Update Services\Tools

Wsusutil.exe postinstall /servicing

Then reset the server node or reboot WSUS, and you’re back in!

Once your back you may find that client scans against WSUS no longer succeed.

To restore client-server communication, enable HTTP Activation on your WSUS server via the Add Features and Roles Wizard in your Server Manager:

Job Done.