ConfigMgr Current Branch Topic ID 611 swd State Messages flood
Issue: Clients are flooding the inboxes\auth\statesys.box\incoming with state messages similar to the below. There isn't much there to go on. The only promising thing to look for was that it's "Topic ID="611"" and Type="611". Researching those topicid of 611 and we found nothing public, and opening a case with Microsoft provided some clues, but this was really a new issue, at least at this scale.
<?xml version="1.0" encoding="UTF-16"?>
<Report>
<ReportHeader>
<Identification>
<Machine>
<ClientInstalled>1</ClientInstalled>
<ClientType>1</ClientType>
<ClientID>GUID:0EE65490-9075-41B1-B64D-AAAAAAAAAAAA</ClientID>
<ClientVersion>5.00.8412.1006</ClientVersion>
<NetBIOSName>CLIENTNAMEHERE</NetBIOSName>
<CodePage>437</CodePage>
<SystemDefaultLCID>1033</SystemDefaultLCID>
<Priority>5</Priority>
</Machine>
</Identification>
<ReportDetails>
<ReportContent>State Message Data</ReportContent>
<ReportType>Full</ReportType>
<Date>20170314180133.967000+000</Date>
<Version>1.0</Version>
<Format>1.0</Format>
</ReportDetails>
</ReportHeader>
<ReportBody>
<StateMessage MessageTime="20170314180133.857000+000" SerialNumber="18823">
<Topic ID="611" Type="611" IDType="0" User="" UserSID=""/>
<State ID="100" Criticality="0"/>
<UserParameters Flags="0" Count="2">
<Param>GUID:0EE65490-9075-41B1-B64D-AAAAAAAAAAAA</Param>
<Param>0</Param>
</UserParameters>
</StateMessage>
</ReportBody>
</Report>
Cause: In a ConfigMgr 1610 environment, and in 1602, 1606 versions of this were available as well, the cause of these messages is because that client is a member of a collection, where either by accident or design, that collection has the setting for "All devices are part of the same server group". The collection contained 40% of all clients in the environment--and in our case checking the box was NOT by design--it was accidental.
What that does, is two things we observed (and others have documented 1 of them, see links below).
1) as per one of the links below, machines in that collection may not ever patch as expected, again. that's because it thinks it's part of a cluster, and if it's not... it's waiting for it's "turn" to patch.
2) Every single device in that collection, once per minute, locally does two "schedule triggers", for two different things:
{00000000-0000-0000-0000-000000000111} -- which is for "Send Unsent State Message"
{00000000-0000-0000-0000-000000000116} -- which is for "State system policy bulk send low"
You'll see that over and over and over again locally on the client in SMSClientMethodProvider.log
and that apparently ends up as .swd files in the statesys box to be processed by the database, with TopicID 611, Type 611. If it's enough devices, and it's hotfix Tuesday and lots of state messages anyway (*cough* for example *cough*)--the auth\statesys.box inbox may become backlogged, and never catch up.
Remediation:
I'm a SQL person, so using this sql query, identify the collections which have that checkbox checked--and confirm you really meant it (If you are here reading this blog post... you likely didn't mean it). If not, uncheck the box for "All devices are part of the same server group" on the collections listed.
;with UseCluster as (select c.SiteID as [CollectionID] from CEP_CollectionExtendedProperties CEP
join collections_g c on CEP.Collectionid=c.Collectionid
where usecluster=1)
select c.*
from UseCluster
join v_collection c on c.collectionid=UseCluster.CollectionID
Monitoring for if this happens again:
Links:
https://blogs.technet.microsoft.com/enterprisemobility/2016/05/16/update-1605-for-configuration-manager-technical-preview-available-now/
https://docs.microsoft.com/en-us/sccm/core/servers/deploy/install/release-notes
https://docs.microsoft.com/en-us/sccm/core/get-started/capabilities-in-technical-preview-1605#BKMK_ServerGroups
https://social.technet.microsoft.com/Forums/en-US/86783d86-0e38-4cb4-acf8-6110acc76c0e/configmgr-1602-error-0x87d006662016410010-while-installing-update?forum=configmanagersecurity
- Created on .