We are seeing the following on an Exchange 2016 box:
It did drop below 10% space on the queue drive briefly and had to have Log Name: Application
Source: MSExchangeApplicationLogic
Date: 18/03/2020 11:17:32
Event ID: 3025
Task Category: Extension
Level: Error
Keywords: Classic
User: N/A
Computer: %computername%.domain.local
Description:
Scenario: GetKillBit. Failed to download killbit list from OMEX server. Exception: System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
--- End of inner exception stack trace ---
at System.Net.TlsStream.EndWrite(IAsyncResult asyncResult)
at System.Net.ConnectStream.WriteHeadersCallback(IAsyncResult ar)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at Microsoft.Exchange.Data.ApplicationLogic.Extension.BaseAsyncOmexCommand.c__DisplayClass2.b__1()
Event Xml:
[code=markup]<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="MSExchangeApplicationLogic" />
<EventID Qualifiers="49156">3025</EventID>
<Level>2</Level>
<Task>3</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2020-03-18T11:17:32.000000000Z" />
<EventRecordID>14931349</EventRecordID>
<Channel>Application</Channel>
<Computer>%computername%.domain.local</Computer>
<Security />
</System>
<EventData>
<Data>GetKillBit</Data>
<Data>System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
--- End of inner exception stack trace ---
at System.Net.TlsStream.EndWrite(IAsyncResult asyncResult)
at System.Net.ConnectStream.WriteHeadersCallback(IAsyncResult ar)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at Microsoft.Exchange.Data.ApplicationLogic.Extension.BaseAsyncOmexCommand.<>c__DisplayClass2.<EndGetResponseCallback>b__1()</Data>
</EventData>
</Event>[/code]
This was around the time of a space issue of the queue disk going below 10% which meant clearing some log files.
The disk was never full though, so why it would have affected the download I don't know...
Posted by: Lemonde - 18 March 2020 11:55:42
Appears to coincide with:
Log Name: Application
Source: MSExchangeApplicationLogic
Date: 18/03/2020 11:17:32
Event ID: 3018
Task Category: Extension
Level: Error
Keywords: Classic
User: N/A
Computer: %computername%.domain.local
Description:
Scenario[ServiceHealth]: GetKillBit. CorrelationId: 704d6dcb-9b37-45fa-9789-2a71bd37ca50. The request failed. Mailbox: Url: https://store.office.com/appinfo/query?rt=XML&corr=704d6dcb-9b37-45fa-9789-2a71bd37ca50 Exception: System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
--- End of inner exception stack trace ---
at System.Net.TlsStream.EndWrite(IAsyncResult asyncResult)
at System.Net.ConnectStream.WriteHeadersCallback(IAsyncResult ar)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at Microsoft.Exchange.Data.ApplicationLogic.Extension.BaseAsyncOmexCommand.<>c__DisplayClass2.<EndGetResponseCallback>b__1()
Event Xml:
[code=markup]<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="MSExchangeApplicationLogic" />
<EventID Qualifiers="49156">3018</EventID>
<Level>2</Level>
<Task>3</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2020-03-18T11:17:32.000000000Z" />
<EventRecordID>14931348</EventRecordID>
<Channel>Application</Channel>
<Computer>%computername%.domain.local</Computer>
<Security />
</System>
<EventData>
<Data>GetKillBit</Data>
<Data>704d6dcb-9b37-45fa-9789-2a71bd37ca50</Data>
<Data>
</Data>
<Data>https://store.office.com/appinfo/query?rt=XML&corr=704d6dcb-9b37-45fa-9789-2a71bd37ca50</Data>
<Data>System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
--- End of inner exception stack trace ---
at System.Net.TlsStream.EndWrite(IAsyncResult asyncResult)
at System.Net.ConnectStream.WriteHeadersCallback(IAsyncResult ar)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at Microsoft.Exchange.Data.ApplicationLogic.Extension.BaseAsyncOmexCommand.<>c__DisplayClass2.<EndGetResponseCallback>b__1()</Data>
</EventData>
</Event>[/code]
Posted by: Lemonde - 26 March 2020 08:00:15
This issue has been known to Microsoft for at least 6 years since Exchange 2013 was first released but still no solution.
The killbit.xml file is in use according to the message, but it is still updated by the system at the time of this message, so probably nothing to worry about, but they could at least update the logging system so as not to record it.
Posted by: Lemonde - 21 April 2020 18:14:16
Is there a reg fix to stop this appearing does anyone know, what can be done to take the service down a peg in the Event Log?
