Troubleshooting “Canceled Web Request”

Recently, when calling web services from the BizTalk environment, we were seeing intermittent instances of the “WebException: The request was aborted: The request was canceled” error message in the application Event Log. This occurred mostly under heavy load situations, but could be duplicated even with fairly small load.

If you search online for this exception, you’ll often see folks just say to “turn off keep-alives” which we all agreed was a cheap way to solve a issue while introducing performance problems. To dig further into why this connection to the WebLogic server was getting dropped, we actually began listening in on protocol communication using Wireshark. I started going bleary-eyed looking for ACK and FIN and everything else, so I went and applied .NET tracing to the BizTalk configuration file (btsntsvc.exe.config). The BizTalk documentation shows you how to set up System.Net logging in BizTalk (for more information on the settings, you can read this).

My config is slightly different, but looks like this …

<system.diagnostics>
    <sources>
      <source name="System.Net">
        <listeners>
          <add name="System.Net"/>
        </listeners>
      </source>
      <source name="System.Net.Sockets">
        <listeners>
          <add name="System.Net"/>
        </listeners>
      </source>
      <source name="System.Net.Cache">
        <listeners>
          <add name="System.Net"/>
        </listeners>
      </source>
    </sources>
    <switches>
      <add name="System.Net" value="Verbose" />
      <add name="System.Net.Sockets" value="Error" />
      <add name="System.Net.Cache"  value="Verbose" />
    </switches>
    <sharedListeners>
      <add name="System.Net"
           type="System.Diagnostics.TextWriterTraceListener"
           initializeData="c:\BizTalkTrace.log"   />
    </sharedListeners>
    <trace autoflush="true" />
  </system.diagnostics>
  

What this yielded was a great log file containing a more readable format than reading straight network communication. Specifically, when the request was canceled message showed up in the Event Log, I jumped to the .NET trace log and found this message …
A connection that was expected to be kept alive was closed by the server.

So indeed, keep-alives were causing a problem. Much more useful than the message that pops up in the Event Log. When we did turn keep-alives off on the destination WebLogic server (just as a test), the problem went away. But, that couldn’t be our final solution. We finally discovered that the keep-alive timeouts were different on the .NET box (BizTalk) and the WebLogic server. WebLogic had a keep-alive of 30 seconds, while it appears that the .NET framework uses the same value for keep-alives as for service timeouts (e.g. 90 seconds). The Windows box was attempting to reuse a connection while the WebLogic box had disconnected it. So, we modified the WebLogic application by synchronizing the keep-alive timeout with the Windows/BizTalk box, and the problem went away complete.

Now, we still get the responsible network connection pooling of keep-alives, while still solving the problem of canceled web requests.

Technorati Tags:



Categories: .NET, BizTalk, General Architecture

9 replies

  1. Dear Richard

    You went for changing the Keep Alive time on the WebLogic server side, but this is not always possible to make changes on other servers, like our situation. Is there a way we can adjust the Http Keep Alive time, on the Windows 2003 Server/BizTalk box?

    Thanks & Regards
    Sajid

  2. Hey Sajid,

    I looked at that, and if I recall, it either wasn’t easy, or wasn’t possible.

  3. Great work!! Specially not just stopping at eazy/quick solution to turn off KeepAlive!!

  4. Hello,

    I have a similar issue with connecting from BizTalk SOAP adapter to WebService in SAP XI.

    Checked that the keep alive time is 60 seconds in XI. What is the Keep Alive time in BizTalk? (default). Seems like it is also 60 seconds. Then why is the issue?

    Any thoughts?

  5. You said the keep alive time is same a httptimeout. I changed the httptimeout in machine.config and did a retest. However still the issue occured under high load.

    Any thoughts?

    Thanks
    Sandeep

  6. Sandeep, I think the default keep alive timeout is 90 seconds. So are you specifically seeing the “A connection that was expected to be kept alive was closed by the server” error in your logs?

  7. Yes i see that when i switch ON trace using the method u described above.

    Yes the parameter is set to 90 seconds by default. I changed it to 60 seconds but still the timeout.

    Someone from XI just told me the value in XI is 5 mins.

    WIll this still occur if the timeout set in XI is 5mins and Biztalk is 60 seconds?

    Please let me know.

    Thanks
    Sandeep

Trackbacks

  1. Setting “KeepAlive” Value in BizTalk Web Service Calls « Richard Seroter’s Architecture Musings
  2. The Adapter Diaries - Oracle Adapter - 3 « Santosh Benjamin’s Weblog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: