RedHat EAP 7 Undertow Performance Tuning

Problem Statement

In our project we were using long polling http servlets using Netty framework. RedHat from EAP 6 TO EAP 7 has moved underlying web container from JBossWeb to Undertow framework. We faced issue in EAP 7 with default settings, where after some HTTP client requests the server was stopped responding to subsequent requests.

Enviornement

• Oracle Linux 7.x 64-bit

• 8 core CPU, 16 GM RAM

• RedHat EAP 7.2 base version

• Java Oracle 1.8

• Max Client HTTP blocking requests - 5000

  • these are long polling requests being kept open for 24x7

Solution Description

The problem we found was, the default tuning parameters comes along with EAP 7 undertow.

Default Settings

• The undertow subsystem relies on the io subsystem to provide XNIO workers. This option specifies the XNIO worker that the listener uses. By default, a listener uses the default worker in the io subsystem.

Example 1. EAP 7 Domain.xml (before tuning)

<subsystem xmlns="urn:jboss:domain:undertow:7.0" default-server="default-server" default-virtual-host="default-host" default-servlet-container="default" default-security-domain="other">
    <buffer-cache name="default"/>
 <server name="default-server">
  <http-listener name="default" socket-binding="http" redirect-socket="https" enable-http2="true"/>
  <https-listener name="https" socket-binding="https" ssl-context="spirit-server-ssl-context" enable-http2="true"/>

<subsystem xmlns="urn:jboss:domain:io:3.0">
 <worker name="default"/>
 <buffer-pool name="default"/>
</subsystem>

• All listeners are tied to an XNIO Worker instance. Usually there will only be a single worker instance that is shared between listeners, however it is possible to create a new worker for each listener. The worker instance manages the listeners IO threads, and also the default blocking task thread pool.

  • IO Threads, perform non blocking tasks, default thread count is cpuCount * 2

  • Task Threads, perform blocking operations (such as Servlet requests), default thread count is cpuCound * 16

• With 5000 HTTP requests load (long polling) we observed only 128 requests served correctly, remaining requests were stuck with no HTTP response coming to clients. This was due to default worker thread pool settings which in our case, 8 core CPU * 16, meaning 128 threads.

After Tuning

Example 2. EAP 7 Domain.xml (after tuning)

<subsystem xmlns="urn:jboss:domain:undertow:7.0" default-server="default-server" default-virtual-host="default-host" default-servlet-container="default" default-security-domain="other">
    <buffer-cache name="default"/>
 <server name="default-server">
  <http-listener name="default" max-connections="100000" socket-binding="http" redirect-socket="https" enable-http2="false" worker="http-worker"/>
  <https-listener name="https" max-connections="100000" socket-binding="https" ssl-context="spirit-server-ssl-context" enable-http2="false" worker="http-worker"/>

<subsystem xmlns="urn:jboss:domain:io:3.0">
 <worker name="default"/>
 <worker name="http-worker" task-core-threads="7000" task-max-threads="7000" task-keepalive="60000" />
 <buffer-pool name="default"/>
</subsystem>

Test Results

• With 5000 HTTP requests load (long polling) and tuning we observed all 5000 requests served correctly. This was due to tuning where we made worker thread pool settings as 7000 core (first) and max (after) threads.

References

• http://undertow.io/undertow-docs/undertow-docs-1.2.0/listeners.html

• https://wildscribe.github.io/JBoss%20EAP/7.2/subsystem/io/worker/index.html

Last updated 2020-03-29 12:28:46 +0550