Introduction

This document provides an overview of the application server and its integrated health state monitoring and is addressed to application administrators.

Glossary

References

• Reliability Engineering

• Guide - 7.10.38.21 Server Health Checks

• Guide - 7.10 API Changes

Process Description

The application server evaluates its own healthiness in two distinct phases reacting to a changing environment to improve its reliability.

First, during application server start-up, self checks annotated with @SelfCheckTaskExecutionInPhase(SelfCheckTaskExecutionPhase.STARTUP) or without such annotation are executed to perform checks which should determine if the application server and its connected services (like the database) are in the expected state/configuration for start-up. If the start-up checks return a combined result status (described below) of DOWN, the start will be aborted with a respective exception, otherwise the start will continue as usual.

Second, during application server runtime, self checks annotated with @SelfCheckTaskExecutionInPhase(SelfCheckTaskExecutionPhase.RUNTIME) are executed in a configurable interval to perform checks which should determine if the application server and its connected services are in the expected state for runtime. If the runtime checks return a combined result status (described below) of OUT_OF_SERVICE, the server will be taken temporarily out of service with a respective warning while such a result status persists. If the returned status is DOWN, the server will shutdown with a respective error. Otherwise, the server will continue to function as usual.

All self checks can be executed during both or only one desired phase. It is possible to also exclude certain self checks per configuration. All self checks are configurable for their specific requirements if needed. Exceptions should be handled by each self check.

A try-run configuration is available to convert the final action of the checks, such as e.g. server termination, into a logging-only action. Therefore, no actual actions except logging are executed according to the check outcome with try-run enabled.

Result of Checks - Combination of Results

Each SelfCheckTask yields in a SelfCheckTaskOutcome. For each set of collectively executed tasks the encapsulating SelfCheckResult contains a list of SelfCheckTaskResults (containing the task and its outcome), whereas the final SelfCheckResult status will be the one from the list with the highest priority. The SelfCheckTaskOutcome status priorities are sorted in ascending order in the following priority enumeration:

• UNKNOWN: -1

• UP: 0

• OUT_OF_SERVICE: 1

• DOWN: 2

Result of Checks - Server Status Change Mapping

The outcome of the runtime self checks is combined as described above and is used to alter the server status if needed. Here you see what result yields to the following server status change action:

• UNKNOWN → NOOP

• UP → NOOP

• OUT_OF_SERVICE → Pause Application Server

• DOWN → Stop Application Server

Health Checks

Database Connection

Checks if the database server answers a simple ping request.
Executed during: STARTUP, RUNTIME

Database Configurations

Multiple checks (catalog collation, available full text search, read committed snapshot, transaction isolation level) to assure the database server is configured as expected.
Executed during: STARTUP

Sites Folder Existence

Checks if the sites root directory is accessible (actually writable) by the current (operating system) user.
Executed during: STARTUP, RUNTIME

HTTP Status Code Health

Checks if the error rate of storefront and REST HTTP response status codes is below configurable thresholds.
Executed during: RUNTIME

Expected application server behavior:

Response Time Health

Checks if the response time of storefront and REST responses is below configurable threshold percentile given a minimum error threshold time.
Executed during: RUNTIME

Health Checks in ICM 7.10

Process Differences

If the combined result of the checks results in SelfCheckTaskOutcome DOWN, the JVM process will be terminated with exit code 1. This enables the node manager to restart the application server.

Available at versions:

• 7.10.38.21-LTS

• 7.10.40.4

Health Checks

Database Connection

Checks if the database server answers are executed during runtime only, due to the different startup behavior of the application server.
Also, an equivalent Oracle database server ping check was introduced.
Executed during: RUNTIME

Shared Directory Existence Instead of Sites Folder Existence

Checks if the shared directory is available via the file system and if it exists.
Executed during: STARTUP, RUNTIME

HTTP Status Code Health

Checks if the error rate of storefront and REST HTTP response status codes is below configurable thresholds.
Executed during: RUNTIME

Response Time Health

Checks if the response time of storefront and REST responses is below configurable threshold percentile given a minimum error threshold time.
Executed during: RUNTIME