Message queue file self-monitoring

When the HPOM management server receives messages, it stores them temporarily in a message queue file before processing them. The maximum size of a this file is 2 GB. The operating system components that the management server uses do not support larger files. If the file size grows beyond 2 GB, the file becomes corrupt. The management server loses messages from the file, and cannot process new messages.

Under normal conditions this problem does not occur. However, the following circumstances could lead the message queue file to grow uncontrollably:

• Message storms (if message storm detection is not enabled).
• Many new messages arriving, but slow processing of the message queue due to DNS name resolution problems.
• Processing of the message queue is suspended due to WMI problems. In this case new messages are still accepted.

To prevent the message queue file becoming corrupt, the management server monitors the size of this file. By default, if the file size approaches 2 GB, the management server rejects new messages until the file size returns to normal. Agents start to buffer rejected messages locally, and try to resend them later. Therefore, you do not lose the messages that the management server rejects.

The management server sends a high-priority message to connected consoles when the file approaches 2 GB, and also when the file drops back to a safe size. When it sends these high-priority messages, the management server bypasses the agent and the message queue. (You cannot create high-priority messages using policies.)

You can change this default behavior in the following ways:

• Define whether the management server continues to accept messages after the message queue file size reaches the maximum.
• Define different message queue file size thresholds.
• Configure the severity and message key of the high-priority messages.
• Configure automatic and operator-initiated commands for the high-priority messages.

Configuration values for message queue file self-monitoring

You can configure message queue file monitoring by changing values in the Message Queue File Monitoring namespace using the Server Configuration dialog box. The following table describes the values that you can change.

Testing message queue file self-monitoring

Before you use customized message queue file self-monitoring in a production environment, test whether it works as you expect. Consider the following questions:

• Do you receive high-priority messages when the message queue file size reaches the configured maximum and then when the file returns to a size below the configured rearm file size? Does the first message have the configured priority?
• Do the high-priority messages have the configured automatic and operator-initiated commands? Do the commands run correctly and is the output correct?
• Does automatic acknowledgement work correctly?
• Does the management server reject incoming messages when the message queue file size reaches the configured maximum? Does the management server accept messages again when the file returns to a size below the configured rearm file size?

Normally, it is difficult to create tests that cause the message queue file size to reach the configured maximum. The permitted ranges for configuration values suit production environments, but for testing it is desirable to configure smaller values.

In a test environment, you can specify values that are smaller than normally permitted for Message queue maximum file size and Message queue rearm file size by defining negative values. The management server interprets these negative values as test values and automatically converts to them to positive values without checking whether the values are permitted.

Caution:
Perform tests in a test environment. Do not use negative values in production environments.

The following example explains a test case that sends a high-priority message if the message queue file size reaches 20000 bytes. In this example, the management server continues to accept new messages. When the message queue file returns to size below 10000 bytes, the management server sends a second high-priority message.

1. Set the following values appropriately:
2. • Severity of queue warning messages
   • Message key prefix for queue warning messages
   • Automatic action for queue warning messages
   • Operator-initiated action for queue warning messages
3. Set Message queue monitoring mode to 1.
4. Set Message queue maximum file size to -20000
5. Set Message queue rearm file size to -10000.
6. Use opcmsg to create test messages and use Windows Explorer to check the size of the file %OvShareDir%\tmp\queues\MsgQueue. When the file size reaches 20000 bytes (which happens quickly), check that a high-priority message appears in the console for the management server node.
7. Check whether the automatic command completes successfully and that the results appear as an annotation to the high-priority message.
8. Start the operator-initiated command, and check the results.
9. Continue to send new messages until the message queue file size reaches approximately 265000 bytes. The file size automatically reduces. Check that a second high-priority message appears in the console.
10. Check whether the automatic and operator-initiated commands complete successfully for the second high-priority message.
11. Check that the second high-priority message automatically acknowledges the first high-priority message.

Related Topics:

• Message storm detection and suppression
• Optimize HPOM node name resolution
• Generic server configuration
• opcmsg