HCL Workload Automation, Version 9.4

Saturation of DB2 transaction log halts processing of jobs

Processing of agent for z/OS jobs terminates when the DB2 transaction log fills up.

If the DB2 transaction log becomes full while a job is running, processing of the job is interrupted and the connection between the agent and the dynamic workload broker is stopped. Any agent for z/OS jobs that are yet to run hang idle and, in the particular case that the submitted plan contains only agent for z/OS jobs, it stops altogether.

Look for the following messages to be sure that a saturated DB2 transaction log is the problem:

In the EELMLOG on z/OS:

EELHT15E THE HTTP CLIENT FAILED TO PROCESS A REQUEST FOR BROKER
EELHT43I HTTP RESPONSE MESSAGE WITH CODE RDBMS_TRANSPORT_PROBLEM

On HCL Workload Automation

From BATCHMAN (occurs on the running job when the transaction log becomes full):

job_name has failed with the error: An error occurred reading the 
job from the job table

job_name has failed with the error: AWKJDB801E An internal error has 
been found while accessing the database. The internal error message is:
"Not enough storage is available in the application heap to process"

AWSBHT032I Workstation broker_workstation_name is now inactive, 
no jobs will be scheduled.

From MAILMAN (reporting that the agent and the dynamic workload broker are unlinking):

AWSBCV082I Workstation broker_workstation_name, message: AWSDEB014I 
Connection timed out
AWSBCV027I Unlinking from broker_workstation_name

From BATCHMAN (after dynamic workload broker has unlinked and job submission to the agent has stopped):

AWSBDY103I Received command MY:UNLINK for run number 42 for 
workstation broker_workstation_name from workstation 
 workstation_name

Workstation broker_workstation_name State is being changed: UNSETTING: 
LINKED=TCP AWSBHT032I Workstation broker_workstation_name is now 
inactive, no jobs will be scheduled.

In the SystemOut.log file:

AWSJDB801E An internal error has been found while accessing the 
database.The internal error message is: "Not enough storage is 
available in the application heap to process the statement.. 
SQLCODE=-954, SQLSTATE=57011, DRIVER=3.61.75".

In the db2diag.log file of the DB2 server:

MESSAGE : ZRC=0x85100009=-2062548983=SQLP_NOSPACE
          "Log File has reached its saturation point"
          DIA8309C Log file was full.

RETCODE : ZRC=0x8B0F0001=-1961951231=SQLO_NOMEM_APPH
          "No memory available in 'Application Heap'"
          DIA8301C No memory available in the application heap.

The saturation of the DB2 server transaction log is due to the fact that the log size is insufficient because of its default settings. To prevent it from becoming saturated in the future, change the DB2 configuration settings to at least the following values:

Log file size (4KB)                (LOGFILSIZ) = 10000
Number of primary log files        (LOGPRIMARY) = 80
Number of secondary log files      (LOGSECOND) = 40

See the DB2 documentation for further information.