HCL Workload Automation, Version 9.4

IBM BigInsights jobs

An IBM BigInsights job defines, schedules, monitors, and manages IBM BigInsights Workbook data sheets or applications.

Prerequisite

You manage IBM BigInsights Workbook data sheets or applications in both a distributed and z/OS environment.

HCL Workload Automation integrates with IBM BigInsights for Hadoop to bring the power of Apache Hadoop to the enterprise. With the HCL Workload Automation plug-in for BigInsights for Hadoop you can:

Monitor and control workflows containing IBM BigInsights workbooks and applications that help enterprise find insights into new and emerging types of data.
Fully automate IBM BigInsights process execution with calendar and event-based scheduling, and a single point of control to handle exceptions, and automate recovery processes.

Before you can define IBM BigInsights jobs, you must create a connection between the HCL Workload Automation agent and the IBM BigInsights server.

IBM BigInsights job definition

A description of the job properties and valid values are detailed in the context-sensitive help in the Dynamic Workload Console by clicking the question mark (?) icon in the top-right corner of the properties pane.

For more information about creating jobs using the various supported product interfaces, see Defining a job.

The following table lists the required and optional attributes for IBM BigInsights jobs:

Table 1. Required and optional attributes for the definition of an IBM BigInsights job
Attribute	Description and value	Required
Connection properties - IBM BigInsights server section
Hostname	The hostname of the IBM BigInsights server.	✓
Port	The port of the IBM BigInsights server.	✓
Protocol	The protocol for connecting to the IBM BigInsights server. Supported values are http and https.
User	The user to be used for accessing the IBM BigInsights server.
Password	The password to be used for accessing the IBM BigInsights server.
Connection properties - Retry options section
Number of retries	The number of times the program retries performing the operation.
Retry interval (seconds)	The number of seconds the program waits before retrying the operation. The default value is 30 seconds.
Action properties - Workbook section
Workbook	The name and path to an IBM BigInsights workbook. Use this option to run a user-specified workbook.	✓
Action properties - Application section
Application Identifier	The application identifier. Use this option to run an Application.	✓
Execution Name	The user-defined identifier for a specific run of the application

Scheduling and stopping the job in HCL Workload Automation

You schedule HCL Workload Automation IBM BigInsights jobs by defining them in job streams. Add the job to a job stream with all the necessary scheduling arguments and submit the job stream.

You can submit jobs by using the Dynamic Workload Console, Application Lab or the conman command line. See Scheduling and submitting jobs and job streams for information about how to schedule and submit jobs and job streams using the various interfaces.

After submission, when the job is running and is reported in EXEC status in HCL Workload Automation, you can stop it if necessary, by using the kill command. However, this action is effective only for the Wait for a file action. If you have defined different actions in your job, the kill command is ignored.

Monitoring the job

If the HCL Workload Automation agent stops when you submit the HCL Workload Automation IBM BigInsights job or while the job is running, as soon as the agent becomes available again HCL Workload Automation begins monitoring the job from where it stopped.

For information about how to monitor jobs using the different product interfaces available, see Monitoring HCL Workload Automation jobs.

Job properties

While the job is running, you can track the status of the job and analyze the properties of the job. In particular, in the Extra Information section, if the job contains variables, you can verify the value passed to the variable from the remote system. Some job streams use the variable passing feature, for example, the value of a variable specified in job 1, contained in job stream A, is required by job 2 in order to run in the same job stream.

For information about how to display the job properties from the various supported interfaces, see Analyzing the job log.

For example, from the conman command line, you can see the job properties by running:

conman sj <job_name>;props

where <job_name> is the IBM BigInsights job name.

The properties are listed in the Extra Information section of the output command.

For more information about passing variables between jobs, see Passing job properties from one job to another in the same job stream instance and User's Guide and Reference: Passing variables between jobs in the same job stream instance.

Job log content

For information about how to display the job log from the various supported interfaces, see Analyzing the job log.

For example, you can see the job log content by running conman sj <job_name>;stdlist, where <job_name> is the IBM BigInsights job name.