| Duration (ms) |
Specify a time in milliseconds. This value is the amount of time
the step will spend collecting records prior to the execution of the
transformation. If this option set to a value of
0, then Number of records triggers
consumption. Either the Duration or the Number
of records option must contain a value greater than
0 to run the transformation.
Note: You must set this field if you are
using Spark as your processing engine.
|
| Number of records |
Specify a number. After every ‘X’ number of records, the specified
transformation will be executed and these ‘X’ records will be passed to the
transformation. If this option set to a value of
0 then Duration triggers
consumption. Either the Duration or the Number
of records option must contain a value greater than
0 to run the transformation.
|
| Maximum concurrent batches |
Specify the maximum number of batches used to collect records at the same time.
The default value is 1, which indicates a single
batch is used for collecting records. This option should only be used when your
consumer step cannot keep pace with the speed at which the data is streaming. Your
computing environment must have adequate CPU and memory for this implementation.
An error will occur if your environment cannot handle the maximum number of
concurrent batches specified.
|
| Message prefetch limit |
Specify a limit for how many incoming messages this step will
queue for processing, as they are received from the broker. Setting this value
forces the broker to manage the backpressure of messages exceeding the specified
limit. The default number of messages to queue is
100000. |
| Ack management |
Select one of the two following options to manage the
acknowlegement:
- Ack when message received
- Select this option to indicate an automatic acknowlegement of
message receipt when the broker sends the message.
Important: If you
select this option, be aware that some messages might be lost before being
consumed.
- Ack when batch completed
- Select this option to indicate to the broker that a batch of
messages has been consumed, after the batch is processed, by this step.
Important: This option is required to
ensure that all messages are consumed by this step.
|