FlinkTasksClient

class statefun_tasks.client.FlinkTasksClient(kafka_broker_url, request_topics, action_topics, reply_topic, group_id=None, serialiser=None, kafka_properties=None, kafka_consumer_properties=None, kafka_producer_properties=None)

Client for submitting TaskRequests / TaskActionRequests

Replies are handled on a dedicated thread created per instance so FlinkTasksClientFactory.get_client() is preferred to instantiating this class directly.

Parameters:

kafka_broker_url – url of the kafka broker (or list of urls) used for ingress and egress
request_topics – dictionary of worker to ingress topic mappings (use None for default) e.g. {‘example/worker’: ‘example.requests’, None: ‘example.default.requests’}
action_toptics – as per request_topics but used for action requests
reply_topic – topic to listen on for responses (a unique consumer group id will be created)
group_id (optional) – kafka group id to use when subscribing to reply_topic
serialiser (optional) – serialiser to use (will use DefaultSerialiser if not set)
kafka_properties (optional) – additional properties to be passed to the KafkaConsumer and KafkaProducer
kafka_consumer_properties (optional) – additional properties to be passed to the KafkaConsumer
kafka_producer_properties (optional) – additional properties to be passed to the KafkaProducer

__init__(kafka_broker_url, request_topics, action_topics, reply_topic, group_id=None, serialiser=None, kafka_properties=None, kafka_consumer_properties=None, kafka_producer_properties=None)

Methods

`__init__`(kafka_broker_url, request_topics, ...)
`cancel_pipeline`(pipeline_or_task[, topic])	Cancels a pipeline
`cancel_pipeline_async`(pipeline_or_task[, topic])	Cancels a pipeline
`get_request`(pipeline_or_task[, topic])	Get the original TaskRequest for a previously submitted pipeline or task
`get_request_async`(pipeline_or_task[, topic])	Get the original TaskRequest for a previously submitted pipeline or task
`get_result`(pipeline_or_task[, topic])	Get the TaskResult for a previously submitted pipeline or task
`get_result_async`(pipeline_or_task[, topic])	Get the TaskResult for a previously submitted pipeline or task
`get_status`(pipeline_or_task[, topic])	Get the status of a previously submitted pipeline or task
`get_status_async`(pipeline_or_task[, topic])	Get the status of a previously submitted pipeline or task
`pause_pipeline`(pipeline_or_task[, topic])	Pauses a pipeline
`pause_pipeline_async`(pipeline_or_task[, topic])	Pauses a pipeline
`submit`(pipeline[, topic])	Submit a pipeline to Flink
`submit_async`(pipeline[, topic])	Submit a pipeline to Flink
`unpause_pipeline`(pipeline_or_task[, topic])	Unpauses a pipeline
`unpause_pipeline_async`(pipeline_or_task[, topic])	Unpauses a pipeline
`serialiser`	Returns the serialiser used by this client

Attributes

serialiser

Returns the serialiser used by this client

cancel_pipeline(pipeline_or_task, topic=None)

Cancels a pipeline

Parameters:

pipeline_or_task (Union[PipelineBuilder, Task]) – the pipeline or task
topic (optional) – override the default ingress topic

Return type:

Future

Returns:

a Future indicating whether the pipeline was successfully paused or not

async cancel_pipeline_async(pipeline_or_task, topic=None)

Cancels a pipeline

Parameters:

pipeline_or_task (Union[PipelineBuilder, Task]) – the pipeline or task
topic (optional) – override the default ingress topic

get_request(pipeline_or_task, topic=None)

Get the original TaskRequest for a previously submitted pipeline or task

Parameters:

pipeline_or_task (Union[PipelineBuilder, Task]) – the pipeline or task
topic (optional) – override the default ingress topic

Return type:

Future

Returns:

a Future encapsulating the original TaskRequest

async get_request_async(pipeline_or_task, topic=None)

Get the original TaskRequest for a previously submitted pipeline or task

Parameters:

pipeline_or_task (Union[PipelineBuilder, Task]) – the pipeline or task
topic (optional) – override the default ingress topic

Returns:

the original TaskRequest

get_result(pipeline_or_task, topic=None)

Get the TaskResult for a previously submitted pipeline or task

Parameters:

pipeline_or_task (Union[PipelineBuilder, Task]) – the pipeline or task
topic (optional) – override the default ingress topic

Return type:

Future

Returns:

a Future encapsulating the original TaskResult

async get_result_async(pipeline_or_task, topic=None)

Get the TaskResult for a previously submitted pipeline or task

Parameters:

pipeline_or_task (Union[PipelineBuilder, Task]) – the pipeline or task
topic (optional) – override the default ingress topic

Returns:

the original TaskResult

get_status(pipeline_or_task, topic=None)

Get the status of a previously submitted pipeline or task

Parameters:

pipeline_or_task (Union[PipelineBuilder, Task]) – the pipeline or task
topic (optional) – override the default ingress topic

Return type:

Future

Returns:

a Future encapsulating the status of the pipeline

async get_status_async(pipeline_or_task, topic=None)

Get the status of a previously submitted pipeline or task

Parameters:

pipeline_or_task (Union[PipelineBuilder, Task]) – the pipeline or task
topic (optional) – override the default ingress topic

Returns:

the status of the pipeline

pause_pipeline(pipeline_or_task, topic=None)

Pauses a pipeline

Parameters:

pipeline_or_task (Union[PipelineBuilder, Task]) – the pipeline or task
topic (optional) – override the default ingress topic

Return type:

Future

Returns:

a Future indicating whether the pipeline was successfully paused or not

async pause_pipeline_async(pipeline_or_task, topic=None)

Pauses a pipeline

Parameters:

pipeline_or_task (Union[PipelineBuilder, Task]) – the pipeline or task
topic (optional) – override the default ingress topic

property serialiser: DefaultSerialiser: Returns the serialiser used by this client

submit(pipeline, topic=None)

Submit a pipeline to Flink

Parameters:

pipeline (PipelineBuilder) – the pipeline
topic (optional) – override the default ingress topic

Return type:

Future

Returns:

a Future encapsulating the result of the pipeline

async submit_async(pipeline, topic=None)

Submit a pipeline to Flink

Parameters:

pipeline (PipelineBuilder) – the pipeline
topic (optional) – override the default ingress topic

Returns:

the result of the pipeline

unpause_pipeline(pipeline_or_task, topic=None)

Unpauses a pipeline

Parameters:

pipeline_or_task (Union[PipelineBuilder, Task]) – the pipeline or task
topic (optional) – override the default ingress topic

Return type:

Future

Returns:

a Future indicating whether the pipeline was successfully paused or not

async unpause_pipeline_async(pipeline_or_task, topic=None)

Unpauses a pipeline

Parameters:

pipeline_or_task (Union[PipelineBuilder, Task]) – the pipeline or task
topic (optional) – override the default ingress topic