pykafka.cluster

class pykafka.cluster.Cluster(hosts, handler, socket_timeout_ms=30000, offsets_channel_socket_timeout_ms=10000, exclude_internal_topics=True, source_address='')

Bases: object

A Cluster is a high-level abstraction of the collection of brokers and topics that makes up a real kafka cluster.

__init__(hosts, handler, socket_timeout_ms=30000, offsets_channel_socket_timeout_ms=10000, exclude_internal_topics=True, source_address='')

Create a new Cluster instance.

Parameters:
  • hosts (str) – Comma-separated list of kafka hosts to used to connect.
  • handler (pykafka.handlers.Handler) – The concurrency handler for network requests.
  • socket_timeout_ms (int) – The socket timeout (in milliseconds) for network requests
  • offsets_channel_socket_timeout_ms (int) – The socket timeout (in milliseconds) when reading responses for offset commit and offset fetch requests.
  • exclude_internal_topics (bool) – Whether messages from internal topics (specifically, the offsets topic) should be exposed to consumers.
  • source_address (str ‘host:port’) – The source address for socket connections
__weakref__

list of weak references to the object (if defined)

_get_metadata()

Get fresh cluster metadata from a broker.

_should_exclude_topic(topic_name)

Should this topic be excluded from the list shown to the client?

_update_brokers(broker_metadata)

Update brokers with fresh metadata.

Parameters:broker_metadata (Dict of {name: metadata} where metadata is pykafka.protocol.BrokerMetadata and name is str.) – Metadata for all brokers.
_update_topics(metadata)

Update topics with fresh metadata.

Parameters:metadata (Dict of {name, metadata} where metadata is pykafka.protocol.TopicMetadata and name is str.) – Metadata for all topics.
brokers

The dict of known brokers for this cluster

get_offset_manager(consumer_group)

Get the broker designated as the offset manager for this consumer group.

Based on Step 1 at https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+consumer+offsets+in+Kafka

Parameters:consumer_group (str) – The name of the consumer group for which to find the offset manager.
handler

The concurrency handler for network requests

topics

The dict of known topics for this cluster

update()

Update known brokers and topics.