pykafka.cluster

class pykafka.cluster.Cluster(hosts, handler, socket_timeout_ms=30000, offsets_channel_socket_timeout_ms=10000, exclude_internal_topics=True, source_address='')

Bases: object

A Cluster is a high-level abstraction of the collection of brokers and topics that makes up a real kafka cluster.

__init__(hosts, handler, socket_timeout_ms=30000, offsets_channel_socket_timeout_ms=10000, exclude_internal_topics=True, source_address='')

Create a new Cluster instance.

Parameters:
  • hosts (bytes) – Comma-separated list of kafka hosts to used to connect. Also accepts a KazooClient connect string
  • handler (pykafka.handlers.Handler) – The concurrency handler for network requests.
  • socket_timeout_ms (int) – The socket timeout (in milliseconds) for network requests
  • offsets_channel_socket_timeout_ms (int) – The socket timeout (in milliseconds) when reading responses for offset commit and offset fetch requests.
  • exclude_internal_topics (bool) – Whether messages from internal topics (specifically, the offsets topic) should be exposed to consumers.
  • source_address (str ‘host:port’) – The source address for socket connections
__weakref__

list of weak references to the object (if defined)

_get_metadata(topics=None)

Get fresh cluster metadata from a broker.

_request_metadata(broker_connects, topics)

Request broker metadata from a set of brokers

Returns the result of the first successful metadata request

Parameters:broker_connects (Iterable of two-element sequences of the format (broker_host, broker_port)) – The set of brokers to which to attempt to connect
_update_brokers(broker_metadata)

Update brokers with fresh metadata.

Parameters:broker_metadata (Dict of {name: metadata} where metadata is pykafka.protocol.BrokerMetadata and name is str.) – Metadata for all brokers.
brokers

The dict of known brokers for this cluster

get_offset_manager(consumer_group)

Get the broker designated as the offset manager for this consumer group.

Based on Step 1 at https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+consumer+offsets+in+Kafka

Parameters:consumer_group (str) – The name of the consumer group for which to find the offset manager.
handler

The concurrency handler for network requests

topics

The dict of known topics for this cluster

update()

Update known brokers and topics.