pykafka.cluster¶
-
class
pykafka.cluster.
Cluster
(hosts, handler, socket_timeout_ms=30000, offsets_channel_socket_timeout_ms=10000, exclude_internal_topics=True, source_address='')¶ Bases:
object
A Cluster is a high-level abstraction of the collection of brokers and topics that makes up a real kafka cluster.
-
__init__
(hosts, handler, socket_timeout_ms=30000, offsets_channel_socket_timeout_ms=10000, exclude_internal_topics=True, source_address='')¶ Create a new Cluster instance.
Parameters: - hosts (bytes) – Comma-separated list of kafka hosts to used to connect.
- handler (
pykafka.handlers.Handler
) – The concurrency handler for network requests. - socket_timeout_ms (int) – The socket timeout (in milliseconds) for network requests
- offsets_channel_socket_timeout_ms (int) – The socket timeout (in milliseconds) when reading responses for offset commit and offset fetch requests.
- exclude_internal_topics (bool) – Whether messages from internal topics (specifically, the offsets topic) should be exposed to consumers.
- source_address (str ‘host:port’) – The source address for socket connections
-
__weakref__
¶ list of weak references to the object (if defined)
-
_get_metadata
(topics=None)¶ Get fresh cluster metadata from a broker.
-
_update_brokers
(broker_metadata)¶ Update brokers with fresh metadata.
Parameters: broker_metadata (Dict of {name: metadata} where metadata is pykafka.protocol.BrokerMetadata
and name is str.) – Metadata for all brokers.
-
brokers
¶ The dict of known brokers for this cluster
-
get_offset_manager
(consumer_group)¶ Get the broker designated as the offset manager for this consumer group.
Based on Step 1 at https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+consumer+offsets+in+Kafka
Parameters: consumer_group (str) – The name of the consumer group for which to find the offset manager.
-
handler
¶ The concurrency handler for network requests
-
topics
¶ The dict of known topics for this cluster
-
update
()¶ Update known brokers and topics.
-