Author: Keith Bourgoin, Emmett Butler
KafkaClient(hosts='127.0.0.1:9092', zookeeper_hosts=None, socket_timeout_ms=30000, offsets_channel_socket_timeout_ms=10000, use_greenlets=False, exclude_internal_topics=True, source_address='')¶
A high-level pythonic client for Kafka
NOTE: KafkaClient holds weak references to Topic instances via
pykafka.cluster.TopicDict. To perform operations directly on these topics, such as examining their partition lists, client code must hold a strong reference to the topics it cares about. If client code doesn’t need to examine Topic instances directly, no strong references are necessary.
__init__(hosts='127.0.0.1:9092', zookeeper_hosts=None, socket_timeout_ms=30000, offsets_channel_socket_timeout_ms=10000, use_greenlets=False, exclude_internal_topics=True, source_address='')¶
Create a connection to a Kafka cluster.
Documentation for source_address can be found at https://docs.python.org/2/library/socket.html#socket.create_connection
- hosts (bytes) – Comma-separated list of kafka hosts to which to connect.
- zookeeper_hosts (bytes) – KazooClient-formatted string of ZooKeeper hosts to which to connect. If not None, this argument takes precedence over hosts
- socket_timeout_ms (int) – The socket timeout (in milliseconds) for network requests
- offsets_channel_socket_timeout_ms (int) – The socket timeout (in milliseconds) when reading responses for offset commit and offset fetch requests.
- use_greenlets (bool) – Whether to perform parallel operations on greenlets instead of OS threads
- exclude_internal_topics (bool) – Whether messages from internal topics (specifically, the offsets topic) should be exposed to the consumer.
- source_address (str ‘host:port’) – The source address for socket connections
list of weak references to the object (if defined)
Update known brokers and topics.
Updates each Topic and Broker, adding new ones as found, with current metadata from the cluster.