pykafka.handlers

Author: Keith Bourgoin, Emmett Butler

class pykafka.handlers.ResponseFuture(handler)

Bases: object

A response which may have a value at some point.

__init__(handler)
__weakref__

list of weak references to the object (if defined)

get(response_cls=None, timeout=None)

Block until data is ready and return.

Raises an exception if there was an error.

set_error(error)

Set error and trigger get method.

set_response(response)

Set response data and trigger get method.

class pykafka.handlers.Handler

Bases: object

Base class for Handler classes

__weakref__

list of weak references to the object (if defined)

spawn(target, *args, **kwargs)

Create the worker that will process the work to be handled

class pykafka.handlers.ThreadingHandler

Bases: pykafka.handlers.Handler

A handler that uses a threading.Thread to perform its work

Event(*args, **kwargs)

A factory function that returns a new event.

Events manage a flag that can be set to true with the set() method and reset to false with the clear() method. The wait() method blocks until the flag is true.

Lock()

allocate_lock() -> lock object (allocate() is an obsolete synonym)

Create a new lock object. See help(LockType) for information about locks.

class Queue(maxsize=0)

Create a queue object with a given maximum size.

If maxsize is <= 0, the queue size is infinite.

empty()

Return True if the queue is empty, False otherwise (not reliable!).

full()

Return True if the queue is full, False otherwise (not reliable!).

get(block=True, timeout=None)

Remove and return an item from the queue.

If optional args ‘block’ is true and ‘timeout’ is None (the default), block if necessary until an item is available. If ‘timeout’ is a non-negative number, it blocks at most ‘timeout’ seconds and raises the Empty exception if no item was available within that time. Otherwise (‘block’ is false), return an item if one is immediately available, else raise the Empty exception (‘timeout’ is ignored in that case).

get_nowait()

Remove and return an item from the queue without blocking.

Only get an item if one is immediately available. Otherwise raise the Empty exception.

join()

Blocks until all items in the Queue have been gotten and processed.

The count of unfinished tasks goes up whenever an item is added to the queue. The count goes down whenever a consumer thread calls task_done() to indicate the item was retrieved and all work on it is complete.

When the count of unfinished tasks drops to zero, join() unblocks.

put(item, block=True, timeout=None)

Put an item into the queue.

If optional args ‘block’ is true and ‘timeout’ is None (the default), block if necessary until a free slot is available. If ‘timeout’ is a non-negative number, it blocks at most ‘timeout’ seconds and raises the Full exception if no free slot was available within that time. Otherwise (‘block’ is false), put an item on the queue if a free slot is immediately available, else raise the Full exception (‘timeout’ is ignored in that case).

put_nowait(item)

Put an item into the queue without blocking.

Only enqueue the item if a free slot is immediately available. Otherwise raise the Full exception.

qsize()

Return the approximate size of the queue (not reliable!).

task_done()

Indicate that a formerly enqueued task is complete.

Used by Queue consumer threads. For each get() used to fetch a task, a subsequent call to task_done() tells the queue that the processing on the task is complete.

If a join() is currently blocking, it will resume when all items have been processed (meaning that a task_done() call was received for every item that had been put() into the queue).

Raises a ValueError if called more times than there were items placed in the queue.

class ThreadingHandler.Semaphore(value=1)

Bases: object

This class implements semaphore objects.

Semaphores manage a counter representing the number of release() calls minus the number of acquire() calls, plus an initial value. The acquire() method blocks if necessary until it can return without making the counter negative. If not given, value defaults to 1.

Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015 Python Software Foundation. All rights reserved.

__enter__(blocking=True, timeout=None)

Acquire a semaphore, decrementing the internal counter by one.

When invoked without arguments: if the internal counter is larger than zero on entry, decrement it by one and return immediately. If it is zero on entry, block, waiting until some other thread has called release() to make it larger than zero. This is done with proper interlocking so that if multiple acquire() calls are blocked, release() will wake exactly one of them up. The implementation may pick one at random, so the order in which blocked threads are awakened should not be relied on. There is no return value in this case.

When invoked with blocking set to true, do the same thing as when called without arguments, and return true.

When invoked with blocking set to false, do not block. If a call without an argument would block, return false immediately; otherwise, do the same thing as when called without arguments, and return true.

When invoked with a timeout other than None, it will block for at most timeout seconds. If acquire does not complete successfully in that interval, return false. Return true otherwise.

__weakref__

list of weak references to the object (if defined)

acquire(blocking=True, timeout=None)

Acquire a semaphore, decrementing the internal counter by one.

When invoked without arguments: if the internal counter is larger than zero on entry, decrement it by one and return immediately. If it is zero on entry, block, waiting until some other thread has called release() to make it larger than zero. This is done with proper interlocking so that if multiple acquire() calls are blocked, release() will wake exactly one of them up. The implementation may pick one at random, so the order in which blocked threads are awakened should not be relied on. There is no return value in this case.

When invoked with blocking set to true, do the same thing as when called without arguments, and return true.

When invoked with blocking set to false, do not block. If a call without an argument would block, return false immediately; otherwise, do the same thing as when called without arguments, and return true.

When invoked with a timeout other than None, it will block for at most timeout seconds. If acquire does not complete successfully in that interval, return false. Return true otherwise.

release()

Release a semaphore, incrementing the internal counter by one.

When the counter is zero on entry and another thread is waiting for it to become larger than zero again, wake up that thread.

ThreadingHandler.Socket = <module 'socket' from '/home/docs/.pyenv/versions/2.7.13/lib/python2.7/socket.pyc'>
class pykafka.handlers.RequestHandler(handler, connection)

Bases: object

Uses a Handler instance to dispatch requests.

class Shared(connection, requests, ending)

Bases: tuple

__getnewargs__()

Return self as a plain tuple. Used by copy and pickle.

__getstate__()

Exclude the OrderedDict from pickling

static __new__(_cls, connection, requests, ending)

Create new instance of Shared(connection, requests, ending)

__repr__()

Return a nicely formatted representation string

_asdict()

Return a new OrderedDict which maps field names to their values

classmethod _make(iterable, new=<built-in method __new__ of type object at 0x7b4ba0>, len=<built-in function len>)

Make a new Shared object from a sequence or iterable

_replace(_self, **kwds)

Return a new Shared object replacing specified fields with new values

connection

Alias for field number 0

ending

Alias for field number 2

requests

Alias for field number 1

class RequestHandler.Task(request, future)

Bases: tuple

__getnewargs__()

Return self as a plain tuple. Used by copy and pickle.

__getstate__()

Exclude the OrderedDict from pickling

static __new__(_cls, request, future)

Create new instance of Task(request, future)

__repr__()

Return a nicely formatted representation string

_asdict()

Return a new OrderedDict which maps field names to their values

classmethod _make(iterable, new=<built-in method __new__ of type object at 0x7b4ba0>, len=<built-in function len>)

Make a new Task object from a sequence or iterable

_replace(_self, **kwds)

Return a new Task object replacing specified fields with new values

future

Alias for field number 1

request

Alias for field number 0

RequestHandler.__init__(handler, connection)
RequestHandler.__weakref__

list of weak references to the object (if defined)

RequestHandler._start_thread()

Run the request processor

RequestHandler.request(request, has_response=True)

Construct a new request

Parameters:has_response – Whether this request will return a response
Returns:pykafka.handlers.ResponseFuture
RequestHandler.start()

Start the request processor.

RequestHandler.stop()

Stop the request processor.