pykafka.utils.struct_helpers¶
Author: Keith Bourgoin, Emmett Butler
-
pykafka.utils.struct_helpers.
unpack_from
(fmt, buff, offset=0)¶ A customized version of struct.unpack_from
This is a conveinence function that makes decoding the arrays, strings, and byte arrays that we get from Kafka significantly easier. It takes the same arguments as struct.unpack_from but adds 3 new formats:
- Wrap a section in [] to indicate an array. e.g.: [ii]
- S for strings (int16 followed by byte array)
- Y for byte arrays (int32 followed by byte array)
Spaces are ignored in the format string, allowing more readable formats
- NOTE: This may be a performance bottleneck. We’re avoiding a lot of memory
- allocations by using the same buffer, but if we could call struct.unpack_from only once, that’s about an order of magnitude faster. However, constructing the format string to do so would erase any gains we got from having the single call.