Skip to main content

Posts

Showing posts from January, 2021

Deserialization error causing Kafka Streams to stop working even in presence of DEFAULT_DESERIALIZATION_EXCEPTION_HANDLER_CLASS_CONFIG property

If you are getting an exception during your kafka stream thread processing for which stack trace resembles what is below and you are wondering why your DeserializationExceptionHandler is not being invoked (which is written to ignore a deserialization error on a particular record), you are probably like me who have spent hours trying to figure out that. Ultimately I had to look closely on the stack trace and underlying Kafka streams library, I figured out this error is not occurring when a record is being processed from Kafka topic. Instead, it is being processed when a record from a State store is being retrieved (Notice the StateSerdes in the exception stack trace). Kafka doesn't use the handler in DEFAULT_DESERIALIZATION_EXCEPTION_HANDLER_CLASS_CONFIG when it reads the RocksDB files (see that the stacktrace mentions the class StateSerdes). That's why it works fine for records coming from the source topic, but fails when deserializing the data in the table. You would need to l