Skip to main content

Deserialization error causing Kafka Streams to stop working even in presence of DEFAULT_DESERIALIZATION_EXCEPTION_HANDLER_CLASS_CONFIG property

If you are getting an exception during your kafka stream thread processing for which stack trace resembles what is below and you are wondering why your DeserializationExceptionHandler is not being invoked (which is written to ignore a deserialization error on a particular record), you are probably like me who have spent hours trying to figure out that.

Ultimately I had to look closely on the stack trace and underlying Kafka streams library, I figured out this error is not occurring when a record is being processed from Kafka topic. Instead, it is being processed when a record from a State store is being retrieved (Notice the StateSerdes in the exception stack trace).

Kafka doesn't use the handler in DEFAULT_DESERIALIZATION_EXCEPTION_HANDLER_CLASS_CONFIG when it reads the RocksDB files (see that the stacktrace mentions the class StateSerdes). That's why it works fine for records coming from the source topic, but fails when deserializing the data in the table.

You would need to look at why the data gets corrupted in the State store itself to reproduce the error.

Failed to process stream task 0_4 due to the following error:

java.lang.RuntimeException: Error deserializing data '' for object 

at org.apache.kafka.streams.state.internals.ValueAndTimestampDeserializer.deserialize(ValueAndTimestampDeserializer.java:54) ~[kafka-streams-2.3.1.jar!/:?]

at org.apache.kafka.streams.state.internals.ValueAndTimestampDeserializer.deserialize(ValueAndTimestampDeserializer.java:27) ~[kafka-streams-2.3.1.jar!/:?]

at org.apache.kafka.streams.state.StateSerdes.valueFrom(StateSerdes.java:160) ~[kafka-streams-2.3.1.jar!/:?]

at org.apache.kafka.streams.state.internals.MeteredKeyValueStore.outerValue(MeteredKeyValueStore.java:267) ~[kafka-streams-2.3.1.jar!/:?]

at org.apache.kafka.streams.state.internals.MeteredKeyValueStore.get(MeteredKeyValueStore.java:148) ~[kafka-streams-2.3.1.jar!/:?]

at org.apache.kafka.streams.processor.internals.ProcessorContextImpl$KeyValueStoreReadWriteDecorator.get(ProcessorContextImpl.java:464) ~[kafka-streams-2.3.1.jar!/:?]

at org.apache.kafka.streams.kstream.internals.KTableSource$KTableSourceProcessor.process(KTableSource.java:102) ~[kafka-streams-2.3.1.jar!/:?]

at org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:117) ~[kafka-streams-2.3.1.jar!/:?]

at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:201) ~[kafka-streams-2.3.1.jar!/:?]

at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:180) ~[kafka-streams-2.3.1.jar!/:?]

at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:133) ~[kafka-streams-2.3.1.jar!/:?]

at org.apache.kafka.streams.processor.internals.SourceNode.process(SourceNode.java:87) ~[kafka-streams-2.3.1.jar!/:?]

at org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:363) ~[kafka-streams-2.3.1.jar!/:?]

at org.apache.kafka.streams.processor.internals.AssignedStreamsTasks.process(AssignedStreamsTasks.java:199) [kafka-streams-2.3.1.jar!/:?]

at org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:425) [kafka-streams-2.3.1.jar!/:?]

at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:912) [kafka-streams-2.3.1.jar!/:?]

at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:819) [kafka-streams-2.3.1.jar!/:?]

at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:788) [kafka-streams-2.3.1.jar!/:?]

Caused by: java.io.EOFException

at org.apache.avro.io.JsonDecoder.advance(JsonDecoder.java:126) ~[avro-1.9.0.11p.jar!/:1.9.0.11p]

at org.apache.avro.io.ExtendedJsonDecoder.readString(ExtendedJsonDecoder.java:249) ~[avro-1.9.0.11p.jar!/:1.9.0.11p]

at org.apache.avro.io.ResolvingDecoder.readString(ResolvingDecoder.java:220) ~[avro-1.9.0.11p.jar!/:1.9.0.11p]

at org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:372) ~[avro-1.9.0.11p.jar!/:1.9.0.11p]

at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:145) ~[avro-1.9.0.11p.jar!/:1.9.0.11p]

at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:126) ~[avro-1.9.0.11p.jar!/:1.9.0.11p]

at org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:212) ~[avro-1.9.0.11p.jar!/:1.9.0.11p]

at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:202) ~[avro-1.9.0.11p.jar!/:1.9.0.11p]

at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:133) ~[avro-1.9.0.11p.jar!/:1.9.0.11p]

at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:126) ~[avro-1.9.0.11p.jar!/:1.9.0.11p]

at org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:212) ~[avro-1.9.0.11p.jar!/:1.9.0.11p]

at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:202) ~[avro-1.9.0.11p.jar!/:1.9.0.11p]

at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:133) ~[avro-1.9.0.11p.jar!/:1.9.0.11p]

at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:126) ~[avro-1.9.0.11p.jar!/:1.9.0.11p]

at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:103) ~[avro-1.9.0.11p.jar!/:1.9.0.11p]

Comments

Popular posts from this blog

Administrative Assistant Job at Kazar Group Dubai, United Arab Emirates

Currently has an open position for a determined Administrative Assistant who is deeply passionate about growing their career. You will take charge of a number of different office support and administrative duties as the Administrative Assistant. Do you embrace the tasks of mail merging, pivot tables, and presentation design? Then we have just the Administrative Assistant position for you. A long-term contract / temporary to hire opportunity, this Administrative Assistant opportunity is located in the Dubai, UAE. How you will make an impact Handle telephone calls Back various projects for other employees Organize word processors, files, and faxes Greet and guide visitors Requirements At least 1 year of Administrative Assistant experience preferred Command of Clerical Duties Customer service experience Comprehensive knowledge of Administrative Assistance Proven ability to use the internet for research Knowledgeable computer skills including word processing, spreadsheets and presentation

IT Support Engineer Job in Dubai United Arab Emirates

We have Openings for IT Support Engineer in Dubai, UAE Location. Please find the details below Job Location : Dubai, UAE Position: IT Support Engineer Payroll Company: Raqmiyat. Contract Duration: 12 Months Extendable Contract. Candidates should be available inside the country UAE. Job Description:- Advanced Computer Technical skills (Hardware, Software and Windows 10 operating system and Office 365 applications) and system application. - Basic hands-on experience for Network technology and troubleshooting. - Basic hands-on experience for Radio and Telephony - Basic hands-on experience for SQL and data base troubleshooting - ITIL Certification is Must. - 6 Days a week & Should be ready to work on 24/7 Environment including night shifts. If Interested please send your updated profile and your details to  ashok.mk@raqmiyat.com