Hi all,
We’re using the source connector with Avro as schema format, it seems that the output.schema.value does not register the namespace property (We modified the default schema and added namespace).
This creates problems for us with deserialization in our Kotlin consumers.
Can you please assist? Thanks!
Hi @Sason_Braha,
I’m not sure exactly what you want to achieve in the end and what your actual problem on the consumption side (Kotlin consumer) is.
In case you’re referring to the avro schema of the source connector which is not making use of the namespace field, what would you want/need it to contain in your case? The thing is, there are different ways to work with the namespace field of Avro. It would be helpful if you can provide a concrete example to better understand your question. Thanks in advance.
Hi @hpgrahsl, thank you.
I’ll elaborate.
We took the default schema from the source connector documentation under output.schema.value and modified it as following:
Here’s our schema: mongo-source.avsc · GitHub.
We use the schema above in output.schema.value. When the schema is registered by the connector in the schema registry - the namespace field is not included.
This creates a problem for us in Kotlin, where we can’t include namespace in the import because deserialization will fail, instead of:
import com.company.common.avro.MongoSourceChangeEvent
We are forced to remove the namespace from the schema and import it without namespace (the schema being registered is also being used to auto generate Avro classes) :
import MongoSourceChangeEvent.
We want to have a proper namespace for our auto generated classes and currently this issue prevents us from doing so.
Did you resolve this at all? We have just hit the same issue, we need namespace to be there but despite being set in the “output.schema.value” the namespace is not coming through to the avro schema on the topic
Hi @Oli_Allan1 and @Sason_Braha,
This sounds like a feature that could be added. I’ve added KAFKA-246 to track the feature for the next release.
It sounds like the work may have already been done in a fork? We’d happily take a PR GitHub - mongodb/mongo-kafka: MongoDB Kafka Connector
Ross
@Ross_Lawley thanks very much, will await the next release (havent done any work on a fork my end)
@Ross_Lawley When you are targeting its release? I am also facing similar issue, I am trying this with .c# avro auto generated class with namespace.
Hi @Ross_Lawley
I am not able to build the PR, It has some test case failure issue, Can you please check and let me know.
I don’t think a PR has been created for this feature yet. Its not planned for this quarter.
Regarding the test failure, you’d have to look at the test report to understand more about what failed and why.
Ross
Hi @Sason_Braha, I’m not sure if this is still an issue but using this single message transform, SetSchemaMetadata | Confluent Documentation, might give you what you need.
Hi @Sason_Braha ,
KAFKA-246 filed based on your post was done and is included in version 1.7.0 of MongoDB Kafka Connector.
This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.
