Streaming Write Configuration Options
On this page
Overview
You can configure the following properties when writing data to MongoDB in streaming mode.
Note
If you use SparkConf to set the connector's write configurations,
prefix spark.mongodb.write. to each property.
Property name | Description |
|---|---|
connection.uri | Required. The connection string configuration key. Default: mongodb://localhost:27017/ |
database | Required. The database name configuration. |
collection | Required. The collection name configuration. |
checkpointLocation | The absolute file path of the directory to which the connector writes checkpoint
information. For more information about checkpoints, see the
Spark Structured Streaming Programming Guide Default: None |
forceDeleteTempCheckpointLocation | A Boolean value that specifies whether to delete existing checkpoint data. Default: false |
mongoClientFactory | MongoClientFactory configuration key. You can specify a custom implementation that must implement the
com.mongodb.spark.sql.connector.connection.MongoClientFactory
interface.Default: com.mongodb.spark.sql.connector.connection.DefaultMongoClientFactory |
Specifying Properties in connection.uri
If you use SparkConf to specify any of the previous settings, you can
either include them in the connection.uri setting or list them individually.
The following code example shows how to specify the
database, collection, and maxBatchSize setting as part of the connection.uri
setting:
spark.mongodb.write.connection.uri=mongodb://127.0.0.1/myDB.myCollection?maxBatchSize=256
To keep the connection.uri shorter and make the settings easier to read, you can
specify them individually instead:
spark.mongodb.write.connection.uri=mongodb://127.0.0.1/ spark.mongodb.write.database=myDB spark.mongodb.write.collection=myCollection spark.mongodb.write.maxBatchSize=256
Important
If you specify a setting in both the connection.uri and on its own line,
the connection.uri setting takes precedence.
For example, in the following configuration, the connection
database is foobar:
spark.mongodb.write.connection.uri=mongodb://127.0.0.1/foobar spark.mongodb.write.database=bar