Jar issues while connecting to mongodb atlas from dataproc 2.1-debian11

MongoDB Atlas
, ,
1

Hi, I’m trying to connect to mongodb-altas using dataproc i’m using:
MongoDB 6.0.14 Enterprise
dataproc image: --image-version 2.1-debian11

using this script to gererate spark session:

spark = SparkSession.builder
.appName(“PySpark_BigQuery_Interaction”)
.config(‘spark.sql.debug.maxToStringFields’, 100000)
.config(‘spark.sql.shuffle.partitions’, ‘400’)
.config(“spark.packages”, “com.google.cloud.spark:spark-bigquery-with-dependencies_2.12:0.17.1,gs://spark-mongodb-jars-111/jars/mongo-java-driver-3.9.1.jar,gs://spark-mongodb-jars-111/jars/mongo-spark-connector_2.12-2.4.0.jar”)
.config(‘spark.sql.inMemoryColumnarStorage.compressed’, True)
.config(‘spark.sql.autoBroadcastJoinThreshold’, -1)
.config(‘spark.storage.memoryFraction’, ‘0.6’)
.config(‘spark.dynamicAllocation.enabled’, ‘true’)
.config(‘maxParallelism’, 10000)
.config(‘preferredMinParallelism’, 2700)
.config(‘spark.executor.memory’, ‘3g’)
.config(‘spark.executor.cores’, 2)
.config(‘spark.executor.instances’, 1)
.config(‘spark.executor.memoryOverhead’, 1000)
.config(‘spark.driver.memoryOverhead’, ‘500m’)
.config(‘spark.driver.memory’, ‘3gb’)
.config(“spark.dynamicAllocation.minExecutors”, 1)
.config(“spark.dynamicAllocation.maxExecutors”, 8)
.config(“spark.dynamicAllocation.initialExecutors”, 2)
.config(“spark.sql.legacy.timeParserPolicy”, “LEGACY”)
.getOrCreate()

df = spark.read.format(“mongodb”)
.option(“spark.mongodb.connection.uri”,url)
.option(“database”,“Test-DB”)
.option(“collection”, “test”)
.load()

and getting this error:

: java.lang.ClassNotFoundException: Failed to find data source: com.mongodb.spark.sql.DefaultSource. Please find packages at http://spark.apache.org/third-party-projects.html
	at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:678)
	at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:213)
	at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:186)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:238)
	at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.ClassNotFoundException: com.mongodb.spark.sql.DefaultSource.DefaultSource
	at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	at org.apache.spark.sql.execution.datasources.DataSource$.$anonfun$lookupDataSource$5(DataSource.scala:652)
	at scala.util.Try$.apply(Try.scala:213)
	at org.apache.spark.sql.execution.datasources.DataSource$.$anonfun$lookupDataSource$4(DataSource.scala:652)
	at scala.util.Failure.orElse(Try.scala:224)
	at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:652)
	... 13 more
```
any help will be much appricated
2

Hi @Yaniv_Kempler , How did you add the MongoDB Spark connector package/JAR file (which is an external library) to your Dataproc environment? There is some discussion regarding this topic here - import - use an external library in pyspark job in a Spark cluster from google-dataproc - Stack Overflow

3

Thanks, solved by adding
–properties ^#^spark:spark.jars.packages=org.mongodb.spark:mongo-spark-connector:10.0.5,com.google.cloud.spark:spark-bigquery-with-dependencies_2.12:0.36.1#

to dataproc creation