scala-在Databricks runtime 10.3上运行spark应用程序时版本冲突
相关标签： # kotlin
我在Databricks集群上运行scala/spark应用程序（它尝试使用EventHub schemaregistry从事件中心主题读取数据），并得到以下错误：
22/02/17 07:36:14 ERROR JacksonVersion: Version '1.0.19-SNAPSHOT' of package 'jackson-dataformat-xml' is not supported (older than earliest supported version - `2.10.0`), please upgrade.
22/02/17 07:36:14 ERROR JacksonVersion: Version '1.0.19-SNAPSHOT' of package 'jackson-datatype-jsr310' is not supported (older than earliest supported version - `2.10.0`), please upgrade.
22/02/17 07:36:14 INFO JacksonVersion: Package versions: jackson-annotations=2.12.3, jackson-core=2.12.3, jackson-databind=2.12.3, jackson-dataformat-xml=1.0.19-SNAPSHOT, jackson-datatype-jsr310=1.0.19-SNAPSHOT, azure-core=1.8.1, Troubleshooting version conflicts: https://aka.ms/azsdk/java/dependency/troubleshoot
22/02/17 07:36:14 ERROR ScalaDriverLocal: User Code Stack Trace:
java.lang.NoSuchMethodError: reactor.netty.http.client.HttpClient.resolver(Lio/netty/resolver/AddressResolverGroup;)Lreactor/netty/transport/ClientTransport;
看起来maven正在尝试使用主工件版本（1.0.19-SNAPSHOT）来“jackson dataformat xml”和“jackson-datatype-jsr310”：
22/02/17 07:36:14 ERROR JacksonVersion: Version '1.0.19-SNAPSHOT' of package 'jackson-dataformat-xml' is not supported (older than earliest supported version - `2.10.0`), please upgrade.
22/02/17 07:36:14 ERROR JacksonVersion: Version '1.0.19-SNAPSHOT' of package 'jackson-datatype-jsr310' is not supported (older than earliest supported version - `2.10.0`), please upgrade.
该项目使用Databricks 10.3 runtime和以下pom：
4.0.0
group
parent
3.1-SNAPSHOT
ingestion
${revision}
jar
ingestion
company
1.0.19-SNAPSHOT
UTF-8
org.apache.logging.log4j
log4j-core
org.apache.spark
spark-core_${scala.compact.version}
provided
org.apache.spark
spark-sql_${scala.compact.version}
provided
org.apache.spark
spark-streaming_${scala.compact.version}
provided
org.apache.spark
spark-avro_${scala.compact.version}
provided
io.delta
delta-core_${scala.compact.version}
org.scalatest
scalatest_${scala.compact.version}
3.2.11
test
org.mockito
mockito-scala_${scala.compact.version}
test
org.yaml
snakeyaml
io.spray
spray-json_${scala.compact.version}
com.databricks
dbutils-api_${scala.compact.version}
com.github.mrpowers
spark-daria_${scala.compact.version}
com.github.mrpowers
spark-fast-tests_${scala.compact.version}
joda-time
joda-time
io.circe
circe-yaml_${scala.compact.version}
io.circe
circe-generic_${scala.compact.version}
io.jvm.uuid
scala-uuid_2.12
org.apache.httpcomponents
httpclient
provided
org.apache.kafka
kafka-clients
com.typesafe
config
com.azure
azure-identity
com.azure
azure-core
com.fasterxml.jackson.dataformat
jackson-dataformat-xml
2.12.3
com.fasterxml.jackson.datatype
jackson-datatype-jsr310
2.12.3
com.azure
azure-data-schemaregistry
commons-cli
commons-cli
com.azure
azure-core-http-netty
src/main/resources
false
src/main/resources-filtered
true
net.alchim31.maven
scala-maven-plugin
scala-compile
compile
testCompile
${scala.version}
-Xms64m
-Xmx1024m
org.apache.maven.plugins
maven-jar-plugin
3.0.2
${project.artifactId}
${project.version}
${project.groupId}
${project.artifactId}
${project.groupId}
${project.version}
${project.groupId}
org.codehaus.mojo
flatten-maven-plugin
1.2.7
true
resolveCiFriendliesOnly
flatten
process-resources
flatten
flatten.clean
clean
clean
org.apache.maven.plugins
maven-assembly-plugin
3.3.0
make-assembly
package
single
${project.artifactId}-${project.version}-assembly
false
jar-with-dependencies
${project.artifactId}
${project.version}
${project.groupId}
${project.artifactId}
${project.groupId}
${project.version}
${project.groupId}
org.scalatest
scalatest-maven-plugin
2.0.0
${project.build.directory}/surefire-reports
test
test
org.scalastyle
scalastyle-maven-plugin
compile
check
这是母项目：
group
parent
3.1-SNAPSHOT
pom
2.12.14
2.12
3.2.1
0.0.5
1.1.0
42.2.9
1.4.0
1.25
1.3.5
1.4.200
1.16.46
1.14.0
1.6.6
0.8.0
2.7
1.0
1.2.3
1.1.0
3.1.0
4.4.0
2.0.0
0.12.3
1.0-m5.1
2.10.6
8.2.2.jre8
circe-yaml_2.12
0.13.1
circe-generic_2.12
0.13.0
0.3.1
0.14.0
3.0.5
4.5.6
1.4.0
1.23.0
1.0.0
2.17.1
1.2
1.0.15
1.11.7
org.apache.logging.log4j
log4j-core
${log4j-core.version}
org.apache.spark
spark-core_${scala.compact.version}
${spark.version}
provided
org.apache.spark
spark-sql_${scala.compact.version}
${spark.version}
provided
org.apache.spark
spark-streaming_${scala.compact.version}
${spark.version}
provided
org.apache.spark
spark-mllib_${scala.compact.version}
${spark.version}
provided
org.apache.kafka
kafka-clients
${kafka.version}
org.apache.kafka
kafka-streams
${kafka.version}
com.fasterxml.jackson.core
jackson-databind
org.apache.spark
spark-avro_${scala.compact.version}
${spark.version}
org.scala-lang
scala-library
${scala.version}
provided
com.microsoft.azure
adal4j
${adal4j.version}
com.databricks
dbutils-api_2.12
${databricks.dbutils.api.version}
org.scala-lang
scala-library
provided
joda-time
joda-time
${joda-time.version}
org.apache.httpcomponents
httpclient
${httpclient.version}
provided
io.delta
delta-core_${scala.compact.version}
${delta.version}
provided
com.typesafe
config
${typesafe.config.version}
org.yaml
snakeyaml
${snakeyaml.version}
io.circe
${circe-yaml.artifact}
${circe-yaml.version}
io.circe
${circe-generic.artifact}
${circe-generic.version}
io.jvm.uuid
scala-uuid_2.12
${scala-uuid_2.12.version}
io.spray
spray-json_${scala.compact.version}
${spray-json.version}
org.mockito
mockito-scala_${scala.compact.version}
${mockito.version}
test
org.typelevel
cats-core_2.12
${cats-core.version}
org.scalacheck
scalacheck_${scala.compact.version}
${scalacheck.version}
test
com.holdenkarau
spark-testing-base_${scala.compact.version}
2.4.5_${spark-testing-base.version}
org.scalatest
scalatest_${scala.compact.version}
${scalatest.version}
test
com.github.mrpowers
spark-fast-tests_${scala.compact.version}
${spark-fast-tests.version}
test
com.github.mrpowers
spark-daria_${scala.compact.version}
${spark-daria.version}
com.azure
azure-identity
${azure-identity.version}
com.azure
azure-core-http-netty
com.azure
azure-core
${azure-core.version}
com.fasterxml.jackson.dataformat
jackson-dataformat-xml
com.fasterxml.jackson.datatype
jackson-datatype-jsr310
io.projectreactor.netty
reactor-netty
com.azure
azure-core-http-netty
${azure-core-http-netty.version}
com.azure
azure-data-schemaregistry
${azure-data-schemaregistry.version}
com.azure
azure-core-http-netty
commons-cli
commons-cli
${commons-cli.version}
external.atlassian.jgitflow
jgitflow-maven-plugin
${jgitflow-maven-plugin.version}
true
true
true
true
true
master
develop
feature/
release/
hotfix/
${project.artifactId}-
net.alchim31.maven
scala-maven-plugin
${scala-maven-plugin.version}
compile
compile
testCompile
-Xms1024M
-Xmx2048M
-XX:+CMSClassUnloadingEnabled
org.scalastyle
scalastyle-maven-plugin
${scalastyle.version}
true
true
true
false
${project.basedir}/src/main/scala
${project.basedir}/src/test/scala
${project.basedir}/scalastyle_config.xml
${project.basedir}/scalastyle-output.xml
UTF-8
check
org.apache.maven.plugins
maven-surefire-plugin
${maven-surefire-plugin.version}
true
jitpack.io
https://jitpack.io
false
dl.binatry.com
http://dl.bintray.com/spark-packages/maven
false
当我在本地运行它时，它工作正常，当试图在集群中运行它时，问题就出现了。
我试图排除一些冲突版本的人工制品，但问题依然存在。
有什么帮助吗？
特别声明：以上内容（图片及文字）均为互联网收集或者用户上传发布，本站仅提供信息存储服务！如有侵权或有涉及法律问题请联系我们。
举报
