返回

scala-在Databricks runtime 10.3上运行spark应用程序时版本冲突

发布时间:2022-02-23 15:35:52 320
# kotlin

我在Databricks集群上运行scala/spark应用程序(它尝试使用EventHub schemaregistry从事件中心主题读取数据),并得到以下错误:

22/02/17 07:36:14 ERROR JacksonVersion: Version '1.0.19-SNAPSHOT' of package 'jackson-dataformat-xml' is not supported (older than earliest supported version - `2.10.0`), please upgrade.
22/02/17 07:36:14 ERROR JacksonVersion: Version '1.0.19-SNAPSHOT' of package 'jackson-datatype-jsr310' is not supported (older than earliest supported version - `2.10.0`), please upgrade.
22/02/17 07:36:14 INFO JacksonVersion: Package versions: jackson-annotations=2.12.3, jackson-core=2.12.3, jackson-databind=2.12.3, jackson-dataformat-xml=1.0.19-SNAPSHOT, jackson-datatype-jsr310=1.0.19-SNAPSHOT, azure-core=1.8.1, Troubleshooting version conflicts: https://aka.ms/azsdk/java/dependency/troubleshoot
22/02/17 07:36:14 ERROR ScalaDriverLocal: User Code Stack Trace: 
java.lang.NoSuchMethodError: reactor.netty.http.client.HttpClient.resolver(Lio/netty/resolver/AddressResolverGroup;)Lreactor/netty/transport/ClientTransport;

看起来maven正在尝试使用主工件版本(1.0.19-SNAPSHOT)来“jackson dataformat xml”和“jackson-datatype-jsr310”:

22/02/17 07:36:14 ERROR JacksonVersion: Version '1.0.19-SNAPSHOT' of package 'jackson-dataformat-xml' is not supported (older than earliest supported version - `2.10.0`), please upgrade.
22/02/17 07:36:14 ERROR JacksonVersion: Version '1.0.19-SNAPSHOT' of package 'jackson-datatype-jsr310' is not supported (older than earliest supported version - `2.10.0`), please upgrade.

该项目使用Databricks 10.3 runtime和以下pom:


4.0.0


    group
    parent
    3.1-SNAPSHOT


ingestion
${revision}
jar
ingestion

    company



    1.0.19-SNAPSHOT
    UTF-8



    
        org.apache.logging.log4j
        log4j-core
    
    
        org.apache.spark
        spark-core_${scala.compact.version}
        provided
    
    
        org.apache.spark
        spark-sql_${scala.compact.version}
        provided
    
    
        org.apache.spark
        spark-streaming_${scala.compact.version}
        provided
    
    
        org.apache.spark
        spark-avro_${scala.compact.version}
        provided
    
    
        io.delta
        delta-core_${scala.compact.version}
    
    
        org.scalatest
        scalatest_${scala.compact.version}
        3.2.11
        test
    
    
        org.mockito
        mockito-scala_${scala.compact.version}
        test
    
    
        org.yaml
        snakeyaml
    
    
        io.spray
        spray-json_${scala.compact.version}
    
    
        com.databricks
        dbutils-api_${scala.compact.version}
    
    
        com.github.mrpowers
        spark-daria_${scala.compact.version}
    
    
        com.github.mrpowers
        spark-fast-tests_${scala.compact.version}
    
    
        joda-time
        joda-time
    
    
        io.circe
        circe-yaml_${scala.compact.version}
    
    
        io.circe
        circe-generic_${scala.compact.version}
    
    
        io.jvm.uuid
        scala-uuid_2.12
    
    
        org.apache.httpcomponents
        httpclient
        provided
    
    
        org.apache.kafka
        kafka-clients
    
    
        com.typesafe
        config
    
    
        com.azure
        azure-identity
    
    
        com.azure
        azure-core
    
    
        com.fasterxml.jackson.dataformat
        jackson-dataformat-xml
        2.12.3
    
    
        com.fasterxml.jackson.datatype
        jackson-datatype-jsr310
        2.12.3
    
    
        com.azure
        azure-data-schemaregistry
    
    
        commons-cli
        commons-cli
    
    
        com.azure
        azure-core-http-netty
    



    
        
            src/main/resources
            false
        
        
            src/main/resources-filtered
            true
        
    

    
        
            net.alchim31.maven
            scala-maven-plugin
            
                
                    scala-compile
                    
                        compile
                        testCompile
                    
                
            
            
                ${scala.version}
                
                    -Xms64m
                    -Xmx1024m
                
            
        

        
            org.apache.maven.plugins
            maven-jar-plugin
            3.0.2
            
                
                    
                        ${project.artifactId}
                        ${project.version}
                        ${project.groupId}
                        ${project.artifactId}
                        ${project.groupId}
                        ${project.version}
                        ${project.groupId}
                    
                
            
        

        
            org.codehaus.mojo
            flatten-maven-plugin
            1.2.7
            
                true
                resolveCiFriendliesOnly
            
            
                
                    flatten
                    process-resources
                    
                        flatten
                    
                
                
                    flatten.clean
                    clean
                    
                        clean
                    
                
            
        

        
            org.apache.maven.plugins
            maven-assembly-plugin
            3.3.0
            
                
                    make-assembly
                    package
                    
                        single
                    
                
            
            
                ${project.artifactId}-${project.version}-assembly
                false
                
                    jar-with-dependencies
                
                
                    
                        ${project.artifactId}
                        ${project.version}
                        ${project.groupId}
                        ${project.artifactId}
                        ${project.groupId}
                        ${project.version}
                        ${project.groupId}
                    
                
            
        

        
            org.scalatest
            scalatest-maven-plugin
            2.0.0
            
                ${project.build.directory}/surefire-reports
            
            
                
                    test
                    
                        test
                    
                
            
        

        
            org.scalastyle
            scalastyle-maven-plugin
            
                
                    compile
                    
                        check
                    
                
            
        
    

这是母项目:

group
parent
3.1-SNAPSHOT
pom

    2.12.14
    2.12
    3.2.1
    0.0.5
    1.1.0
    42.2.9
    1.4.0
    1.25
    1.3.5
    1.4.200
    1.16.46
    1.14.0
    1.6.6
    0.8.0
    2.7
    1.0
    1.2.3
    1.1.0
    3.1.0
    4.4.0
    2.0.0
    0.12.3
    1.0-m5.1
    2.10.6
    8.2.2.jre8
    circe-yaml_2.12
    0.13.1
    circe-generic_2.12
    0.13.0
    0.3.1
    0.14.0
    3.0.5
    4.5.6
    1.4.0
    1.23.0
    1.0.0
    2.17.1
    1.2
    1.0.15
    1.11.7


    
        
            org.apache.logging.log4j
            log4j-core
            ${log4j-core.version}
        
        
            org.apache.spark
            spark-core_${scala.compact.version}
            ${spark.version}
            provided
        
        
            org.apache.spark
            spark-sql_${scala.compact.version}
            ${spark.version}
            provided
        
        
            org.apache.spark
            spark-streaming_${scala.compact.version}
            ${spark.version}
            provided
        
        
            org.apache.spark
            spark-mllib_${scala.compact.version}
            ${spark.version}
            provided
        
        
            org.apache.kafka
            kafka-clients
            ${kafka.version}
        
        
            org.apache.kafka
            kafka-streams
            ${kafka.version}
            
                
                    com.fasterxml.jackson.core
                    jackson-databind
                
            
        
        
            org.apache.spark
            spark-avro_${scala.compact.version}
            ${spark.version}
        
        
            org.scala-lang
            scala-library
            ${scala.version}
            provided
        
        
            com.microsoft.azure
            adal4j
            ${adal4j.version}
        
        
            com.databricks
            dbutils-api_2.12
            ${databricks.dbutils.api.version}
            
                
                    org.scala-lang
                    scala-library
                
            
            provided
        
        
            joda-time
            joda-time
            ${joda-time.version}
        
        
            org.apache.httpcomponents
            httpclient
            ${httpclient.version}
            provided
        
        
            io.delta
            delta-core_${scala.compact.version}
            ${delta.version}
            provided
        
        
            com.typesafe
            config
            ${typesafe.config.version}
        
        
            org.yaml
            snakeyaml
            ${snakeyaml.version}
        
        
            io.circe
            ${circe-yaml.artifact}
            ${circe-yaml.version}
        
        
            io.circe
            ${circe-generic.artifact}
            ${circe-generic.version}
        
        
            io.jvm.uuid
            scala-uuid_2.12
            ${scala-uuid_2.12.version}
        
        
            io.spray
            spray-json_${scala.compact.version}
            ${spray-json.version}
        
        
            org.mockito
            mockito-scala_${scala.compact.version}
            ${mockito.version}
            test
        
        
            org.typelevel
            cats-core_2.12
            ${cats-core.version}
        
        
            org.scalacheck
            scalacheck_${scala.compact.version}
            ${scalacheck.version}
            test
        
        
            com.holdenkarau
            spark-testing-base_${scala.compact.version}
            2.4.5_${spark-testing-base.version}
        
        
            org.scalatest
            scalatest_${scala.compact.version}
            ${scalatest.version}
            test
        
        
            com.github.mrpowers
            spark-fast-tests_${scala.compact.version}
            ${spark-fast-tests.version}
            test
        
        
            com.github.mrpowers
            spark-daria_${scala.compact.version}
            ${spark-daria.version}
        
        
            com.azure
            azure-identity
            ${azure-identity.version}
            
                
                    com.azure
                    azure-core-http-netty
                
            
        
        
            com.azure
            azure-core
            ${azure-core.version}
            
                
                    com.fasterxml.jackson.dataformat
                    jackson-dataformat-xml
                
                
                    com.fasterxml.jackson.datatype
                    jackson-datatype-jsr310
                
                
                    io.projectreactor.netty
                    reactor-netty
                
            
        
        
            com.azure
            azure-core-http-netty
            ${azure-core-http-netty.version}
        
        
            com.azure
            azure-data-schemaregistry
            ${azure-data-schemaregistry.version}
            
                
                    com.azure
                    azure-core-http-netty
                
            
        
        
            commons-cli
            commons-cli
            ${commons-cli.version}
        
    


    
        
            external.atlassian.jgitflow
            jgitflow-maven-plugin
            ${jgitflow-maven-plugin.version}
            
                true
                true
                true
                true
                true
                
                    master
                    develop
                    feature/
                    release/
                    hotfix/
                    ${project.artifactId}-
                
            
        
        
            net.alchim31.maven
            scala-maven-plugin
            ${scala-maven-plugin.version}
            
                
                    compile
                    
                        compile
                        testCompile
                    
                
            
            
                
                    -Xms1024M
                    -Xmx2048M
                    -XX:+CMSClassUnloadingEnabled
                
            
        
        
            org.scalastyle
            scalastyle-maven-plugin
            ${scalastyle.version}
            
                true
                true
                true
                false
                ${project.basedir}/src/main/scala
                ${project.basedir}/src/test/scala
                ${project.basedir}/scalastyle_config.xml
                ${project.basedir}/scalastyle-output.xml
                UTF-8
            
            
                
                    
                        check
                    
                
            
        
        
            org.apache.maven.plugins
            maven-surefire-plugin
            ${maven-surefire-plugin.version}
            
                true
            
        
    


    
        jitpack.io
        https://jitpack.io
        
            false
        
    
    
        dl.binatry.com
        http://dl.bintray.com/spark-packages/maven
        
            false
        
    

当我在本地运行它时,它工作正常,当试图在集群中运行它时,问题就出现了。

我试图排除一些冲突版本的人工制品,但问题依然存在。

有什么帮助吗?

特别声明:以上内容(图片及文字)均为互联网收集或者用户上传发布,本站仅提供信息存储服务!如有侵权或有涉及法律问题请联系我们。
举报
评论区(0)
按点赞数排序
用户头像