kıvılcım gönder "Hizmet 'Sürücü' noktası üzerinde bağlamak" hata

Question

Ben wordcount kıvılcım java örneği çalıştırmak için aşağıdaki komutu kullanılır: - Ben çalıştırdığınızda, aşağıdaki çıkış

time spark-submit --deploy-mode cluster --master spark://192.168.0.7:6066 --class org.apache.spark.examples.JavaWordCount /home/pi/Desktop/example/new/target/javaword.jar /books_50.txt 

geçerli: -

Running Spark using the REST application submission protocol. 
16/07/18 03:55:41 INFO rest.RestSubmissionClient: Submitting a request to launch an application in spark://192.168.0.7:6066. 
16/07/18 03:55:44 INFO rest.RestSubmissionClient: Submission successfully created as driver-20160718035543-0000. Polling submission state... 
16/07/18 03:55:44 INFO rest.RestSubmissionClient: Submitting a request for the status of submission driver-20160718035543-0000 in spark://192.168.0.7:6066. 
16/07/18 03:55:44 INFO rest.RestSubmissionClient: State of driver driver-20160718035543-0000 is now RUNNING. 
16/07/18 03:55:44 INFO rest.RestSubmissionClient: Driver is running on worker worker-20160718041005-192.168.0.12-42405 at 192.168.0.12:42405. 
16/07/18 03:55:44 INFO rest.RestSubmissionClient: Server responded with CreateSubmissionResponse: 
{ 
    "action" : "CreateSubmissionResponse", 
    "message" : "Driver successfully submitted as driver-20160718035543-0000", 
    "serverSparkVersion" : "1.6.2", 
    "submissionId" : "driver-20160718035543-0000", 
    "success" : true 
} 

Ben onun günlüğü için belirli işçi (192.168.0.12) kontrol edilmiş ve şöyle der: -

Launch Command: "/usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/bin/java" "-cp" "/opt/spark/conf/:/opt/spark/lib/spark-assembly-1.6.2-hadoop2.6.0.jar:/opt/spark/lib/datanucleus-api-jdo-3.2.6.jar:/opt/spark/lib/datanucleus-core-3.2.10.jar:/opt/spark/lib/datanucleus-rdbms-3.2.9.jar" "-Xms1024M" "-Xmx1024M" "-Dspark.driver.supervise=false" "-Dspark.app.name=org.apache.spark.examples.JavaWordCount" "-Dspark.submit.deployMode=cluster" "-Dspark.jars=file:/home/pi/Desktop/example/new/target/javaword.jar" "-Dspark.master=spark://192.168.0.7:7077" "-Dspark.executor.memory=10M" "org.apache.spark.deploy.worker.DriverWrapper" "spark://Worker@192.168.0.12:42405" "/opt/spark/work/driver-20160718035543-0000/javaword.jar" "org.apache.spark.examples.JavaWordCount" "/books_50.txt" 
======================================== 

log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory). 
log4j:WARN Please initialize the log4j system properly. 
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. 
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 
16/07/18 04:10:58 INFO SecurityManager: Changing view acls to: pi 
16/07/18 04:10:58 INFO SecurityManager: Changing modify acls to: pi 
16/07/18 04:10:58 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(pi); users with modify permissions: Set(pi) 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
Exception in thread "main" java.net.BindException: Cannot assign requested address: Service 'Driver' failed after 16 retries! Consider explicitly setting the appropriate port for the service 'Driver' (for example spark.ui.port for SparkUI) to an available port or increasing spark.port.maxRetries. 
    at sun.nio.ch.Net.bind0(Native Method) 
    at sun.nio.ch.Net.bind(Net.java:433) 
    at sun.nio.ch.Net.bind(Net.java:425) 
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) 
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) 
    at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:125) 
    at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:485) 
    at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1089) 
    at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:430) 
    at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:415) 
    at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:903) 
    at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:198) 
    at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:348) 
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357) 
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) 
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) 
    at java.lang.Thread.run(Thread.java:745) 

My spark-env.sh (Usta) içerir dosya: - (işçi için)

export SPARK_MASTER_WEBUI_PORT="8080" 
export SPARK_MASTER_IP="192.168.0.7" 
export SPARK_EXECUTOR_MEMORY="10M" 

Benim spark-env.sh dosyası içerir: -

export SPARK_WORKER_WEBUI_PORT="8080" 
export SPARK_MASTER_IP="192.168.0.7" 
export SPARK_EXECUTOR_MEMORY="10M" 

Lütfen yardım ... !!

Answer 1

Bu benim aynı adımları çalıştı, ancak iş çalıştırmak mümkün olan muhtemelen Spark 1.2.1 standalone cluster mode spark-submit is not working

kopyasıdır. Mümkünse, tam spark-env.sh ve kıvılcım varsayılanlarını gönderin.

Answer 2

Anasistem adını /etc/hosts dosyanızda girmeniz gerekir. gibi şey:

127.0.0.1 localhost "hostname" 

Answer 3

Ben kabuk çalıştırmak çalışırken aynı sorunu vardı ve SPARK_LOCAL_IP ortam değişkenini ayarlayarak bu çalışma almak başardı. , Daha kalıcı bir çözüm için

SPARK_LOCAL_IP=127.0.0.1 ./bin/spark-shell

senin Kıvılcım kökünün conf dizininde bir spark-env.sh dosyası oluşturmak: Sen kabuk çalıştırırken komut satırından bu atayabilirsiniz. aşağıdaki satırı ekleyin:

SPARK_LOCAL_IP=127.0.0.1

chmod +x ./conf/spark-env.sh kullanarak komut yürütme izinleri verin ve bu varsayılan olarak bu ortam değişkeni ayarlar.

Answer 4

Bağımlılıkları yönetmek için Maven/SBT kullanıyorum ve Spark çekirdeği bir jar dosyasında yer alıyor.

Sen (burada Scala) "spark.driver.bindAddress" ayarlayarak zamanında SPARK_LOCAL_IP geçersiz kılabilirsiniz:

val config = new SparkConf() 
config.setMaster("local[*]") 
config.setAppName("Test App") 
config.set("spark.driver.bindAddress", "127.0.0.1") 
val sc = new SparkContext(config)