The spark-submit command is a utility to run or submit a Spark or PySpark application program (or job) to the cluster by specifying options and configurations, the application you are submitting can be written in Scala, Java, or Python (PySpark). spark-submit command supports the following.
client or cluster deployment modes.Related:
In this article, I will explain different spark-submit command options and configurations along with how to use a uber jar or zip file for Scala and Java, using Python .py file, and finally how to submit the application on Yarn. Mesos, Kubernetes, and standalone cluster managers.
Spark binary comes with spark-submit.sh script file for Linux, Mac, and spark-submit.cmd command file for windows, these scripts are available at $SPARK_HOME/bin directory.
If you are using Cloudera distribution, you may also find spark2-submit.sh which is used to run Spark 2.x applications. By adding this Cloudera supports both Spark 1.x and Spark 2.x applications to run in parallel.
spark-submit command internally uses org.apache.spark.deploy.SparkSubmit class with the options and command line arguments you specify.
Below is a spark-submit command with the most-used command options.
You can also submit the application like below without using the script.
Below I have explained some of the common options, configurations, and specific options to use with Scala and Python. You can also get all options available by running the below command.
Using --deploy-mode, you specify where to run the Spark application driver program. Spark support cluster and client deployment modes.
| Value | Description |
|---|---|
| cluster | In cluster mode, the driver runs on one of the worker nodes, and this node shows as a driver on the Spark Web UI of your application. cluster mode is used to run production jobs. |
| client | In client mode, the driver runs locally where you are submitting your application from. client mode is majorly used for interactive and debugging purposes. Note that in client mode only the driver runs locally and all other executors run on different nodes on the cluster. |
Using --master option, you specify what cluster manager to use to run your application. Spark currently supports Yarn, Mesos, Kubernetes, Stand-alone, and local. The uses of these are explained below.
| Cluster Manager | Value | Description |
|---|---|---|
| Yarn | yarn | Use yarn if your cluster resources are managed by Hadoop Yarn. |
| Mesos | mesos://HOST:PORT | use mesos://HOST:PORT for Mesos cluster manager, replace the host and port of Mesos cluster manager. |
| Standalone | spark://HOST:PORT | Use spark://HOST:PORT for Standalone cluster, replace the host and port of stand-alone cluster. |
| Kubernetes | k8s://HOST:PORT k8s://https://HOST:PORT | Use k8s://HOST:PORT for Kubernetes, replace the host and port of Kubernetes. This by default connects with https, but if you wanted to use unsecured use k8s://https://HOST:PORT |
| local | local local[k] local[K,F] | Use local to run locally with a one worker thread. Use local[k] and specify k with the number of cores you have locally, this runs application with k worker threads. use local[k,F] and specify F with number of attempts it should run when failed. |
Example: Below submits applications to yarn managed cluster.
Value 80 on the above example is a command-line argument for the spark program SparkPi. The above example calculates a PI value of 80.
While submitting an application, you can also specify how much memory and cores you wanted to give for driver and executors.
| Option | Description |
|---|---|
| –driver-memory | Memory to be used by the Spark driver. |
| –driver-cores | CPU cores to be used by the Spark driver |
| –num-executors | The total number of executors to use. |
| –executor-memory | Amount of memory to use for the executor process. |
| –executor-cores | Number of CPU cores to use for the executor process. |
| –total-executor-cores | The total number of executor cores to use. |
Example:
| Options | Description |
|---|---|
| –files | Use comma-separated files you wanted to use. Usually, these can be files from your resource folder. Using this option, Spark submits all these files to cluster. |
| –verbose | Displays the verbose information. For example, writes all configurations spark application uses to the log file. |
Note: Files specified with --files are uploaded to the cluster.
Example: Below example submits the application to yarn cluster manager by using cluster deployment mode and with 8g driver memory, 16g, and 2 cores for each executor.
Spark submit supports several configurations using --config, these configurations are used to specify Application configurations, shuffle parameters, runtime configurations.
Most of these configurations are the same for Spark applications written in Java, Scala, and Python(PySpark)
| Configuration key | Configuration Description |
|---|---|
| spark.sql.shuffle.partitions | Number of partitions to create for wider shuffle transformations (joins and aggregations). |
| spark.executor.memoryOverhead | The amount of additional memory to be allocated per executor process in cluster mode, it is typically memory for JVM overheads. (Not supported for PySpark) |
| spark.serializer | org.apache.spark.serializer.<br>JavaSerializer (default)org.apache.spark.serializer.KryoSerializer |
| spark.sql.files.maxPartitionBytes | The maximum number of bytes to be used for every partition when reading files. Default 128MB. |
| spark.dynamicAllocation.enabled | Specifies whether to dynamically increase or decrease the number of executors based on the workload. Default true. |
| spark.dynamicAllocation .minExecutors | A minimum number of executors to use when dynamic allocation is enabled. |
| spark.dynamicAllocation .maxExecutors | A maximum number of executors to use when dynamic allocation is enabled. |
| spark.executor.extraJavaOptions | Specify JVM options (see example below) |
Besides these, Spark also supports many more configurations.
Example :
Alternatively, you can also set these globally @ $SPARK_HOME/conf/spark-defaults.conf to apply for every Spark application. And you can also set using SparkConf programmatically.
First preference goes to SparkConf, then spark-submit –config and then configs mentioned in spark-defaults.conf
Regardless of which language you use, most of the options are the same however, there are few options that are specific to a language, for example, to run a Spark application written in Scala or Java, you need to use the additional following options.
| Option | Description |
|---|---|
| –jars | If you have all dependency jar’s in a folder, you can pass all these jars using this spark submit –jars option. All your jar files should be comma-separated. for example –jars jar1.jar,jar2.jar, jar3.jar. |
| –packages | All transitive dependencies will be handled when using this command. |
| –class | Scala or Java class you wanted to run. This should be a fully qualified name with the package for example org.apache.spark.examples.SparkPi. |
Note: Files specified with --jars and --packages are uploaded to the cluster.
Example :
When you wanted to spark-submit a PySpark application, you need to specify the .py file you wanted to run and specify the .egg file or .zip file for dependency libraries.
Below are some of the options & configurations specific to PySpark application. besides these you can also use most of the options & configs that are covered above.
| PySpark Specific Configurations | Description |
|---|---|
| –py-files | Use --py-files to add .py, .zip or .egg files. |
| –config spark.executor.pyspark.memory | The amount of memory to be used by PySpark for each executor. |
| –config spark.pyspark.driver.python | Python binary executable to use for PySpark in driver. |
| –config spark.pyspark.python | Python binary executable to use for PySpark in both driver and executors. |
Note: Files specified with --py-files are uploaded to the cluster before it runs the application. You also upload these files ahead and refer them in your PySpark application.
Example 1 :
Example 2 : Below example uses other python files as dependencies.
Here, we are submitting spark application on a Mesos-managed cluster using deployment mode with 5G memory and 8 cores for each executor.
The below example runs Spark application on a Kubernetes managed cluster using cluster deployment mode with 5G memory and 8 cores for each executor.
The below example runs Spark application on a Standalone cluster using cluster deployment mode with 5G memory and 8 cores for each executor.
Happy Learning !!
或是邮件反馈可也:
askdama[AT]googlegroups.com
订阅 substack 体验古早写作:
关注公众号, 持续获得相关各种嗯哼:
