atom feed15 messages in org.apache.spark.userRe: Purpose of spark-submit?
FromSent OnAttachments
Robert JamesJul 8, 2014 6:22 pm 
Patrick WendellJul 9, 2014 12:45 am 
Koert KuipersJul 9, 2014 5:20 am 
Surendranauth HiramanJul 9, 2014 5:30 am 
Robert JamesJul 9, 2014 6:47 am 
Jerry LamJul 9, 2014 7:14 am 
AndreiJul 9, 2014 8:34 am 
Sandy RyzaJul 9, 2014 9:05 am 
Koert KuipersJul 9, 2014 9:14 am 
Jerry LamJul 9, 2014 9:25 am 
Sandy RyzaJul 9, 2014 9:28 am 
Ron GonzalezJul 9, 2014 9:37 am 
Ron GonzalezJul 9, 2014 9:40 am 
Andrew OrJul 9, 2014 6:39 pm 
Koert KuipersJul 10, 2014 6:10 am 
Subject:Re: Purpose of spark-submit?
From:Koert Kuipers (
Date:Jul 9, 2014 5:20:56 am

not sure I understand why unifying how you submit app for different platforms and dynamic configuration cannot be part of SparkConf and SparkContext?

for classpath a simple script similar to "hadoop classpath" that shows what needs to be added should be sufficient.

on spark standalone I can launch a program just fine with just SparkConf and SparkContext. not on yarn, so the spark-launch script must be doing a few things extra there I am missing... which makes things more difficult because I am not sure its realistic to expect every application that needs to run something on spark to be launched using spark-submit. On Jul 9, 2014 3:45 AM, "Patrick Wendell" <> wrote:

It fulfills a few different functions. The main one is giving users a way to inject Spark as a runtime dependency separately from their program and make sure they get exactly the right version of Spark. So a user can bundle an application and then use spark-submit to send it to different types of clusters (or using different versions of Spark).

It also unifies the way you bundle and submit an app for Yarn, Mesos, etc... this was something that became very fragmented over time before this was added.

Another feature is allowing users to set configuration values dynamically rather than compile them inside of their program. That's the one you mention here. You can choose to use this feature or not. If you know your configs are not going to change, then you don't need to set them with spark-submit.

On Wed, Jul 9, 2014 at 10:22 AM, Robert James <> wrote:

What is the purpose of spark-submit? Does it do anything outside of the standard val conf = new SparkConf ... val sc = new SparkContext ... ?