Editing or uploading a job script

If creating a new job, on the Jobs page, choose the Spark script editor option to create a Spark job or choose the Python Shell script editor to create a Python shell job. You can either write a new script, or upload an existing script. If you choose Spark script editor, you can write or upload either a Scala or Python script. If you choose Python Shell script editor, you can only write or upload a Python script.

After choosing the option to create a new job, in the Options section that appears, you can choose to either start with a starter script (Create a new script with boilerplate code), or you can upload a local file to use as the job script.

If you chose Spark script editor, you can upload either Python or Scala script files. Scala scripts must have the file extension .scala. Python scripts must be recognized as files of type Python. If you chose Python Shell script editor, you can upload only Python script files.

When you are finished making your choices, choose Create to create the job and open the visual editor.

Go to the visual job editor for the new or saved job, and then choose the Script tab.

If you didn't create a new job using one of the script editor options, and you have never edited the script for an existing job, the Script tab displays the heading Script (Locked). This means the script editor is in read-only mode. Choose Edit script to unlock the script for editing.

To make the script editable, Amazon Glue Studio converts your job from a visual job to a script-only job. If you unlock the script for editing, you can't use the visual editor anymore for this job after you save it.

In the confirmation window, choose Confirm to continue or Cancel to keep the job available for visual editing.

If you choose Confirm, the Visual tab no longer appears in the editor. You can use Amazon Glue Studio to modify the script using the script editor, modify the job details or schedule, or view job runs.

Edit the script as needed.

When you are done editing the script, choose Save to save the job and permanently convert the job from visual to script-only.

(Optional) You can download the script from the Amazon Glue Studio console by choosing the Download button on the Script tab. When you choose this button, a new browser window opens, displaying the script from its location in Amazon S3. The Script filename and Script path parameters in the Job details tab of the job determine the name and location of the script file in Amazon S3.

When you save the job, Amazon Glue save the job script at the location specified by these fields. If you modify the script file at this location within Amazon S3, Amazon Glue Studio will load the modified script the next time you edit the job.

Create a new job by choosing the Spark script editor option.

Under Options, choose Create a new script with boilerplate code.

Choose the Job details tab and set Language to Scala (instead of Python 3).

Choose the Script tab.

Remove the Python boilerplate text. You can replace it with the following Scala boilerplate text.

import com.amazonaws.services.glue.{DynamicRecord, GlueContext}
import org.apache.spark.SparkContext
import com.amazonaws.services.glue.util.JsonOptions
import com.amazonaws.services.glue.util.GlueArgParser
import com.amazonaws.services.glue.util.Job

object MyScript {
  def main(args: Array[String]): Unit = {
    val sc: SparkContext = new SparkContext()
    val glueContext: GlueContext = new GlueContext(sc)

    }
}

Write your Scala job script in the editor. Add additional import statements as needed.