Enabling the Apache Spark web UI for Amazon Glue jobs - Amazon Glue
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China.

Enabling the Apache Spark web UI for Amazon Glue jobs

You can use the Apache Spark web UI to monitor and debug Amazon Glue ETL jobs running on the Amazon Glue job system. You can configure the Spark UI using the Amazon Glue console or the Amazon Command Line Interface (Amazon CLI).

Configuring the Spark UI (console)

Follow these steps to configure the Spark UI using the Amazon Web Services Management Console.

To create a job with the Spark UI enabled

  1. Sign in to the Amazon Web Services Management Console and open the Amazon Glue console at https://console.amazonaws.cn/glue/.

  2. In the navigation pane, choose Jobs.

  3. Choose Add job.

  4. In Configure the job properties, open the Monitoring options.

  5. In the Spark UI tab, choose Enable.

  6. Specify an Amazon S3 path for storing the Spark event logs for the job.

To edit an existing job to enable the Spark UI

  1. Open the Amazon Glue console at https://console.amazonaws.cn/glue/.

  2. In the navigation pane, choose Jobs.

  3. Choose an existing job in the job list.

  4. Choose Action, and then choose Edit job.

  5. Open the Monitoring options.

  6. In the Spark UI tab, choose Enable.

  7. Enter an Amazon S3 path for storing the Spark event logs for the job.

To set up user preferences for new jobs to enable the Spark UI

  1. Open the Amazon Glue console at https://console.amazonaws.cn/glue/.

  2. In the upper-right corner, choose User preferences.

  3. Open the Monitoring options.

  4. In the Spark UI tab, choose Enable.

  5. Specify an Amazon S3 path for storing the Spark event logs for the job.

To set up the job run options to enable the Spark UI

  1. Open the Amazon Glue console at https://console.amazonaws.cn/glue/.

  2. In the navigation pane, choose Jobs.

  3. Choose an existing job in the job lists.

  4. Choose Scripts and Edit Job. You navigate to the code pane.

  5. Choose Run job.

  6. Open the Monitoring options.

  7. In the Spark UI tab, choose Enable.

  8. Specify an Amazon S3 path for storing the Spark event logs for the job.

Configuring the Spark UI (Amazon CLI)

To enable the Spark UI feature using the Amazon CLI, pass in the following job parameters to Amazon Glue jobs. For more information, see Special Parameters Used by Amazon Glue.

'--enable-spark-ui': 'true', '--spark-event-logs-path': 's3://s3-event-log-path'

Every 30 seconds, Amazon Glue flushes the Spark event logs to the Amazon S3 path that you specify.