Configuring Amazon Glue interactive sessions for Jupyter and Amazon Glue Studio notebooks

Introduction to Jupyter Magics

Jupyter Magics are commands that can be run at the beginning of a cell or as a whole cell body. Magics start with % for line-magics and %% for cell-magics. Line-magics such as %region and %connections can be run with multiple magics in a cell, or with code included in the cell body like the following example.

%region us-east-2
%connections my_rds_connection
dy_f = glue_context.create_dynamic_frame.from_catalog(database='rds_tables', table_name='sales_table')         
        

Cell magics must use the entire cell and can have the command span multiple lines. An example of %%sql is below.

%%sql
select * from rds_tables.sales_table
        

Magics supported by Amazon Glue interactive sessions for Jupyter

The following are magics that you can use with Amazon Glue interactive sessions for Jupyter notebooks.

Sessions magics

Magics for selecting job types

Amazon Glue for Spark config magics

The %%configure magic is a json-formatted dictionary consisting of all configuration parameters for a session. Each parameter can be specified here or through individual magics.

%%tags
{"billing":"Data-Platform", "team":"analytics"}
                      

%status

Session ID: <sessionId>
 Status: READY
 Role: <example-role>
 CreatedOn: 2023-05-26 11:12:17.056000-07:00
 GlueVersion: 3.0
 Job Type: glueetl
 Tags: {'owner':'example-owner', 'team':'analytics', 'billing':'Data-Platform'}
 Worker Type: G.4X
 Number of Workers: 5
 Region: us-west-2
 Applying the following default arguments:
 --glue_kernel_version 0.38.0
 --enable-glue-datacatalog true
 Arguments Passed: ['--glue_kernel_version: 0.38.0', '--enable-glue-datacatalog: true']                
                

%%assume_role
{
  'arn:aws:iam::XXXXXXXXXXXX:role/AWSGlueServiceRole'
}
                

 %%assume_role
{{
    "aws_access_key_id" = "XXXXXXXXXXXX",
    "aws_secret_access_key" = "XXXXXXXXXXXX",
    "aws_session_token" = "XXXXXXXXXXXX"
}}

%%configure cell magic arguments

The %%configure magic is a json-formatted dictionary consisting of all configuration parameters for a session. Each parameter can be specified here or through individual magics. See below for examples for arguments supported by the %%configure cell magic. Use the -- prefix for run arguments specified for the job. Example:

%%configure
{
   "--user-jars-first": "true",
   "--enable-glue-datacatalog": "false"
}     
        

For more information on job parameters, see Job parameters.

Session Configuration

%%configure
{
  "max_retries": "0"
}                      
                          

%%configure
{
  "max_concurrent_runs": "3"
}

Session parameters

%%configure
{
  "--enable-spark-ui": "true"
}

%%configure
{
  "--spark-event-logs-path": "s3://path/to/event/logs/"
}                           
                          

%%configure 
{
  "script_location": "s3://new-folder-here"
}                            
                          

%%configure
{
    "--security_configuration": {
"encryption_type": "kms",
"kms_key_id": "YOUR_KMS_KEY_ARN"
}
}
                  

%%configure 
{
  "--job-language": "scala"
}                            
                  

%%configure 
{
  "--class": "className"
}                            
                  

%%configure 
{
  "--user-jars-first": "true"
}                            
                  

%%configure 
{
  "--use-postgres-driver": "true"
}                            
                  

%%configure 
{
  "--extra-files": "s3://path/to/additional/files/"
}                            
                  

%%configure
{
  "--job-bookmark-option": "job-bookmark-enable"
}                            
                  

%%configure 
{
  "--TempDir": "s3://path/to/temp/dir"
}                            
                  

%%configure 
{
  "--enable-s3-parquet-optimized-committer": "false"
}                            
                  

%%configure 
{
  "--enable-rename-algorithm-v2": "true"
}                            
                  

%%configure 
{
  "--enable-glue-datacatalog": "true"
}                            
                  

%%configure 
{
  "--enable-metrics": "true"
}                            
                  

%%configure 
{
  "--enable-continuous-cloudwatch-log": "true"
}                            
                  

%%configure 
{
  "--enable-continuous-log-filter": "true"
}                            
                  

%%configure 
{
  "--continuous-log-stream-prefix": "prefix"
}                            
                  

%%configure 
{
  "--continuous-log-conversionPattern": "pattern"
}                      
                  

%%configure
{
    "--conf": "spark.hadoop.hive.metastore.glue.catalogid=123456789012 --conf hive.metastore.client.factory.class=com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory --conf hive.metastore.schema.verification=false"
}       
        

%%configure 
{
  "timeout": "30"
}

%%configure 
{
  "––enable-auto-scaling": "true"
}

Spark jobs (ETL & streaming) magics

%connections my_rds_connection
                    dy_f = glue_context.create_dynamic_frame.from_catalog(database='rds_tables', table_name='sales_table')

Magics for Ray jobs

Action magics

import matplotlib.pyplot as plt

# Set X-axis and Y-axis values
x = [5, 2, 8, 4, 9]
y = [10, 4, 8, 5, 2]
  
# Create a bar chart 
plt.bar(x, y)
  
# Show the plot
%matplot plt      
                

import plotly.express as px
                  
#Create a graphical figure
fig = px.line(x=["a","b","c"], y=[1,3,2], title="sample figure")

#Show the figure
%plotly fig

Naming sessions

Amazon Glue interactive sessions are Amazon resources and require a name. Names should be unique for each session and may be restricted by your IAM administrators. For more information, see Interactive sessions with IAM. The Jupyter kernel automatically generates unique session names for you. However sessions can be named manually in two ways:

A session name is generated as follows:

Prefixing session names allows you to recognize your session when listing it in the Amazon CLI or console.

Specifying an IAM role for interactive sessions

You must specify an Amazon Identity and Access Management (IAM) role to use with Amazon Glue ETL code that you run with interactive sessions.

The role requires the same IAM permissions as those required to run Amazon Glue jobs. See Create an IAM role for Amazon Glue for more information on creating a role for Amazon Glue jobs and interactive sessions.

IAM roles can be specified in two ways:

Configuring sessions with named profiles

Amazon Glue interactive sessions uses the same credentials as the Amazon Command Line Interface or boto3, and interactive sessions honors and works with named profiles like the Amazon CLI found in ~/.aws/config (Linux and MacOS) or %USERPROFILE%\.aws\config (Windows). For more information, see Using named profiles .

Interactive sessions takes advantage of named profiles by allowing the Amazon Glue Service Role and Session ID Prefix to be specified in a profile. To configure a profile role, add a line for the iam_role key and/or session_id_prefix to your named profile as shown below. The session_id_prefix does not require quotes. For example, if you want to add a session_id_prefix, enter the value of the session_id_prefix=myprefix.

[default]
region=us-east-1
aws_access_key_id=AKIAIOSFODNN7EXAMPLE 
aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
glue_iam_role=arn:aws:iam::<AccountID>:role/<GlueServiceRole> 
session_id_prefix=<prefix_for_session_names>

[user1] 
region=eu-west-1
aws_access_key_id=AKIAI44QH8DHBEXAMPLE 
aws_secret_access_key=je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY
glue_iam_role=arn:aws:iam::<AccountID>:role/<GlueServiceRoleUser1> 
session_id_prefix=<prefix_for_session_names_for_user1>          
        

If you have a custom method of generating credentials, you can also configure your profile to use the credential_process parameter in your ~/.aws/config file. For example:

[profile developer]
region=us-east-1
credential_process = "/Users/Dave/generate_my_credentials.sh" --username helen         
        

You can find more details about sourcing credentials through the credential_process parameter here: Sourcing credentials with an external process.

If a region or iam_role are not set in the profile that you are using, you must specify them using the %region and %iam_role magics in the first cell that you run.