Configuring job properties for Python shell jobs in Amazon Glue

To create a Python .egg or .whl file

1. Create an Amazon Redshift cluster in a virtual private cloud (VPC), and add some data to a table.

2. Create an Amazon Glue connection for the VPC-SecurityGroup-Subnet combination that you used to create the cluster. Test that the connection is successful.

3. Create a directory named redshift_example, and create a file named setup.py. Paste the following code into setup.py.

   from setuptools import setup
   
   setup(
       name="redshift_module",
       version="0.1",
       packages=['redshift_module']
   ) 
   

4. In the redshift_example directory, create a redshift_module directory. In the redshift_module directory, create the files __init__.py and pygresql_redshift_common.py.

5. Leave the __init__.py file empty. In pygresql_redshift_common.py, paste the following code. Replace port, db_name, user, and password_for_user with details specific to your Amazon Redshift cluster. Replace table_name with the name of the table in Amazon Redshift.

   import pg
   
   
   def get_connection(host):
       rs_conn_string = "host=%s port=%s dbname=%s user=%s password=%s" % (
           host, port, db_name, user, password_for_user)
   
       rs_conn = pg.connect(dbname=rs_conn_string)
       rs_conn.query("set statement_timeout = 1200000")
       return rs_conn
   
   
   def query(con):
       statement = "Select * from table_name;"
       res = con.query(statement)
       return res
       
   

6. If you're not already there, change to the redshift_example directory.

7. Do one of the following:

   • To create an .egg file, run the following command.

     python setup.py bdist_egg

   • To create a .whl file, run the following command.

     python setup.py bdist_wheel

8. Install the dependencies that are required for the preceding command.

9. The command creates a file in the dist directory:

   • If you created an egg file, it's named redshift_module-0.1-py2.7.egg.

   • If you created a wheel file, it's named redshift_module-0.1-py2.7-none-any.whl.

   Upload this file to Amazon S3.

   In this example, the uploaded file path is either s3://DOC-EXAMPLE-BUCKET/EGG-FILE or s3://DOC-EXAMPLE-BUCKET/WHEEL-FILE.

10. Create a Python file to be used as a script for the Amazon Glue job, and add the following code to the file.

    from redshift_module import pygresql_redshift_common as rs_common
    
    con1 = rs_common.get_connection(redshift_endpoint)
    res = rs_common.query(con1)
    
    print "Rows in the table cities are: "
    
    print res
          
    

11. Upload the preceding file to Amazon S3. In this example, the uploaded file path is s3://DOC-EXAMPLE-BUCKET/scriptname.py.

12. Create a Python shell job using this script. On the Amazon Glue console, on the Job properties page, specify the path to the .egg/.whl file in the Python library path box. If you have multiple .egg/.whl files and Python files, provide a comma-separated list in this box.

    When modifying or renaming .egg files, the file names must use the default names generated by the "python setup.py bdist_egg" command or must adhere to the Python module naming conventions. For more information, see the Style Guide for Python Code.

    Using the Amazon CLI, create a job with a command, as in the following example.

    aws glue create-job --name python-redshift-test-cli --role Role --command '{"Name" :  "pythonshell", "ScriptLocation" : "s3://DOC-EXAMPLE-BUCKET/scriptname.py"}' 
         --connections Connections="connection-name" --default-arguments '{"--extra-py-files" : ["s3://DOC-EXAMPLE-BUCKET/EGG-FILE", "s3://DOC-EXAMPLE-BUCKET/WHEEL-FILE"]}'
    

    When the job runs, the script prints the rows created in the table_name table in the Amazon Redshift cluster.