Configuring Git integration in Amazon Glue

Integrate with Git version control systems Amazon CodeCommit and GitHub

Edit Amazon Glue jobs in Amazon Glue Studio whether you use visual jobs or script jobs and sync them to a repository

Parameterize sources and targets in jobs

Pull jobs from a repository and edit them in Amazon Glue Studio

Test jobs by pulling from branches and/or pushing to branches utilizing multi-branch workflows in Amazon Glue Studio

Download files from a repository and upload jobs into Amazon Glue Studio for cross-account job creation

Use your automation tool of choice (for example, Jenkins, Amazon CodeDeploy, etc.)

AWSGlueServiceRole

AWSGlueConsoleFullAccess

glue:UpdateJobFromSourceControl — to be able to update Amazon Glue with a job present in a version control system

glue:UpdateSourceControlFromJob — to be able to update the version control system with a job stored in Amazon Glue

s3:GetObject — to be able to retrieve the script for the job while pushing to version control system

s3:PutObject — to be able to update the script when pulling a job from a source control system

you need a folder and a JSON file of the job definition that matches the job name. For example, see the job definition below. The branch in your repository should contain a path my-visual-job/my-visual-job.json where both the folder and the JSON file match the job name

{
  "name" : "my-visual-job",
  "description" : "",
  "role" : "arn:aws:iam::aws_account_id:role/Rolename",
  "command" : {
    "name" : "glueetl",
    "scriptLocation" : "s3://foldername/scripts/my-visual-job.py",
    "pythonVersion" : "3"
  },
  "codeGenConfigurationNodes" : "{\"node-nodeID\":{\"S3CsvSource\":{\"AdditionalOptions\":{\"EnableSamplePath\":false,\"SamplePath\":\"s3://notebook-test-input/netflix_titles.csv\"},\"Escaper\":\"\",\"Exclusions\":[],\"Name\":\"Amazon S3\",\"OptimizePerformance\":false,\"OutputSchemas\":[{\"Columns\":[{\"Name\":\"show_id\",\"Type\":\"string\"},{\"Name\":\"type\",\"Type\":\"string\"},{\"Name\":\"title\",\"Type\":\"choice\"},{\"Name\":\"director\",\"Type\":\"string\"},{\"Name\":\"cast\",\"Type\":\"string\"},{\"Name\":\"country\",\"Type\":\"string\"},{\"Name\":\"date_added\",\"Type\":\"string\"},{\"Name\":\"release_year\",\"Type\":\"bigint\"},{\"Name\":\"rating\",\"Type\":\"string\"},{\"Name\":\"duration\",\"Type\":\"string\"},{\"Name\":\"listed_in\",\"Type\":\"string\"},{\"Name\":\"description\",\"Type\":\"string\"}]}],\"Paths\":[\"s3://dalamgir-notebook-test-input/netflix_titles.csv\"],\"QuoteChar\":\"quote\",\"Recurse\":true,\"Separator\":\"comma\",\"WithHeader\":true}}}"
}            
          

you need a folder, a JSON file of the job definition, and the script

the folder and JSON file should match the job name. The script name needs to match the scriptLocation in the job definition along with the file extension. For example, in the job definition below, the branch in your repository should contain a path my-script-job/my-script-job.json and my-script-job/my-script-job.py. The script name should match the name in the scriptLocation including the extension of the script

{
  "name" : "my-script-job",
  "description" : "",
  "role" : "arn:aws:iam::aws_account_id:role/Rolename",
  "command" : {
    "name" : "glueetl",
    "scriptLocation" : "s3://foldername/scripts/my-script-job.py",
    "pythonVersion" : "3"
  }
}