Example interactions
Amazon Q data integration in Amazon Glue allows you enter your question in the Amazon Q panel. You can enter a question regarding data integration functionality provided by Amazon Glue. A detailed answer, together with reference documents, will be returned.
Another use case is generating Amazon Glue ETL job scripts. You can ask a question regarding how to perform a data extract, transform, load job. A generated PySpark script will be returned.
Amazon Q chat interactions
On the Amazon Glue console, start authoring a new job, and ask Amazon Q: "Please provide a Glue script that reads from Snowflake, renames the fields, and writes to Redshift."
You will notice that the code is generated. With this response, you can learn and understand how you can author Amazon Glue code for your purpose. You can copy/paste the generated code to the script editor and configure placeholders. After you configure an Amazon Identity and Access Management (IAM) role and Amazon Glue connections on the job, save and run the job. When the job is complete, you can start querying the table exported from Snowflake in Amazon Redshift.
The following prompt reads data from two different sources, filters and projects them individually, joins on a common key, and writes the output to a third target. Ask Amazon Q: “I want to read data from S3 in Parquet format, and select some fields. I also want to read data from DynamoDB, select some fields, and filter some rows. I want to union these two datasets and write the results to OpenSearch.
The code is generated. When the job is complete, your index is available in OpenSearch and can be used by your downstream workloads.
Amazon Glue Studio notebook interactions
Add a new cell and enter your comment to describe what you want to achieve. After you press Tab and Enter, the recommended code is shown.
First intent is to extract the data: "Give me code that reads a Glue Data Catalog table", followed by "Give me code to apply a filter transform with star_rating>3" and "Give me code that writes the frame into S3 as Parquet".
Similar to the Amazon Q chat experience, the code is recommended. If you press Tab, then the recommended code is chosen.
You can run each cell by filling in the appropriate options for your sources in the generated code. At any point in the runs, you can also preview a sample of your dataset by using the show() method.
Complex prompts
You can generate a full script with a single complex prompt. "I have JSON data in S3 and data in Oracle that needs combining. Please provide a Glue script that reads from both sources, does a join, and then writes results to Redshift."
You may notice that, on the notebook, Amazon Q data integration in Amazon Glue generated the same code snippet that was generated in the Amazon Q chat.
You can run the notebook as a job, either by choosing Run or programmatically.