Migrating from CUR to Data Exports CUR 2.0 - Amazon Data Exports
Method one: Create an export with an SQL query using the CUR schemaMethod two: Create an export of CUR 2.0 with its new schema
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Migrating from CUR to Data Exports CUR 2.0

Amazon Data Exports allows you to create exports of Cost and Usage Report 2.0 (CUR 2.0). The CUR 2.0 table provides the same information as Cost and Usage Reports (CUR) along with some improvements. Data Exports enables you to create a CUR 2.0 export that is backwards compatible with the data pipelines you’ve been using to process CUR.

CUR 2.0 provides the following improvements over CUR:

  • Consistent schema: CUR 2.0 contains a fixed set of columns, whereas the columns included for CUR can vary monthly depending on your usage of Amazon services, cost categories, and resource tags.

  • Nested data: CUR 2.0 reduces data sparsity by collapsing certain columns from CUR into individual columns with key-value pairs of the collapsed columns. Optionally, you can query the nested keys in Data Exports as separate columns to match the original CUR schema and data.

  • Additional columns: CUR 2.0 contains two additional columns: bill_payer_account_name and line_item_usage_account_name.

For more detailed information about the schema of CUR 2.0, see the Data Exports table dictionary.

You can migrate to CUR 2.0 in Data Exports in two ways:

  • Method one: Create an export with an SQL query using the CUR schema

  • Method two: Create an export of CUR 2.0 with its new schema

Method one: Create an export with an SQL query using the CUR schema

You can create an export with an SQL query. The export schema matches what you receive today in CUR. You do this using the Amazon API or SDK.

  1. Determine (a) the list of columns and (b) the CUR content settings (Include resource IDs, Split cost allocation data, and Time granularity) needed in order to match your CUR today.

    1. You can determine the list of columns either by viewing the schema of one of your CUR files or going to the manifest file and extracting the list of columns from there.

    2. You can determine the CUR content settings by going to Data Exports in the console and choosing your CUR export to view its details.

  2. Write an SQL query that selects the columns you identified from the CUR 2.0 table named COST_AND_USAGE_REPORT.

    1. All column names in the CUR 2.0 table are in snake case (for example, line_item_usage_amount). For your SQL statement, you might need to convert the previous column names to snake case.

    2. For your SQL statement, you need to convert all resource_tag and cost_category columns, and certain product and discount columns, to have the dot operator in order to select the nested columns in CUR 2.0. For example, to select the product_from_location column in CUR 2.0, write an SQL statement selecting product.from_location.

      Example: SELECT product.from_location FROM COST_AND_USAGE_REPORT

      This selects the from_location column of the product map column.

    3. By default, the column selected with a dot operator is named by the attribute (for example, from_location). To match your existing CUR, you’ll need to declare an alias for the column in order to have the same as before.

      Example: SELECT product.from_location AS product_from_location FROM COST_AND_USAGE_REPORT

      For more details on nested columns, see the Data Exports table dictionary.

  3. Write the CUR content settings, identified in step 1, into the table configuration format for the CreateExport API. You need to provide these table configurations with your data query in the next step.

  4. In the Amazon SDK/CLI for Data Exports, use the CreateExport API to input your SQL query and table configurations into the data-query field.

    1. Specify delivery preferences, such as the target Amazon S3 bucket and the overwrite preference. We recommend choosing the same delivery preferences you had before. For more information on the required fields, see Amazon Data Exports in the Amazon Billing and Cost Management API Reference.

    2. Update the permissions of the target Amazon S3 bucket to allow Data Exports to write to the bucket. For more information, see Setting up an Amazon S3 bucket for data exports.

  5. Direct your data ingestion pipeline to read data from the directory in the Amazon S3 bucket where your CUR 2.0 is being delivered.

Method two: Create an export of CUR 2.0 with its new schema

You can create an export of CUR 2.0 with its new schema of nested columns and additional columns. However, you’ll need to adjust your current data pipeline to process these new columns. You do this using the console, the Amazon API, or SDK.

  1. Determine the CUR content settings (Include resource IDs, Split cost allocation data, and Time granularity) needed in order to match your CUR today.

    • You can determine the CUR content settings by going to Data Exports in the console and choosing your CUR export to view its details.

  2. Using either the Data Exports console page (Option A) or the Amazon SDK/CLI (Option B), create an export of CUR 2.0 that selects all columns from the “Cost and usage report” table.

  3. (Option A) To create the export in the console:

    1. In the navigation pane, choose Data Exports.

    2. On the Data Exports page, choose Create.

    3. Choose Standard data export.

      For the Cost and Usage Report (CUR 2.0) table, all columns are selected by default.

    4. Specify the CUR content settings that you identified in step 1.

    5. Under Data table delivery options, choose your options.

    6. Choose Create.

  4. (Option B) To create the export using the Amazon API/SDK, first write a query that selects all columns in the COST_AND_USAGE_REPORT table.

    1. Use the GetTable API to determine the complete list of columns and receive the full schema.

    2. Write the CUR content settings, identified in step 1, into the table configuration format for the CreateExport API.

    3. Use the CreateExport API to input your SQL query and table configurations into the data-query field.

    4. Specify delivery preferences, such as the target Amazon S3 bucket and the overwrite preference. We recommend choosing the same delivery preferences you had before. For more information on the required fields, see Amazon Data Exports in the Amazon Billing and Cost Management API Reference.

    5. Update the permissions of the target Amazon S3 bucket to allow Data Exports to write to the bucket. For more information, see Setting up an Amazon S3 bucket for data exports.

  5. Direct your data ingestion pipeline to read data from the directory in the Amazon S3 bucket where your CUR 2.0 is being delivered.

    You also need to update your data ingestion pipeline and your business intelligence tools to process the following new columns with nested key-values: product, resource_tags, cost_category, and discounts.