Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Getting started with Schema Registry

The following sections provide an overview and walk you through setting up and using Schema Registry. For information about Schema Registry concepts and components, see Amazon Glue Schema Registry.

Installing SerDe Libraries

The SerDe libraries provide a framework for serializing and deserializing data.

You will install the open source serializer for your applications producing data (collectively the "serializers"). The serializer handles serialization, compression, and the interaction with the Schema Registry. The serializer automatically extracts the schema from a record being written to a Schema Registry compatible destination, such as Amazon MSK. Likewise, you will install the open source deserializer on your applications consuming data.

To install the libraries on producers and consumers:

Using Amazon CLI for the Amazon Glue Schema Registry APIs

To use the Amazon CLI for the Amazon Glue Schema Registry APIs, make sure to update your Amazon CLI to the latest version.

Creating a registry

You may use the default registry or create as many new registries as necessary using the Amazon Glue APIs or Amazon Glue console.

Amazon Glue APIs

You can use these steps to perform this task using the Amazon Glue APIs.

To add a new registry, use the CreateRegistry action (Python: create_registry) API. Specify RegistryName as the name of the registry to be created, with a max length of 255, containing only letters, numbers, hyphens, underscores, dollar signs, or hash marks.

Specify a Description as a string not more than 2048 bytes long, matching the URI address multi-line string pattern.

Optionally, specify one or more Tags for your registry, as a map array of key-value pairs.

aws glue create-registry --registry-name registryName1 --description description

When your registry is created it is assigned an Amazon Resource Name (ARN), which you can view in the RegistryArn of the API response. Now that you've created a registry, create one or more schemas for that registry.

Amazon Glue console

To add a new registry in the Amazon Glue console:

When your registry is created it is assigned an Amazon Resource Name (ARN), which you can view by choosing the registry from the list in Schema registries. Now that you've created a registry, create one or more schemas for that registry.

Dealing with a specific record (JAVA POJO) for JSON

You can use a plain old Java object (POJO) and pass the object as a record. This is similar to the notion of a specific record in AVRO. The mbknor-jackson-jsonschema can generate a JSON schema for the POJO passed. This library can also inject additional information in the JSON schema.

The Amazon Glue Schema Registry library uses the injected "className" field in schema to provide a fully classified class name. The "className" field is used by the deserializer to deserialize into an object of that class.

 Example class :

@JsonSchemaDescription("This is a car")
@JsonSchemaTitle("Simple Car Schema")
@Builder
@AllArgsConstructor
@EqualsAndHashCode
// Fully qualified class name to be added to an additionally injected property
// called className for deserializer to determine which class to deserialize
// the bytes into
@JsonSchemaInject(
        strings = {@JsonSchemaString(path = "className",
                value = "com.amazonaws.services.schemaregistry.integrationtests.generators.Car")}
)
// List of annotations to help infer JSON Schema are defined by https://github.com/mbknor/mbknor-jackson-jsonSchema
public class Car {
    @JsonProperty(required = true)
    private String make;

    @JsonProperty(required = true)
    private String model;

    @JsonSchemaDefault("true")
    @JsonProperty
    public boolean used;

    @JsonSchemaInject(ints = {@JsonSchemaInt(path = "multipleOf", value = 1000)})
    @Max(200000)
    @JsonProperty
    private int miles;

    @Min(2000)
    @JsonProperty
    private int year;

    @JsonProperty
    private Date purchaseDate;

    @JsonProperty
    @JsonFormat(shape = JsonFormat.Shape.NUMBER)
    private Date listedDate;

    @JsonProperty
    private String[] owners;

    @JsonProperty
    private Collection<Float> serviceChecks;

    // Empty constructor is required by Jackson to deserialize bytes
    // into an Object of this class
    public Car() {}
}

Creating a schema

You can create a schema using the Amazon Glue APIs or the Amazon Glue console.

Amazon Glue APIs

You can use these steps to perform this task using the Amazon Glue APIs.

To add a new schema, use the CreateSchema action (Python: create_schema) API.

Specify a RegistryId structure to indicate a registry for the schema. Or, omit the RegistryId to use the default registry.

Specify a SchemaName consisting of letters, numbers, hyphens, or underscores, and DataFormat as AVRO or JSON. DataFormat once set on a schema is not changeable.

Specify a Compatibility mode:

Optionally, specify Tags for your schema.

Specify a SchemaDefinition to define the schema in Avro, JSON, or Protobuf data format. See the examples.

For Avro data format:

aws glue create-schema --registry-id RegistryName="registryName1" --schema-name testschema --compatibility NONE --data-format AVRO --schema-definition "{\"type\": \"record\", \"name\": \"r1\", \"fields\": [ {\"name\": \"f1\", \"type\": \"int\"}, {\"name\": \"f2\", \"type\": \"string\"} ]}"

aws glue create-schema --registry-id RegistryArn="arn:aws:glue:us-east-2:901234567890:registry/registryName1" --schema-name testschema --compatibility NONE --data-format AVRO  --schema-definition "{\"type\": \"record\", \"name\": \"r1\", \"fields\": [ {\"name\": \"f1\", \"type\": \"int\"}, {\"name\": \"f2\", \"type\": \"string\"} ]}"

For JSON data format:

aws glue create-schema --registry-id RegistryName="registryName" --schema-name testSchemaJson --compatibility NONE --data-format JSON --schema-definition "{\"$schema\": \"http://json-schema.org/draft-07/schema#\",\"type\":\"object\",\"properties\":{\"f1\":{\"type\":\"string\"}}}"

aws glue create-schema --registry-id RegistryArn="arn:aws:glue:us-east-2:901234567890:registry/registryName" --schema-name testSchemaJson --compatibility NONE --data-format JSON --schema-definition "{\"$schema\": \"http://json-schema.org/draft-07/schema#\",\"type\":\"object\",\"properties\":{\"f1\":{\"type\":\"string\"}}}"

For Protobuf data format:

aws glue create-schema --registry-id RegistryName="registryName" --schema-name testSchemaProtobuf --compatibility NONE --data-format PROTOBUF --schema-definition "syntax = \"proto2\";package org.test;message Basic { optional int32 basic = 1;}"

aws glue create-schema --registry-id RegistryArn="arn:aws:glue:us-east-2:901234567890:registry/registryName" --schema-name testSchemaProtobuf --compatibility NONE --data-format PROTOBUF --schema-definition "syntax = \"proto2\";package org.test;message Basic { optional int32 basic = 1;}"

Amazon Glue console

To add a new schema using the Amazon Glue console:

The schema is created and appears in the list under Schemas.

Working with Avro data format

Avro provides data serialization and data exchange services. Avro stores the data definition in JSON format making it easy to read and interpret. The data itself is stored in binary format.

For information on defining an Apache Avro schema, see the Apache Avro specification.

Working with JSON data format

Data can be serialized with JSON format. JSON Schema format defines the standard for JSON Schema format.

Updating a schema or registry

Once created you can edit your schemas, schema versions, or registry.

Updating a registry

You can update a registry using the Amazon Glue APIs or the Amazon Glue console. The name of an existing registry cannot be edited. You can edit the description for a registry.

Amazon Glue APIs

To update an existing registry, use the UpdateRegistry action (Python: update_registry) API.

Specify a RegistryId structure to indicate the registry that you want to update. Pass a Description to change the description for a registry.

aws glue update-registry --description updatedDescription --registry-id RegistryArn="arn:aws:glue:us-east-2:901234567890:registry/registryName1"

Amazon Glue console

To update a registry using the Amazon Glue console:

Updating a schema

You can update the description or compatibility setting for a schema.

To update an existing schema, use the UpdateSchema action (Python: update_schema) API.

Specify a SchemaId structure to indicate the schema that you want to update. One of VersionNumber or Compatibility has to be provided.

Code example 11:

aws glue update-schema --description testDescription --schema-id SchemaName="testSchema1",RegistryName="registryName1" --schema-version-number LatestVersion=true --compatibility NONE

aws glue update-schema --description testDescription --schema-id SchemaArn="arn:aws:glue:us-east-2:901234567890:schema/registryName1/testSchema1" --schema-version-number LatestVersion=true --compatibility NONE

Adding a schema version

When you add a schema version, you will need to compare the versions to make sure the new schema will be accepted.

To add a new version to an existing schema, use the RegisterSchemaVersion action (Python: register_schema_version) API.

Specify a SchemaId structure to indicate the schema for which you want to add a version, and a SchemaDefinition to define the schema.

Code example 12:

aws glue register-schema-version --schema-definition "{\"type\": \"record\", \"name\": \"r1\", \"fields\": [ {\"name\": \"f1\", \"type\": \"int\"}, {\"name\": \"f2\", \"type\": \"string\"} ]}" --schema-id SchemaArn="arn:aws:glue:us-east-1:901234567890:schema/registryName/testschema"

aws glue register-schema-version --schema-definition "{\"type\": \"record\", \"name\": \"r1\", \"fields\": [ {\"name\": \"f1\", \"type\": \"int\"}, {\"name\": \"f2\", \"type\": \"string\"} ]}" --schema-id SchemaName="testschema",RegistryName="testregistry"

The schema(s) version appears in the list of versions. If the version changed the compatibility mode, the version will be marked as a checkpoint.

Example of a schema version comparison

When you choose to Compare with previous version, you will see the previous and new versions displayed together. Changed information will be highlighted as follows:

You can also compare against earlier versions.

Deleting a schema or registry

Deleting a schema, a schema version, or a registry are permanent actions that cannot be undone.

Deleting a schema

You may want to delete a schema when it will no longer be used within a registry, using the Amazon Web Services Management Console, or the DeleteSchema action (Python: delete_schema) API.

Deleting one or more schemas is a permanent action that cannot be undone. Make sure that the schema or schemas are no longer needed.

To delete a schema from the registry, call the DeleteSchema action (Python: delete_schema) API, specifying the SchemaId structure to identify the schema.

For example:

aws glue delete-schema --schema-id SchemaArn="arn:aws:glue:us-east-2:901234567890:schema/registryName1/schemaname"

aws glue delete-schema --schema-id SchemaName="TestSchema6-deleteschemabyname",RegistryName="default-registry"

Amazon Glue console

To delete a schema from the Amazon Glue console:

The schema(s) you specified are deleted from the registry.

Deleting a schema version

As schemas accumulate in the registry, you may want to delete unwanted schema versions using the Amazon Web Services Management Console, or the DeleteSchemaVersions action (Python: delete_schema_versions) API. Deleting one or more schema versions is a permanent action that cannot be undone. Make sure that the schema versions are no longer needed.

When deleting schema versions, take note of the following constraints:

Specify the SchemaId structure to identify the schema, and specify Versions as a range of versions to delete. For more information on specifying a version or range of versions, see DeleteRegistry action (Python: delete_registry). The schema versions you specified are deleted from the registry.

Calling the ListSchemaVersions action (Python: list_schema_versions) API after this call will list the status of the deleted versions.

For example:

aws glue delete-schema-versions --schema-id SchemaName="TestSchema6",RegistryName="default-registry" --versions "1-1"

aws glue delete-schema-versions --schema-id SchemaArn="arn:aws:glue:us-east-2:901234567890:schema/default-registry/TestSchema6-NON-Existent" --versions "1-1"

The schema versions you specified are deleted from the registry.

Deleting a registry

You may want to delete a registry when the schemas it contains should no longer be organized under that registry. You will need to reassign those schemas to another registry.

Deleting one or more registries is a permanent action that cannot be undone. Make sure that the registry or registries no longer needed.

The default registry can be deleted using the Amazon CLI.

Amazon Glue API

To delete the entire registry including the schema and all of its versions, call the DeleteRegistry action (Python: delete_registry) API. Specify a RegistryId structure to identify the registry.

For example:

aws glue delete-registry --registry-id RegistryArn="arn:aws:glue:us-east-2:901234567890:registry/registryName1"

aws glue delete-registry --registry-id RegistryName="TestRegistry-deletebyname"

To get the status of the delete operation, you can call the GetRegistry API after the asynchronous call.

Amazon Glue console

To delete a registry from the Amazon Glue console:

The registries you selected are deleted from Amazon Glue.

IAM examples for serializers

For serializers, you should create a minimal policy similar to that below to give you the ability to find the schemaVersionId for a given schema definition. Note, you should have read permissions on the registry in order to read the schemas in the registry. You can limit the registries that can be read by using the Resource clause.

Code example 13:

{
    "Sid" : "GetSchemaByDefinition",
    "Effect" : "Allow",
    "Action" : 
	[
        "glue:GetSchemaByDefinition"
    ],
        "Resource" : ["arn:aws:glue:us-east-2:012345678:registry/registryname-1",
                      "arn:aws:glue:us-east-2:012345678:schema/registryname-1/schemaname-1",
                      "arn:aws:glue:us-east-2:012345678:schema/registryname-1/schemaname-2"
                     ]  
}

Further, you can also allow producers to create new schemas and versions by including the following extra methods. Note, you should be able to inspect the registry in order to add/remove/evolve the schemas inside it. You can limit the registries that can be inspected by using the Resource clause.

Code example 14:

{
    "Sid" : "RegisterSchemaWithMetadata",
    "Effect" : "Allow",
    "Action" : 
	[
        "glue:GetSchemaByDefinition",
        "glue:CreateSchema",
        "glue:RegisterSchemaVersion",
        "glue:PutSchemaVersionMetadata",
    ],
    "Resource" : ["arn:aws-cn:glue:aws-region:123456789012:registry/registryname-1",
                  "arn:aws-cn:glue:aws-region:123456789012:schema/registryname-1/schemaname-1",
                  "arn:aws-cn:glue:aws-region:123456789012:schema/registryname-1/schemaname-2"
                 ]  
}

IAM examples for deserializers

For deserializers (consumer side), you should create a policy similar to that below to allow the deserializer to fetch the schema from the Schema Registry for deserialization. Note, you should be able to inspect the registry in order to fetch the schemas inside it.

Code example 15:

{
    "Sid" : "GetSchemaVersion",
    "Effect" : "Allow",
    "Action" : 
	[
        "glue:GetSchemaVersion"
    ],
    "Resource" : ["*"]  
}

Private connectivity using Amazon PrivateLink

You can use Amazon PrivateLink to connect your data producer’s VPC to Amazon Glue by defining an interface VPC endpoint for Amazon Glue. When you use a VPC interface endpoint, communication between your VPC and Amazon Glue is conducted entirely within the Amazon network. For more information, see Using Amazon Glue with VPC Endpoints.

Accessing Amazon CloudWatch metrics

Amazon CloudWatch metrics are available as part of CloudWatch’s free tier. You can access these metrics in the CloudWatch Console. API-Level metrics include CreateSchema (Success and Latency), GetSchemaByDefinition, (Success and Latency), GetSchemaVersion (Success and Latency), RegisterSchemaVersion (Success and Latency), PutSchemaVersionMetadata (Success and Latency). Resource-level metrics include Registry.ThrottledByLimit, SchemaVersion.ThrottledByLimit, SchemaVersion.Size.

Sample Amazon CloudFormation template for Schema Registry

The following is a sample template for creating Schema Registry resources in Amazon CloudFormation. To create this stack in your account, copy the above template into a file SampleTemplate.yaml, and run the following command:

aws cloudformation create-stack --stack-name ABCSchemaRegistryStack --template-body "'cat SampleTemplate.yaml'"

This example uses AWS::Glue::Registry to create a registry, AWS::Glue::Schema to create a schema, AWS::Glue::SchemaVersion to create a schema version, and AWS::Glue::SchemaVersionMetadata to populate schema version metadata.

Description: "A sample CloudFormation template for creating Schema Registry resources."
Resources:
  ABCRegistry:
    Type: "AWS::Glue::Registry"
    Properties:
      Name: "ABCSchemaRegistry"
      Description: "ABC Corp. Schema Registry"
      Tags:
        - Key: "Project"
          Value: "Foo"
  ABCSchema:
    Type: "AWS::Glue::Schema"
    Properties:
      Registry:
        Arn: !Ref ABCRegistry
      Name: "TestSchema"
      Compatibility: "NONE"
      DataFormat: "AVRO"
      SchemaDefinition: >
        {"namespace":"foo.avro","type":"record","name":"user","fields":[{"name":"name","type":"string"},{"name":"favorite_number","type":"int"}]}
      Tags:
        - Key: "Project"
          Value: "Foo"
  SecondSchemaVersion:
    Type: "AWS::Glue::SchemaVersion"
    Properties:
      Schema:
        SchemaArn: !Ref ABCSchema
      SchemaDefinition: >
        {"namespace":"foo.avro","type":"record","name":"user","fields":[{"name":"status","type":"string", "default":"ON"}, {"name":"name","type":"string"},{"name":"favorite_number","type":"int"}]}
  FirstSchemaVersionMetadata:
    Type: "AWS::Glue::SchemaVersionMetadata"
    Properties:
      SchemaVersionId: !GetAtt ABCSchema.InitialSchemaVersionId
      Key: "Application"
      Value: "Kinesis"
  SecondSchemaVersionMetadata:
    Type: "AWS::Glue::SchemaVersionMetadata"
    Properties:
      SchemaVersionId: !Ref SecondSchemaVersion
      Key: "Application"
      Value: "Kinesis"