Output Data - Amazon SageMaker
Output DirectoriesConfidence ScoreWorker MetadataOutput MetadataClassification Job OutputMulti-label Classification Job OutputBounding Box Job OutputNamed Entity RecognitionLabel Verification Job OutputSemantic Segmentation Job OutputVideo Frame Object Detection OutputVideo Frame Object Tracking Output3D Point Cloud Semantic Segmentation Output3D Point Cloud Object Detection Output3D Point Cloud Object Tracking Output3D-2D Object Tracking Point Cloud Object Tracking Output
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Output Data

The output from a labeling job is placed in the Amazon S3 location that you specified in the console or in the call to the CreateLabelingJob operation. Output data appears in this location when the workers have submitted one or more tasks, or when tasks expire. Note that it may take a few minutes for output data to appear in Amazon S3 after the worker submits the task or the task expires.

Each line in the output data file is identical to the manifest file with the addition of an attribute and value for the label assigned to the input object. The attribute name for the value is defined in the console or in the call to the CreateLabelingJob operation. You can't use -metadata in the label attribute name. If you are running an image semantic segmentation, 3D point cloud semantic segmentation, or 3D point cloud object tracking job, the label attribute must end with -ref. For any other type of job, the attribute name can't end with -ref.

The output of the labeling job is the value of the key-value pair with the label. The label and the value overwrites any existing JSON data in the input file with the new value.

For example, the following is the output from an image classification labeling job where the input data files were stored in an Amazon S3 AWSDOC-EXAMPLE-BUCKET and the label attribute name was defined as sport. In this example the JSON object is formatted for readability, in the actual output file the JSON object is on a single line. For more information about the data format, see JSON Lines. 

{
    "source-ref": "s3://AWSDOC-EXAMPLE-BUCKET/image_example.png",
    "sport":0,
    "sport-metadata":
    {
        "class-name": "football",
        "confidence": 0.00,
        "type":"groundtruth/image-classification",
        "job-name": "identify-sport",
        "human-annotated": "yes",
        "creation-date": "2018-10-18T22:18:13.527256"
    }
}

The value of the label can be any valid JSON. In this case the label's value is the index of the class in the classification list. Other job types, such as bounding box, have more complex values.

Any key-value pair in the input manifest file other than the label attribute is unchanged in the output file. You can use this to pass data to your application.

The output from a labeling job can be used as the input to another labeling job. You can use this when you are chaining together labeling jobs. For example, you can send one labeling job to determine the sport that is being played. Then you send another using the same data to determine if the sport is being played indoors or outdoors. By using the output data from the first job as the manifest for the second job, you can consolidate the results of the two jobs into one output file for easier processing by your applications.

The output data file is written to the output location periodically while the job is in progress. These intermediate files contain one line for each line in the manifest file. If an object is labeled, the label is included. If the object hasn't been labeled, it is written to the intermediate output file identically to the manifest file.

Output Directories

Ground Truth creates several directories in your Amazon S3 output path. These directories contain the results of your labeling job and other artifacts of the job. The top-level directory for a labeling job is given the same name as your labeling job; the output directories are placed beneath it. For example, if you named your labeling job find-people, your output would be in the following directories:

s3://AWSDOC-EXAMPLE-BUCKET/find-people/activelearning
s3://AWSDOC-EXAMPLE-BUCKET/find-people/annotations
s3://AWSDOC-EXAMPLE-BUCKET/find-people/inference
s3://AWSDOC-EXAMPLE-BUCKET/find-people/manifests
s3://AWSDOC-EXAMPLE-BUCKET/find-people/training

Each directory contains the following output:

Active Learning Directory

The activelearning directory is only present when you are using automated data labeling. It contains the input and output validation set for automated data labeling, and the input and output folder for automatically labeled data.

Annotations Directory

The annotations directory contains all of the annotations made by the workforce. These are the responses from individual workers that have not been consolidated into a single label for the data object.

There are three subdirectories in the annotations directory.

  • The first, worker-response, contains the responses from individual workers. This contains a subdirectory for each iteration, which in turn contains a subdirectory for each data object in that iteration. The worker response data for each data object is stored in a timestamped JSON file that contains the answers submitted by each worker for that data object, and if you use a private workforce, metadata about those workers. To learn more about this metadata, see Worker Metadata.

  • The second, consolidated-annotation, contains information required to consolidate the annotations in the current batch into labels for your data objects.

  • The third, intermediate, contains the output manifest for the current batch with any completed labels. This file is updated as the label for each data object is completed.

Note

We recommend that you do not use files that are not mentioned in the documentation.

Inference Directory

The inference directory is only present when you are using automated data labeling. This directory contains the input and output files for the SageMaker batch transform used while labeling data objects.

Manifest Directory

The manifest directory contains the output manifest from your labeling job. There is one subdirectory in the manifest directory, output. The output directory contains the output manifest file for your labeling job. The file is named output.manifest.

Training Directory

The training directory is only present when you are using automated data labeling. This directory contains the input and output files used to train the automated data labeling model.

Confidence Score

When you have more than one worker annotate a single task, your label results from annotation consolidation. Ground Truth calculates a confidence score for each label. A confidence score is a number between 0 and 1 that indicates how confident Ground Truth is in the label. You can use the confidence score to compare labeled data objects to each other, and to identify the least or most confident labels.

You should not interpret the value of a confidence score as an absolute value, or compare confidence scores across labeling jobs. For example, if all of the confidence scores are between 0.98 and 0.998, you should only compare the data objects with each other and not rely on the high confidence scores.

You should not compare the confidence scores of human-labeled data objects and auto-labeled data objects. The confidence scores for humans are calculated using the annotation consolidation function for the task, while the confidence scores for automated labeling are calculated using a model that incorporates object features. The two models generally have different scales and average confidence.

For a bounding box labeling job, Ground Truth calculates a confidence score per box. You can compare confidence scores within one image or across images for the same labeling type (human or auto). You can't compare confidence scores across labeling jobs.

If a single worker annotates a task (NumberOfHumanWorkersPerDataObject is set to 1 or in the console, you enter 1 for Number of workers per dataset object), the confidence score is set to 0.00.

Worker Metadata

Ground Truth provides information that you can use to track individual workers in task output data. The following data is located in the directories under the worker-response located in the Annotations Directory:

  • The acceptanceTime is the time that the worker accepted the task. The format of this date and time stamp is YYYY-MM-DDTHH:MM:SS.mmmZ for the year (YYYY), month (MM), day (DD), hour (HH), minute (MM), second (SS) and millisecond (mmm). The date and time are separated by a T.

  • The submissionTime is the time that the worker submitted their annotations using the Submit button. The format of this date and time stamp is YYYY-MM-DDTHH:MM:SS.mmmZ for the year (YYYY), month (MM), day (DD), hour (HH), minute (MM), second (SS) and millisecond (mmm). The date and time are separated by a T.

  • timeSpentInSeconds reports the total time, in seconds, that a worker actively worked on that task. This metric does not include time when a worker paused or took a break.

  • The workerId is unique to each worker.

  • If you use a private workforce, in workerMetadata, you see the following.

    • The identityProviderType is the service used to manage the private workforce.

    • The issuer is the Cognito user pool or OIDC Identity Provider (IdP) issuer associated with the work team assigned to this human review task.

    • A unique sub identifier refers to the worker. If you create a workforce using Amazon Cognito, you can retrieve details about this worker (such as the name or user name) using this ID using Amazon Cognito. To learn how, see Managing and Searching for User Accounts in Amazon Cognito Developer Guide.

The following is an example of the output you may see if you use Amazon Cognito to create a private workforce. This is identified in the identityProviderType.

"submissionTime": "2020-12-28T18:59:58.321Z",
"acceptanceTime": "2020-12-28T18:59:15.191Z", 
"timeSpentInSeconds": 40.543,
"workerId": "a12b3cdefg4h5i67",
"workerMetadata": {
    "identityData": {
        "identityProviderType": "Cognito",
        "issuer": "https://cognito-idp.aws-region.amazonaws.com/aws-region_123456789",
        "sub": "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"
    }
}

The following is an example of the workerMetadata you may see if you use your own OIDC IdP to create a private workforce:

"workerMetadata": {
        "identityData": {
            "identityProviderType": "Oidc",
            "issuer": "https://example-oidc-ipd.com/adfs",
            "sub": "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"
        }
}

To learn more about using private workforces, see Use a Private Workforce.

Output Metadata

The output from each job contains metadata about the label assigned to data objects. These elements are the same for all jobs with minor variations. The following example shows the metadata elements:

    "confidence": 0.00,
    "type": "groundtruth/image-classification",
    "job-name": "identify-animal-species",
    "human-annotated": "yes",
    "creation-date": "2020-10-18T22:18:13.527256"

The elements have the following meaning:

  • confidence – The confidence that Ground Truth has that the label is correct. For more information, see Confidence Score.

  • type – The type of classification job. For job types, see Built-in Task Types.

  • job-name – The name assigned to the job when it was created.

  • human-annotated – Whether the data object was labeled by a human or by automated data labeling. For more information, see Automate Data Labeling.

  • creation-date – The date and time that the label was created.

Classification Job Output

The following are sample outputs (output manifest files) from an image classification job and a text classification job. They include the label that Ground Truth assigned to the data object, the value for the label, and metadata that describes the label.

In addition to the standard metadata elements, the metadata for a classification job includes the text value of the label's class. For more information, see Image Classification - MXNet.

The red, italicized text in the examples below depends on labeling job specifications and output data. 

{
    "source-ref":"s3://AWSDOC-EXAMPLE-BUCKET/example_image.jpg",
    "species":"0",
    "species-metadata":
    {
        "class-name": "dog",
        "confidence": 0.00,
        "type": "groundtruth/image-classification",
        "job-name": "identify-animal-species",
        "human-annotated": "yes",
        "creation-date": "2018-10-18T22:18:13.527256"
    }
}
{
    "source":"The food was delicious",
    "mood":"1",
    "mood-metadata":
    {
        "class-name": "positive",
        "confidence": 0.8,
        "type": "groundtruth/text-classification",
        "job-name": "label-sentiment",
        "human-annotated": "yes",
        "creation-date": "2020-10-18T22:18:13.527256"
    }
}

Multi-label Classification Job Output

The following are example output manifest files from a multi-label image classification job and a multi-label text classification job. They include the labels that Ground Truth assigned to the data object (for example, the image or piece of text) and metadata that describes the labels the worker saw when completing the labeling task.

The label attribute name parameter (for example, image-label-attribute-name) contains an array of all of the labels selected by at least one of the workers who completed this task. This array contains integer keys (for example, [1,0,8]) that correspond to the labels found in class-map. In the multi-label image classification example, bicycle, person, and clothing were selected by at least one of the workers who completed the labeling task for the image, exampleimage.jpg.

The confidence-map shows the confidence score that Ground Truth assigned to each label selected by a worker. To learn more about Ground Truth confidence scores, see Confidence Score.

The red, italicized text in the examples below depends on labeling job specifications and output data.

The following is an example of a multi-label image classification output manifest file. 

{
    "source-ref": "s3://AWSDOC-EXAMPLE-BUCKET/example_image.jpg",
    "image-label-attribute-name":[1,0,8],
    "image-label-attribute-name-metadata":
       {
        "job-name":"labeling-job/image-label-attribute-name",
        "class-map":
            {
                "1":"bicycle","0":"person","8":"clothing"
            },
        "human-annotated":"yes",
        "creation-date":"2020-02-27T21:36:25.000201",
        "confidence-map":
            {
                "1":0.95,"0":0.77,"8":0.2
            },
        "type":"groundtruth/image-classification-multilabel"
        }
}

The following is an example of a multi-label text classification output manifest file. In this example, approving, sad and critical were selected by at least one of the workers who completed the labeling task for the object exampletext.txt found in AWSDOC-EXAMPLE-BUCKET.

{
    "source-ref": "AWSDOC-EXAMPLE-BUCKET/text_file.txt",
    "text-label-attribute-name":[1,0,4],
    "text-label-attribute-name-metadata":
       {
        "job-name":"labeling-job/text-label-attribute-name",
        "class-map":
            {
                "1":"approving","0":"sad","4":"critical"
            },
        "human-annotated":"yes",
        "creation-date":"2020-02-20T21:36:25.000201",
        "confidence-map":
            {
                "1":0.95,"0":0.77,"4":0.2
            },
        "type":"groundtruth/text-classification-multilabel"
        }
}

Bounding Box Job Output

The following is sample output (output manifest file) from a bounding box job. For this task, three bounding boxes are returned. The label value contains information about the size of the image, and the location of the bounding boxes.

The class_id element is the index of the box's class in the list of available classes for the task. The class-map metadata element contains the text of the class.

The metadata has a separate confidence score for each bounding box. The metadata also includes the class-map element that maps the class_id to the text value of the class. For more information, see Object Detection - MXNet.

The red, italicized text in the examples below depends on labeling job specifications and output data. 

{
    "source-ref": "s3://AWSDOC-EXAMPLE-BUCKET/example_image.png",
    "bounding-box-attribute-name":
    {
        "image_size": [{ "width": 500, "height": 400, "depth":3}],
        "annotations":
        [
            {"class_id": 0, "left": 111, "top": 134,
                    "width": 61, "height": 128},
            {"class_id": 5, "left": 161, "top": 250,
                     "width": 30, "height": 30},
            {"class_id": 5, "left": 20, "top": 20,
                     "width": 30, "height": 30}
        ]
    },
    "bounding-box-attribute-name-metadata":
    {
        "objects":
        [
            {"confidence": 0.8},
            {"confidence": 0.9},
            {"confidence": 0.9}
        ],
        "class-map":
        {
            "0": "dog",
            "5": "bone"
        },
        "type": "groundtruth/object-detection",
        "human-annotated": "yes",
        "creation-date": "2018-10-18T22:18:13.527256",
        "job-name": "identify-dogs-and-toys"
    }
 }

The output of a bounding box adjustment job looks like the following JSON. Note that the original JSON is kept intact and two new jobs are listed, each with “adjust-” prepended to the original attribute’s name. 

{
    "source-ref": "S3 bucket location",
    "bounding-box-attribute-name":
    {
        "image_size": [{ "width": 500, "height": 400, "depth":3}],
        "annotations":
        [
            {"class_id": 0, "left": 111, "top": 134,
                    "width": 61, "height": 128},
            {"class_id": 5, "left": 161, "top": 250,
                     "width": 30, "height": 30},
            {"class_id": 5, "left": 20, "top": 20,
                     "width": 30, "height": 30}
        ]
    },
    "bounding-box-attribute-name-metadata":
    {
        "objects":
        [
            {"confidence": 0.8},
            {"confidence": 0.9},
            {"confidence": 0.9}
        ],
        "class-map":
        {
            "0": "dog",
            "5": "bone"
        },
        "type": "groundtruth/object-detection",
        "human-annotated": "yes",
        "creation-date": "2018-10-18T22:18:13.527256",
        "job-name": "identify-dogs-and-toys"
    },
    "adjusted-bounding-box":
    {
        "image_size": [{ "width": 500, "height": 400, "depth":3}],
        "annotations":
        [
            {"class_id": 0, "left": 110, "top": 135,
                    "width": 61, "height": 128},
            {"class_id": 5, "left": 161, "top": 250,
                     "width": 30, "height": 30},
            {"class_id": 5, "left": 10, "top": 10,
                     "width": 30, "height": 30}
        ]
    },
    "adjusted-bounding-box-metadata":
    {
        "objects":
        [
            {"confidence": 0.8},
            {"confidence": 0.9},
            {"confidence": 0.9}
        ],
        "class-map":
        {
            "0": "dog",
            "5": "bone"
        },
        "type": "groundtruth/object-detection",
        "human-annotated": "yes",
        "creation-date": "2018-11-20T22:18:13.527256",
        "job-name": "adjust-bounding-boxes-on-dogs-and-toys",
        "adjustment-status": "adjusted"
    }
}

In this output, the job's type doesn't change, but an adjustment-status field is added. This field has the value of adjusted or unadjusted. If multiple workers have reviewed the object and at least one adjusted the label, the status is adjusted.

Named Entity Recognition

The following is an example output manifest file from a named entity recognition (NER) labeling task. For this task, seven entities are returned.

In the output manifest, the JSON object, annotations, includes a list of the labels (label categories) that you provided.

Worker responses are in a list named entities. Each entity in this list is a JSON object that contains a label value that matches one in the labels list, an integer startOffset value for labeled span's starting Unicode offset, and an integer endOffset value for the ending Unicode offset.

The metadata has a separate confidence score for each entity. If a single worker labeled each data object, the confidence value for each entity will be zero.

The red, italicized text in the examples below depends on labeling job inputs and worker responses.

{
    "source": "Amazon SageMaker is a cloud machine-learning platform that was launched in November 2017. SageMaker enables developers to create, train, and deploy machine-learning (ML) models in the cloud. SageMaker also enables developers to deploy ML models on embedded systems and edge-devices",
    "ner-labeling-job-attribute-name": {
        "annotations": {
            "labels": [
                {
                    "label": "Date",
                    "shortDisplayName": "dt"
                },
                {
                    "label": "Verb",
                    "shortDisplayName": "vb"
                },
                {
                    "label": "Thing",
                    "shortDisplayName": "tng"
                },
                {
                    "label": "People",
                    "shortDisplayName": "ppl"
                }
            ],
            "entities": [
                {
                    "label": "Thing",
                    "startOffset": 22,
                    "endOffset": 53
                },
                {
                    "label": "Thing",
                    "startOffset": 269,
                    "endOffset": 281
                },
                {
                    "label": "Verb",
                    "startOffset": 63,
                    "endOffset": 71
                },
                {
                    "label": "Verb",
                    "startOffset": 228,
                    "endOffset": 234
                },
                {
                    "label": "Date",
                    "startOffset": 75,
                    "endOffset": 88
                },
                {
                    "label": "People",
                    "startOffset": 108,
                    "endOffset": 118
                },
                {
                    "label": "People",
                    "startOffset": 214,
                    "endOffset": 224
                }
            ]
        }
    },
    "ner-labeling-job-attribute-name-metadata": {
        "job-name": "labeling-job/example-ner-labeling-job",
        "type": "groundtruth/text-span",
        "creation-date": "2020-10-29T00:40:39.398470",
        "human-annotated": "yes",
        "entities": [
            {
                "confidence": 0
            },
            {
                "confidence": 0
            },
            {
                "confidence": 0
            },
            {
                "confidence": 0
            },
            {
                "confidence": 0
            },
            {
                "confidence": 0
            },
            {
                "confidence": 0
            }
        ]
    }
}

Label Verification Job Output

The output (output manifest file) of a bounding box verification job looks different than the output of a bounding box annotation job. That's because the workers have a different type of task. They're not labeling objects, but evaluating the accuracy of prior labeling, making a judgment, and then providing that judgment and perhaps some comments.

If human workers are verifying or adjusting prior bounding box labels, the output of a verification job would look like the following JSON. The red, italicized text in the examples below depends on labeling job specifications and output data. 

{
    "source-ref":"s3://AWSDOC-EXAMPLE-BUCKET/image_example.png",
    "bounding-box-attribute-name":
    {
        "image_size": [{ "width": 500, "height": 400, "depth":3}],
        "annotations":
        [
            {"class_id": 0, "left": 111, "top": 134,
                    "width": 61, "height": 128},
            {"class_id": 5, "left": 161, "top": 250,
                     "width": 30, "height": 30},
            {"class_id": 5, "left": 20, "top": 20,
                     "width": 30, "height": 30}
        ]
    },
    "bounding-box-attribute-name-metadata":
    {
        "objects":
        [
            {"confidence": 0.8},
            {"confidence": 0.9},
            {"confidence": 0.9}
        ],
        "class-map":
        {
            "0": "dog",
            "5": "bone"
        },
        "type": "groundtruth/object-detection",
        "human-annotated": "yes",
        "creation-date": "2018-10-18T22:18:13.527256",
        "job-name": "identify-dogs-and-toys"
    },
    "verify-bounding-box-attribute-name":"1",
    "verify-bounding-box-attribute-name-metadata":
    {
        "class-name": "bad",
        "confidence": 0.93,
        "type": "groundtruth/label-verification",
        "job-name": "verify-bounding-boxes",
        "human-annotated": "yes",
        "creation-date": "2018-11-20T22:18:13.527256",
        "worker-feedback": [
            {"comment": "The bounding box on the bird is too wide on the right side."},
            {"comment": "The bird on the upper right is not labeled."}
        ]
    }
}

Although the type on the original bounding box output was groundtruth/object-detection, the new type is groundtruth/label-verification. Also note that the worker-feedback array provides worker comments. If the worker doesn't provide comments, the empty fields are excluded during consolidation.

Semantic Segmentation Job Output

The following is the output manifest file from a semantic segmentation labeling job. The value of the label for this job is a reference to a PNG file in an Amazon S3 bucket.

In addition to the standard elements, the metadata for the label includes a color map that defines which color is used to label the image, the class name associated with the color, and the confidence score for each color. For more information, see Semantic Segmentation Algorithm.

The red, italicized text in the examples below depends on labeling job specifications and output data. 

{
    "source-ref": "s3://AWSDOC-EXAMPLE-BUCKET/example_city_image.png",
    "city-streets-ref": "S3 bucket location",
    "city-streets-ref-metadata": {
      "internal-color-map": {
        "0": {
           "class-name": "BACKGROUND",
           "confidence": 0.9,
           "hex-color": "#ffffff"
        },
        "1": {
           "class-name": "buildings",
           "confidence": 0.9,
           "hex-color": "#2acf59"
        },
        "2":  {
           "class-name": "road",
           "confidence": 0.9,
           "hex-color": "#f28333"
       }
     },
     "type": "groundtruth/semantic-segmentation",
     "human-annotated": "yes",
     "creation-date": "2018-10-18T22:18:13.527256",
     "job-name": "label-city-streets",
     },
     "verify-city-streets-ref":"1",
     "verify-city-streets-ref-metadata":
     {
        "class-name": "bad",
        "confidence": 0.93,
        "type": "groundtruth/label-verification",
        "job-name": "verify-city-streets",
        "human-annotated": "yes",
        "creation-date": "2018-11-20T22:18:13.527256",
        "worker-feedback": [
            {"comment": "The mask on the leftmost building is assigned the wrong side of the road."},
            {"comment": "The curb of the road is not labeled but the instructions say otherwise."}
        ]
     }
}

Confidence is scored on a per-image basis. Confidence scores are the same across all classes within an image.

The output of a semantic segmentation adjustment job looks similar to the following JSON.

{
    "source-ref": "s3://AWSDOC-EXAMPLE-BUCKET/example_city_image.png",
    "city-streets-ref": "S3 bucket location",
    "city-streets-ref-metadata": {
      "internal-color-map": {
        "0": {
           "class-name": "BACKGROUND",
           "confidence": 0.9,
           "hex-color": "#ffffff"
        },
        "1": {
           "class-name": "buildings",
           "confidence": 0.9,
           "hex-color": "#2acf59"
        },
        "2":  {
           "class-name": "road",
           "confidence": 0.9,
           "hex-color": "#f28333"
       }
     },
     "type": "groundtruth/semantic-segmentation",
     "human-annotated": "yes",
     "creation-date": "2018-10-18T22:18:13.527256",
     "job-name": "label-city-streets",
     },
     "adjusted-city-streets-ref": "s3://AWSDOC-EXAMPLE-BUCKET/example_city_image.png",
     "adjusted-city-streets-ref-metadata": {
      "internal-color-map": {
        "0": {
           "class-name": "BACKGROUND",
           "confidence": 0.9,
           "hex-color": "#ffffff"
        },
        "1": {
           "class-name": "buildings",
           "confidence": 0.9,
           "hex-color": "#2acf59"
        },
        "2":  {
           "class-name": "road",
           "confidence": 0.9,
           "hex-color": "#f28333"
       }
     },
     "type": "groundtruth/semantic-segmentation",
     "human-annotated": "yes",
     "creation-date": "2018-11-20T22:18:13.527256",
     "job-name": "adjust-label-city-streets",
     }
}

Video Frame Object Detection Output

The following is the output manifest file from a video frame object detection labeling job. The red, italicized text in the examples below depends on labeling job specifications and output data.

In addition to the standard elements, the metadata includes a class map that lists each class that has at least one label in the sequence. The metadata also includes job-name which is the name you assigned to the labeling job. For adjustment tasks, If one or more bounding boxes were modified, there is an adjustment-status parameter in the metadata for audit workflows that is set to adjusted. 


{
    "source-ref": "s3://DOC-EXAMPLE-BUCKET/example-path/input-manifest.json",
    "CarObjectDetection-ref": "s3://AWSDOC-EXAMPLE-BUCKET/output/labeling-job-name/annotations/consolidated-annotation/output/0/SeqLabel.json",
    "CarObjectDetection-ref-metadata": {
        "class-map": {
            "0": "car",
            "1": "bus"
        },
        "job-name": "labeling-job/labeling-job-name",
        "human-annotated": "yes",
        "creation-date": "2021-09-29T05:50:35.566000",
        "type": "groundtruth/video-object-detection"
        }
}

Ground Truth creates one output sequence file for each sequence of video frames that was labeled. Each output sequence file contains the following:

  • All annotations for all frames in a sequence in the detection-annotations list of JSON objects.

  • For each frame that was annotated by a worker, the frame file name (frame), number (frame-no), a list of JSON objects containing annotations (annotations), and if applicable, frame-attributes. The name of this list is defined by the task type you use: polylines, polygons, keypoints, and for bounding boxes, annotations.

    Each JSON object contains information about a single annotation and associated label. The following table outlines the parameters you'll see for each video frame task type.

    Task Type Parameters

    Bounding Box

    Box dimensions: height and width

    Box top, left corner pixel location: top and left

    Keypoint

    		 Keypoint vertices: { "x": int, "y": int }

    Polygon

    A list of polygon vertices: vertices

    Polygon vertices: { "x": int, "y": int }

    A polygon is a closed shape and so the first point will also represent the last point.

    Polyline

    A list of polyline vertices: vertices

    Polyline vertices: { "x": int, "y": int }

    In addition to task type specific values, you will see the following in each JSON object:

    • Values of any label-category-attributes that were specified for that label.

    • The class-id of the box. Use the class-map in the output manifest file to see which label category this ID maps to.

The following is an example of a SeqLabel.json file from a bounding box video frame object detection labeling job. This file will be located under s3://your-output-bucket/output-prefix/annotations/consolidated-annotation/output/annotation-number/

{
    "detection-annotations": [
        {
            "annotations": [
                {
                    "height": 41,
                    "width": 53,
                    "top": 152,
                    "left": 339,
                    "class-id": "1",
                    "label-category-attributes": {
                        "occluded": "no",
                        "size": "medium"
                    }
                },
                {
                    "height": 24,
                    "width": 37,
                    "top": 148,
                    "left": 183,
                    "class-id": "0",
                    "label-category-attributes": {
                        "occluded": "no",
                    }
                }
            ],
            "frame-no": 0,
            "frame": "frame_0000.jpeg", 
            "frame-attributes": {name: value, name: value}
        },
        {
            "annotations": [
                {
                    "height": 41,
                    "width": 53,
                    "top": 152,
                    "left": 341,
                    "class-id": "0",
                    "label-category-attributes": {}
                },
                {
                    "height": 24,
                    "width": 37,
                    "top": 141,
                    "left": 177,
                    "class-id": "0",
                    "label-category-attributes": {
                        "occluded": "no",
                    }
                }
            ],
            "frame-no": 1,
            "frame": "frame_0001.jpeg",
            "frame-attributes": {name: value, name: value}
        }
    ]
}

Video Frame Object Tracking Output

The following is the output manifest file from a video frame object tracking labeling job. The red, italicized text in the examples below depends on labeling job specifications and output data.

In addition to the standard elements, the metadata includes a class map that lists each class that has at least one label in the sequence of frames. The metadata also includes job-name which is the name you assigned to the labeling job. For adjustment tasks, If one or more bounding boxes were modified, there is an adjustment-status parameter in the metadata for audit workflows that is set to adjusted. 

{
    "source-ref": "s3://DOC-EXAMPLE-BUCKET/example-path/input-manifest.json",
    "CarObjectTracking-ref": "s3://AWSDOC-EXAMPLE-BUCKET/output/labeling-job-name/annotations/consolidated-annotation/output/0/SeqLabel.json",
    "CarObjectTracking-ref-metadata": {
        "class-map": {
            "0": "car",
            "1": "bus"
        },
        "job-name": "labeling-job/labeling-job-name",
        "human-annotated": "yes",
        "creation-date": "2021-09-29T05:50:35.566000",
        "type": "groundtruth/video-object-tracking"
        }
 }

Ground Truth creates one output sequence file for each sequence of video frames that was labeled. Each output sequence file contains the following:

  • All annotations for all frames in a sequence in the tracking-annotations list of JSON objects.

  • For each frame that was annotated by a worker, the frame (frame), number (frame-no), a list of JSON objects containing annotations (annotations), and if applicable, frame attributes (frame-attributes). The name of this list is defined by the task type you use: polylines, polygons, keypoints, and for bounding boxes, annotations.

    Each JSON object contains information about a single annotation and associated label. The following table outlines the parameters you'll see for each video frame task type.

    Task Type Parameters

    Bounding Box

    Box dimensions: height and width

    Box top, left corner pixel location: top and left

    Keypoint

    		 Keypoint vertices: { "x": int, "y": int }

    Polygon

    A list of polygon vertices: vertices

    Polygon vertices: { "x": int, "y": int }

    A polygon is a closed shape and so the first point will also represent the last point.

    Polyline

    A list of polyline vertices: vertices

    Polyline vertices: { "x": int, "y": int }

    In addition to task type specific values, you will see the following in each JSON object:

    • Values of any label-category-attributes that were specified for that label.

    • The class-id of the box. Use the class-map in the output manifest file to see which label category this ID maps to.

    • An object-id which identifies an instance of a label. This ID will be the same across frames if a worker identified the same instance of an object in multiple frames. For example, if a car appeared in multiple frames, all bounding boxes uses to identify that car would have the same object-id.

    • The object-name which is the instance ID of that annotation.

The following is an example of a SeqLabel.json file from a bounding box video frame object tracking labeling job. This file will be located under s3://your-output-bucket/output-prefix/annotations/consolidated-annotation/output/annotation-number/

{
    "tracking-annotations": [
        {
            "annotations": [
                {
                    "height": 36,
                    "width": 46,
                    "top": 178,
                    "left": 315,
                    "class-id": "0",
                    "label-category-attributes": {
                        "occluded": "no"
                    },
                    "object-id": "480dc450-c0ca-11ea-961f-a9b1c5c97972",
                    "object-name": "car:1"
                }
            ],
            "frame-no": 0,
            "frame": "frame_0001.jpeg",
            "frame-attributes": {}
        },
        {
            "annotations": [
                {
                    "height": 30,
                    "width": 47,
                    "top": 163,
                    "left": 344,
                    "class-id": "1",
                    "label-category-attributes": {
                        "occluded": "no",
                        "size": "medium"
                    },
                    "object-id": "98f2b0b0-c0ca-11ea-961f-a9b1c5c97972",
                    "object-name": "bus:1"
                },
                {
                    "height": 28,
                    "width": 33,
                    "top": 150,
                    "left": 192,
                    "class-id": "0",
                    "label-category-attributes": {
                        "occluded": "partially"
                    },
                    "object-id": "480dc450-c0ca-11ea-961f-a9b1c5c97972",
                    "object-name": "car:1"
                }
            ],
            "frame-no": 1,
            "frame": "frame_0002.jpeg",
            "frame-attributes": {name: value, name: value}
        }
    ]
}

3D Point Cloud Semantic Segmentation Output

The following is the output manifest file from a 3D point cloud semantic segmentation labeling job.

In addition to the standard elements, the metadata for the label includes a color map that defines which color is used to label the image, the class name associated with the color, and the confidence score for each color. Additionally, there is an adjustment-status parameter in the metadata for audit workflows that is set to adjusted if the color mask is modified. If you added one or more frameAttributes to your label category configuration file, worker responses for frame attributes are in the JSON object, dataset-object-attributes.

The your-label-attribute-ref parameter contains the location of a compressed file with a .zlib extension. When you uncompress this file, it contains an array. Each index in the array corresponds to the index of an annotated point in the input point cloud. The value of the array at a given index gives the class of the point at the same index in the point cloud, based on the semantic color map found in the color-map parameter of the metadata.

You can use Python code similar to the following to decompress a .zlib file:

import zlib
from array import array

# read the label file
compressed_binary_file = open(zlib_file_path/file.zlib, 'rb').read()

# uncompress the label file
binary_content = zlib.decompress(compressed_binary_file)

# load labels to an array
my_int_array_data = array('B', binary_content);

print(my_int_array_data)

The code block above will produce an output similar to the following. Each element of the printed array contains the class of a point at the that index in the point cloud. For example, my_int_array_data[0] = 1 means point[0] in the input point cloud has a class 1. In the following output manifest file example, class 0 corresponds with "Background", 1 with Car, and 2 with Pedestrian.

>> array('B', [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2])

The following is an example of a semantic segmentation 3D point cloud labeling job output manifest file. The red, italicized text in the examples below depends on labeling job specifications and output data. 

{
   "source-ref": "s3://AWSDOC-EXAMPLE-BUCKET/examplefolder/frame1.bin",
   "source-ref-metadata":{
      "format": "binary/xyzi",
      "unix-timestamp": 1566861644.759115,
      "ego-vehicle-pose":{...}, 
      "prefix": "s3://AWSDOC-EXAMPLE-BUCKET/lidar_singleframe_dataset/prefix",
      "images": [{...}] 
    },
    "lidar-ss-label-attribute-ref": "s3://your-output-bucket/labeling-job-name/annotations/consolidated-annotation/output/dataset-object-id/filename.zlib",
    "lidar-ss-label-attribute-ref-metadata": { 
        'color-map': {
            "0": {
                "class-name": "Background",
                "hex-color": "#ffffff",
                "confidence": 0.00
            },
            "1": {
                "class-name": "Car",
                "hex-color": "#2ca02c",
                "confidence": 0.00
            },
            "2": {
                "class-name": "Pedestrian",
                "hex-color": "#1f77b4",
                "confidence": 0.00
            },
            "3": {
                "class-name": "Tree",
                "hex-color": "#ff7f0e",
                "confidence": 0.00
            }
        },
        'type': 'groundtruth/point_cloud_single_frame_semantic_segmentation', 
        'human-annotated': 'yes',
        'creation-date': '2019-11-12T01:18:14.271944',
        'job-name': 'labeling-job-name',
        //only present for adjustment audit workflow
        "adjustment-status": "adjusted", // "adjusted" means the label was adjusted
        "dataset-object-attributes": {name: value, name: value}
    }
}

3D Point Cloud Object Detection Output

The following is sample output from a 3D point cloud objected detection job. For this task type, the data about 3D cuboids is returned in the 3d-bounding-box parameter, in a list named annotations. In this list, each 3D cuboid is described using the following information.

  • Each class, or label category, that you specify in your input manifest is associated with a class-id. Use the class-map to identify the class associated with each class ID.

  • These classes are used to give each 3D cuboid an object-name in the format <class>:<integer> where integer is a unique number to identify that cuboid in the frame.

  • center-x, center-y, and center-z are the coordinates of the center of the cuboid, in the same coordinate system as the 3D point cloud input data used in your labeling job.

  • length, width, and height describe the dimensions of the cuboid.

  • yaw is used to describe the orientation (heading) of the cuboid in radians.

    Note

    yaw is now in the right-handed Cartesian system. Since this feature was added on September 02, 2022 19:02:17 UTC, you can convert the yaw measurement in the output data prior to that using the following (all units are in radians):

    old_yaw_in_output = pi - yaw

  • In our definition, +x is to the right, +y is to the forward, and +z is up from the ground plane. The rotation order is x - y - z. The roll, pitch and yaw are represented in the right-handed Cartesian system. In 3D space, roll is along the x-axis, pitch is along the y-axis and yaw is along the z-axis. All three are counterclockwise.

  • If you included label attributes in your input manifest file for a given class, a label-category-attributes parameter is included for all cuboids for which workers selected label attributes.

If one or more cuboids were modified, there is an adjustment-status parameter in the metadata for audit workflows that is set to adjusted. If you added one or more frameAttributes to your label category configuration file, worker responses for frame attributes are in the JSON object, dataset-object-attributes.

The red, italicized text in the examples below depends on labeling job specifications and output data. The ellipses (...) denote a continuation of that list, where additional objects with the same format as the proceeding object can appear.

{
   "source-ref": "s3://AWSDOC-EXAMPLE-BUCKET/examplefolder/frame1.txt",
   "source-ref-metadata":{
      "format": "text/xyzi",
      "unix-timestamp": 1566861644.759115, 
      "prefix": "s3://AWSDOC-EXAMPLE-BUCKET/lidar_singleframe_dataset/prefix",
      "ego-vehicle-pose": {
            "heading": {
                "qx": -0.02111296123795955,
                "qy": -0.006495469416730261,
                "qz": -0.008024565904865688,
                "qw": 0.9997181192298087
            },
            "position": {
                "x": -2.7161461413869947,
                "y": 116.25822288149078,
                "z": 1.8348751887989483
            }
       },
       "images": [
            {
                "fx": 847.7962624528487,
                "fy": 850.0340893791985,
                "cx": 576.2129134707038,
                "cy": 317.2423573573745,
                "k1": 0,
                "k2": 0,
                "k3": 0,
                "k4": 0,
                "p1": 0,
                "p2": 0,
                "skew": 0,
                "unix-timestamp": 1566861644.759115,
                "image-path": "images/frame_0_camera_0.jpg", 
                "position": {
                    "x": -2.2722515189268138,
                    "y": 116.86003310568965,
                    "z": 1.454614668542299
                },
                "heading": {
                    "qx": 0.7594754093069037,
                    "qy": 0.02181790885672969,
                    "qz": -0.02461725233103356,
                    "qw": -0.6496916273040025
                },
                "camera_model": "pinhole"
            }
        ]
    },
   "3d-bounding-box": 
    {
       "annotations": [
            {
                "label-category-attributes": {
                    "Occlusion": "Partial",
                    "Type": "Sedan"
                },
                "object-name": "Car:1",
                "class-id": 0,
                "center-x": -2.616382013657516,
                "center-y": 125.04149850484193,
                "center-z": 0.311272296465834,
                "length": 2.993000265181146,
                "width": 1.8355260519692056,
                "height": 1.3233490884304047,
                "roll": 0,
                "pitch": 0,
                "yaw": 1.6479308313703527
            },
            {
                "label-category-attributes": {
                    "Occlusion": "Partial",
                    "Type": "Sedan"
                },
                "object-name": "Car:2",
                "class-id": 0,
                "center-x": -5.188984560617168,
                "center-y": 99.7954483288783,
                "center-z": 0.2226435567445657,
                "length": 4,
                "width": 2,
                "height": 2,
                "roll": 0,
                "pitch": 0,
                "yaw": 1.6243170732068055
            }
        ]
    },
    "3d-bounding-box-metadata":
    {
        "objects": [], 
        "class_map": 
        {
            "0": "Car",
        },
        "type": "groundtruth/point_cloud_object_detection",
        "human-annotated": "yes", 
        "creation-date": "2018-10-18T22:18:13.527256",
        "job-name": "identify-3d-objects",
        "adjustment-status": "adjusted",
        "dataset-object-attributes": {name: value, name: value}
    }
}

3D Point Cloud Object Tracking Output

The following is an example of an output manifest file from a 3D point cloud object tracking labeling job. The red, italicized text in the examples below depends on labeling job specifications and output data. The ellipses (...) denote a continuation of that list, where additional objects with the same format as the proceeding object can appear.

In addition to the standard elements, the metadata includes a class map that lists each class that has at least one label in the sequence. If one or more cuboids were modified, there is an adjustment-status parameter in the metadata for audit workflows that is set to adjusted. 

{
   "source-ref": "s3://AWSDOC-EXAMPLE-BUCKET/myfolder/seq1.json",
    "lidar-label-attribute-ref": "s3://<CustomerOutputLocation>/<labelingJobName>/annotations/consolidated-annotation/output/<datasetObjectId>/SeqLabel.json",
    "lidar-label-attribute-ref-metadata": { 
        "objects": 
        [
            {   
                "frame-no": 300,
                "confidence": []
            },
            {
                "frame-no": 301,
                "confidence": []
            },
            ...
        ],    
        'class-map': {'0': 'Car', '1': 'Person'}, 
        'type': 'groundtruth/point_cloud_object_tracking', 
        'human-annotated': 'yes',
        'creation-date': '2019-11-12T01:18:14.271944',
        'job-name': 'identify-3d-objects',
        "adjustment-status": "adjusted" 
    }
}

In the above example, the cuboid data for each frame in seq1.json is in SeqLabel.json in the Amazon S3 location, s3://<customerOutputLocation>/<labelingJobName>/annotations/consolidated-annotation/output/<datasetObjectId>/SeqLabel.json. The following is an example of this label sequence file.

For each frame in the sequence, you see the frame-number, frame-name, if applicable, frame-attributes, and a list of annotations. This list contains 3D cubiods that were drawn for that frame. Each annotation includes the following information:

  • An object-name in the format <class>:<integer> where class identifies the label category and integer is a unique ID across the dataset.

  • When workers draw a cuboid, it is associated with a unique object-id which is associated with all cuboids that identify the same object across multiple frames.

  • Each class, or label category, that you specified in your input manifest is associated with a class-id. Use the class-map to identify the class associated with each class ID.

  • center-x, center-y, and center-z are the coordinates of the center of the cuboid, in the same coordinate system as the 3D point cloud input data used in your labeling job.

  • length, width, and height describe the dimensions of the cuboid.

  • yaw is used to describe the orientation (heading) of the cuboid in radians.

    Note

    yaw is now in the right-handed Cartesian system. Since this feature was added on September 02, 2022 19:02:17 UTC, you can convert the yaw measurement in the output data prior to that using the following (all units are in radians):

    old_yaw_in_output = pi - yaw

  • In our definition, +x is to the right, +y is to the forward, and +z is up from the ground plane. The rotation order is x - y - z. The roll, pitch and yaw are represented in the right-handed Cartesian system. In 3D space, roll is along the x-axis, pitch is along the y-axis and yaw is along the z-axis. All three are counterclockwise.

  • If you included label attributes in your input manifest file for a given class, a label-category-attributes parameter is included for all cuboids for which workers selected label attributes. 

{
    "tracking-annotations": [
        {
            "frame-number": 0,
            "frame-name": "0.txt.pcd",
            "frame-attributes": {name: value, name: value},
            "annotations": [
                {
                    "label-category-attributes": {},
                    "object-name": "Car:4",
                    "class-id": 0,
                    "center-x": -2.2906369208300674,
                    "center-y": 103.73924823843463,
                    "center-z": 0.37634114027023313,
                    "length": 4,
                    "width": 2,
                    "height": 2,
                    "roll": 0,
                    "pitch": 0,
                    "yaw": 1.5827222214406014,
                    "object-id": "ae5dc770-a782-11ea-b57d-67c51a0561a1"
                },
                {
                    "label-category-attributes": {
                        "Occlusion": "Partial",
                        "Type": "Sedan"
                    },
                    "object-name": "Car:1",
                    "class-id": 0,
                    "center-x": -2.6451293634707413,
                    "center-y": 124.9534455706848,
                    "center-z": 0.5020834081743839,
                    "length": 4,
                    "width": 2,
                    "height": 2.080488827301309,
                    "roll": 0,
                    "pitch": 0,
                    "yaw": -1.5963335581398077,
                    "object-id": "06efb020-a782-11ea-b57d-67c51a0561a1"
                },
                {
                    "label-category-attributes": {
                        "Occlusion": "Partial",
                        "Type": "Sedan"
                    },
                    "object-name": "Car:2",
                    "class-id": 0,
                    "center-x": -5.205611313118477,
                    "center-y": 99.91731932137061,
                    "center-z": 0.22917217081212138,
                    "length": 3.8747142207671956,
                    "width": 1.9999999999999918,
                    "height": 2,
                    "roll": 0,
                    "pitch": 0,
                    "yaw": 1.5672228760316775,
                    "object-id": "26fad020-a782-11ea-b57d-67c51a0561a1"
                }
            ]
        },
        {
            "frame-number": 1,
            "frame-name": "1.txt.pcd",
            "frame-attributes": {},
            "annotations": [
                {
                    "label-category-attributes": {},
                    "object-name": "Car:4",
                    "class-id": 0,
                    "center-x": -2.2906369208300674,
                    "center-y": 103.73924823843463,
                    "center-z": 0.37634114027023313,
                    "length": 4,
                    "width": 2,
                    "height": 2,
                    "roll": 0,
                    "pitch": 0,
                    "yaw": 1.5827222214406014,
                    "object-id": "ae5dc770-a782-11ea-b57d-67c51a0561a1"
                },
                {
                    "label-category-attributes": {
                        "Occlusion": "Partial",
                        "Type": "Sedan"
                    },
                    "object-name": "Car:1",
                    "class-id": 0,
                    "center-x": -2.6451293634707413,
                    "center-y": 124.9534455706848,
                    "center-z": 0.5020834081743839,
                    "length": 4,
                    "width": 2,
                    "height": 2.080488827301309,
                    "roll": 0,
                    "pitch": 0,
                    "yaw": -1.5963335581398077,
                    "object-id": "06efb020-a782-11ea-b57d-67c51a0561a1"
                },
                {
                    "label-category-attributes": {
                        "Occlusion": "Partial",
                        "Type": "Sedan"
                    },
                    "object-name": "Car:2",
                    "class-id": 0,
                    "center-x": -5.221311072916759,
                    "center-y": 100.4639841045424,
                    "center-z": 0.22917217081212138,
                    "length": 3.8747142207671956,
                    "width": 1.9999999999999918,
                    "height": 2,
                    "roll": 0,
                    "pitch": 0,
                    "yaw": 1.5672228760316775,
                    "object-id": "26fad020-a782-11ea-b57d-67c51a0561a1"
                }
            ]
        }       
    ]
}

3D-2D Object Tracking Point Cloud Object Tracking Output

The following is an example of an output manifest file from a 3D point cloud object tracking labeling job. The red, italicized text in the examples below depends on labeling job specifications and output data. The ellipses (...) denote a continuation of that list, where additional objects with the same format as the proceeding object can appear.

In addition to the standard elements, the metadata includes a class map that lists each class that has at least one label in the sequence. If one or more cuboids were modified, there is an adjustment-status parameter in the metadata for audit workflows that is set to adjusted. 

{
  "source-ref": "s3://iad-groundtruth-lidar-test-bucket/artifacts/gt-point-cloud-demos/sequences/seq2.json",
  "source-ref-metadata": {
    "json-paths": [
      "number-of-frames",
      "prefix",
      "frames{frame-no, frame}"
    ]
  },
  "3D2D-linking-ref": "s3://iad-groundtruth-lidar-test-bucket/xyz/3D2D-linking/annotations/consolidated-annotation/output/0/SeqLabel.json",
  "3D2D-linking-ref-metadata": {
    "objects": [
      {
        "frame-no": 0,
        "confidence": []
      },
      {
        "frame-no": 1,
        "confidence": []
      },
      {
        "frame-no": 2,
        "confidence": []
      },
      {
        "frame-no": 3,
        "confidence": []
      },
      {
        "frame-no": 4,
        "confidence": []
      },
      {
        "frame-no": 5,
        "confidence": []
      },
      {
        "frame-no": 6,
        "confidence": []
      },
      {
        "frame-no": 7,
        "confidence": []
      },
      {
        "frame-no": 8,
        "confidence": []
      },
      {
        "frame-no": 9,
        "confidence": []
      }
    ],
    "class-map": {
      "0": "Car"
    },
    "type": "groundtruth/point_cloud_object_tracking",
    "human-annotated": "yes",
    "creation-date": "2023-01-19T02:55:10.206508",
    "job-name": "mcm-linking"
  },
  "3D2D-linking-chain-ref": "s3://iad-groundtruth-lidar-test-bucket/xyz/3D2D-linking-chain/annotations/consolidated-annotation/output/0/SeqLabel.json",
  "3D2D-linking-chain-ref-metadata": {
    "objects": [
      {
        "frame-no": 0,
        "confidence": []
      },
      {
        "frame-no": 1,
        "confidence": []
      },
      {
        "frame-no": 2,
        "confidence": []
      },
      {
        "frame-no": 3,
        "confidence": []
      },
      {
        "frame-no": 4,
        "confidence": []
      },
      {
        "frame-no": 5,
        "confidence": []
      },
      {
        "frame-no": 6,
        "confidence": []
      },
      {
        "frame-no": 7,
        "confidence": []
      },
      {
        "frame-no": 8,
        "confidence": []
      },
      {
        "frame-no": 9,
        "confidence": []
      }
    ],
    "class-map": {
      "0": "Car"
    },
    "type": "groundtruth/point_cloud_object_tracking",
    "human-annotated": "yes",
    "creation-date": "2023-01-19T03:29:49.149935",
    "job-name": "3d2d-linking-chain"
  }
}

In the above example, the cuboid data for each frame in seq2.json is in SeqLabel.json in the Amazon S3 location, s3://<customerOutputLocation>/<labelingJobName>/annotations/consolidated-annotation/output/<datasetObjectId>/SeqLabel.json. The following is an example of this label sequence file.

For each frame in the sequence, you see the frame-number, frame-name, if applicable, frame-attributes, and a list of annotations. This list contains 3D cubiods that were drawn for that frame. Each annotation includes the following information:

  • An object-name in the format <class>:<integer> where class identifies the label category and integer is a unique ID across the dataset.

  • When workers draw a cuboid, it is associated with a unique object-id which is associated with all cuboids that identify the same object across multiple frames.

  • Each class, or label category, that you specified in your input manifest is associated with a class-id. Use the class-map to identify the class associated with each class ID.

  • center-x, center-y, and center-z are the coordinates of the center of the cuboid, in the same coordinate system as the 3D point cloud input data used in your labeling job.

  • length, width, and height describe the dimensions of the cuboid.

  • yaw is used to describe the orientation (heading) of the cuboid in radians.

    Note

    yaw is now in the right-handed Cartesian system. Since this feature was added on September 02, 2022 19:02:17 UTC, you can convert the yaw measurement in the output data prior to that using the following (all units are in radians):

    old_yaw_in_output = pi - yaw

  • In our definition, +x is to the right, +y is to the forward, and +z is up from the ground plane. The rotation order is x - y - z. The roll, pitch and yaw are represented in the right-handed Cartesian system. In 3D space, roll is along the x-axis, pitch is along the y-axis and yaw is along the z-axis. All three are counterclockwise.

  • If you included label attributes in your input manifest file for a given class, a label-category-attributes parameter is included for all cuboids for which workers selected label attributes. 

{
  "lidar": {
    "tracking-annotations": [
      {
        "frame-number": 0,
        "frame-name": "0.txt.pcd",
        "annotations": [
          {
            "label-category-attributes": {
              "Type": "Sedan"
            },
            "object-name": "Car:1",
            "class-id": 0,
            "center-x": 12.172361721602815,
            "center-y": 120.23067521992364,
            "center-z": 1.590525771183712,
            "length": 4,
            "width": 2,
            "height": 2,
            "roll": 0,
            "pitch": 0,
            "yaw": 0,
            "object-id": "505b39e0-97a4-11ed-8903-dd5b8b903715"
          },
          {
            "label-category-attributes": {},
            "object-name": "Car:4",
            "class-id": 0,
            "center-x": 17.192725195301094,
            "center-y": 114.55705365827872,
            "center-z": 1.590525771183712,
            "length": 4,
            "width": 2,
            "height": 2,
            "roll": 0,
            "pitch": 0,
            "yaw": 0,
            "object-id": "1afcb670-97a9-11ed-9a84-ff627d099e16"
          }
        ],
        "frame-attributes": {}
      },
      {
        "frame-number": 1,
        "frame-name": "1.txt.pcd",
        "annotations": [
          {
            "label-category-attributes": {
              "Type": "Sedan"
            },
            "object-name": "Car:1",
            "class-id": 0,
            "center-x": -1.6841480600695489,
            "center-y": 126.20198882749516,
            "center-z": 1.590525771183712,
            "length": 4,
            "width": 2,
            "height": 2,
            "roll": 0,
            "pitch": 0,
            "yaw": 0,
            "object-id": "505b39e0-97a4-11ed-8903-dd5b8b903715"
          },
          {
            "label-category-attributes": {},
            "object-name": "Car:4",
            "class-id": 0,
            "center-x": 17.192725195301094,
            "center-y": 114.55705365827872,
            "center-z": 1.590525771183712,
            "length": 4,
            "width": 2,
            "height": 2,
            "roll": 0,
            "pitch": 0,
            "yaw": 0,
            "object-id": "1afcb670-97a9-11ed-9a84-ff627d099e16"
          }
        ],
        "frame-attributes": {}
      },
      {
        "frame-number": 2,
        "frame-name": "2.txt.pcd",
        "annotations": [
          {
            "label-category-attributes": {
              "Type": "Sedan"
            },
            "object-name": "Car:1",
            "class-id": 0,
            "center-x": -1.6841480600695489,
            "center-y": 126.20198882749516,
            "center-z": 1.590525771183712,
            "length": 4,
            "width": 2,
            "height": 2,
            "roll": 0,
            "pitch": 0,
            "yaw": 0,
            "object-id": "505b39e0-97a4-11ed-8903-dd5b8b903715"
          },
          {
            "label-category-attributes": {},
            "object-name": "Car:4",
            "class-id": 0,
            "center-x": 17.192725195301094,
            "center-y": 114.55705365827872,
            "center-z": 1.590525771183712,
            "length": 4,
            "width": 2,
            "height": 2,
            "roll": 0,
            "pitch": 0,
            "yaw": 0,
            "object-id": "1afcb670-97a9-11ed-9a84-ff627d099e16"
          }
        ],
        "frame-attributes": {}
      }
    ]
  },
  "camera-0": {
    "tracking-annotations": [
      {
        "frame-no": 0,
        "frame": "0.txt.pcd",
        "annotations": [
          {
            "label-category-attributes": {
              "Occlusion": "Partial"
            },
            "object-name": "Car:2",
            "class-id": 0,
            "width": 223,
            "height": 164,
            "top": 225,
            "left": 486,
            "object-id": "5229df60-97a4-11ed-8903-dd5b8b903715"
          }
        ],
        "frame-attributes": {}
      },
      {
        "frame-no": 1,
        "frame": "1.txt.pcd",
        "annotations": [
          {
            "label-category-attributes": {},
            "object-name": "Car:4",
            "class-id": 0,
            "width": 252,
            "height": 246,
            "top": 237,
            "left": 473,
            "object-id": "1afcb670-97a9-11ed-9a84-ff627d099e16"
          }
        ],
        "frame-attributes": {}
      }
    ]
  }
}

The cuboid and bounding box for an object are linked through a common object-id.