Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Monitor Labeling Job Status

To monitor the status of your labeling jobs, you can set up an Amazon CloudWatch Events (CloudWatch Events) rule for Amazon SageMaker Ground Truth (Ground Truth) to send an event to CloudWatch Events when a labeling job status changes to Completed, Failed, or Stopped or when a worker accepts, declines, submits, or returns a task.

Once you create a rule, you can add a target to it. CloudWatch Events uses this target to invoke another Amazon service to process the event. For example, you can create a target using a Amazon Simple Notification Service (Amazon SNS) topic to send a notification to your email when a labeling job status changes.

Prerequisites:

To create a CloudWatch Events rule, you will need an Amazon Identity and Access Management (IAM) role with an events.amazonaws.com trust policy attached. The following is an example of an events.amazonaws.com trust policy.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": [
          "events.amazonaws.com"
        ]
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Send Events to CloudWatch Events

To configure a CloudWatch Events rule to get status updates, or events, for your Ground Truth labeling jobs, use the Amazon Command Line Interface (Amazon CLI) put-rule command. You can filter events that are sent to your rule by status change. For example, you can create a rule that notifies you only if a labeling job status changes to Completed. When using the put-rule command, specify the following to receive labeling job statuses:

To configure a CloudWatch Events rule to watch for all status changes, use the following command and replace the placeholder text. For example, replace "GTLabelingJobStateChanges" with a unique CloudWatch Events rule name and "arn:aws:iam::111122223333:role/MyRoleForThisRule" with the Amazon Resource Number (ARN) of an IAM role with an events.amazonaws.com trust policy attached.

aws events put-rule --name "GTLabelingJobStateChanges" 
    --event-pattern "{\"source\":[\"aws.sagemaker\"],\"detail-type\":[\"SageMaker Ground Truth Labeling Job State Change\"]}" 
    --role-arn "arn:aws:iam::111122223333:role/MyRoleForThisRule" 
    --region "region"

To filter by job status, use the \"detail\":{\"LabelingJobStatus\":[\"Status\"]}}" syntax. Valid values for Status are Completed, Failed, and Stopped.

The following example creates a CloudWatch Events rule that notifies you when a labeling job in us-west-2 (Oregon) changes to Completed.

aws events put-rule --name "LabelingJobCompleted" 
    --event-pattern "{\"source\":[\"aws.sagemaker\"],\"detail-type\":[\"SageMaker Ground Truth Labeling Job State Change\"], \"detail\":{\"LabelingJobStatus\":[\"Completed\"]}}"  
    --role-arn "arn:aws:iam::111122223333:role/MyRoleForThisRule" 
    --region us-west-2

The following example creates a CloudWatch Events rule that notifies you when a labeling job in us-east-1 (Virginia) changes to Completed or Failed.

aws events put-rule --name "LabelingJobCompletedOrFailed" 
    --event-pattern "{\"source\":[\"aws.sagemaker\"],\"detail-type\":[\"SageMaker Ground Truth Labeling Job State Change\"], \"detail\":{\"LabelingJobStatus\":[\"Completed\", \"Failed\"]}}"  
    --role-arn "arn:aws:iam::111122223333:role/MyRoleForThisRule" 
    --region us-east-1

To learn more about the put-rule request, see Event Patterns in CloudWatch Events in the Amazon CloudWatch Events User Guide.

Set Up a Target to Process Events

After you have created a rule, events similar to the following are sent to CloudWatch Events. In this example, the labeling job test-labeling-job's status changed to Completed.

{
    "version": "0",
    "id": "111e1111-11d1-111f-b111-1111b11dcb11",
    "detail-type": "SageMaker Ground Truth Labeling Job State Change",
    "source": "aws.sagemaker",
    "account": "111122223333",
    "time": "2018-10-06T12:26:13Z",
    "region": "us-east-1",
    "resources": [
        "arn:aws:sagemaker:us-east-1:111122223333:labeling-job/test-labeling-job"
    ],
    "detail": {      
        "LabelingJobStatus": "Completed"
    }
}

To process events, you need to set up a target. For example, if you want to receive an email when your labeling job status changes, use a procedure in Setting Up Amazon SNS Notifications in the Amazon CloudWatch User Guide to set up an Amazon SNS topic and subscribe your email to it. Once you have create a topic, you can use it to create a target.

Labeling Job Expiration

If your labeling job is not completed after 30 days, it will expire. If your labeling job expires, you can chain the job to create a new labeling job that will only send unlabeled data to workers. For more information, and to learn how to create a labeling job using chaining, see Chaining Labeling Jobs.

Declining Tasks

Workers are able to decline tasks.

Workers decline a task if the instructions are not clear, input data is not displaying correctly, or if they encounter some other issue with the task. If the number of workers per dataset object (NumberOfHumanWorkersPerDataObject) decline the task, the data object is marked as expired and will not be sent to additional workers.