# Model deployment at the edge with SageMaker Edge Manager
**Warning**
SageMaker Edge Manager is being discontinued on April 26th, 2024. For more information about continuing to deploy your models to edge devices, see [SageMaker Edge Manager end of life](edge-eol.md).
Amazon SageMaker Edge Manager provides model management for edge devices so you can optimize, secure, monitor, and maintain machine learning models on fleets of edge devices such as smart cameras, robots, personal computers, and mobile devices.
## Why Use Edge Manager?
Many machine learning (ML) use cases require running ML models on a fleet of edge devices, which allows you to get predictions in real-time, preserves the privacy of the end users, and lowers the cost of network connectivity. With the increasing availability of low-power edge hardware designed for ML, it is now possible to run multiple complex neural network models on edge devices.
However, operating ML models on edge devices is challenging, because devices, unlike cloud instances, have limited compute, memory, and connectivity. After the model is deployed, you need to continuously monitor the models, because model drift can cause the quality of model to decay overtime. Monitoring models across your device fleets is difficult because you need to write custom code to collect data samples from your device and recognize skew in predictions. In addition, models are often hard-coded into the application. To update the model, you must rebuild and update the entire application or device firmware, which can disrupt your operations.
With SageMaker Edge Manager, you can optimize, run, monitor, and update machine learning models across fleets of devices at the edge.
## How Does it Work?
At a high level, there are five main components in the SageMaker Edge Manager workflow: compiling models with SageMaker Neo, packaging Neo-compiled models, deploying models to your devices, running models on the SageMaker AI inference engine (Edge Manager agent), and maintaining models on the devices.
![\[The five main components in the SageMaker Edge Manager workflow.\]](http://docs.amazonaws.cn/en_us/sagemaker/latest/dg/images/smith/smith_overview.png)
SageMaker Edge Manager uses SageMaker Neo to optimize your models for the target hardware in one click, then to cryptographically sign your models before deployment. Using SageMaker Edge Manager, you can sample model input and output data from edge devices and send it to the cloud for monitoring and analysis, and view a dashboard that tracks and visually reports on the operation of the deployed models within the SageMaker AI console.
SageMaker Edge Manager extends capabilities that were previously only available in the cloud to the edge, so developers can continuously improve model quality by using Amazon SageMaker Model Monitor for drift detection, then relabel the data with SageMaker AI Ground Truth and retrain the models in SageMaker AI.
## How Do I Use SageMaker Edge Manager?
If you are a first time user of SageMaker Edge Manager, we recommend that you do the following:
1. **Read the [Getting Started](https://docs.amazonaws.cn/sagemaker/latest/dg/edge-manager-getting-started.html) section** - This section walks you through setting up your first edge packaging job and creating your first fleet.
1. **Explore Edge Manager Jupyter notebook examples **- Example notebooks are stored in the [amazon-sagemaker-examples](https://github.com/aws/amazon-sagemaker-examples) GitHub repository in the [sagemaker\$1edge\$1manager](https://github.com/aws/amazon-sagemaker-examples/tree/master/sagemaker_edge_manager) folder.
# First Steps with Amazon SageMaker AI Edge Manager
This guide demonstrates how to complete the necessary steps to register, deploy, and manage a fleet of devices, and how to satisfy Amazon SageMaker AI Edge Manager prerequisites.
**Topics**
+ [Setting Up](edge-getting-started-step1.md)
+ [Prepare Your Model for Deployment](edge-getting-started-step2.md)
+ [Register and Authenticate Your Device Fleet](edge-getting-started-step3.md)
+ [Download and Set Up Edge Manager](edge-getting-started-step4.md)
+ [Run Agent](edge-getting-started-step5.md)
# Setting Up
Before you begin using SageMaker Edge Manager to manage models on your device fleets, you must first create IAM Roles for both SageMaker AI and Amazon IoT. You will also want to create at least one Amazon S3 bucket where you will store your pre-trained model, the output of your SageMaker Neo compilation job, as well as input data from your edge devices.
## Sign up for an Amazon Web Services account
If you do not have an Amazon Web Services account, use the following procedure to create one.
**To sign up for Amazon Web Services**
1. Open [http://www.amazonaws.cn/](http://www.amazonaws.cn/) and choose **Sign Up**.
1. Follow the on-screen instructions.
Amazon sends you a confirmation email after the sign-up process is complete. At any time, you can view your current account activity and manage your account by going to [http://www.amazonaws.cn/](http://www.amazonaws.cn/) and choosing **My Account**.
## Secure IAM users
After you sign up for an Amazon Web Services account, safeguard your administrative user by turning on multi-factor authentication (MFA). For instructions, see [Enable a virtual MFA device for an IAM user (console)](https://docs.amazonaws.cn/IAM/latest/UserGuide/id_credentials_mfa_enable_virtual.html#enable-virt-mfa-for-iam-user) in the *IAM User Guide*.
To give other users access to your Amazon Web Services account resources, create IAM users. To secure your IAM users, turn on MFA and only give the IAM users the permissions needed to perform their tasks.
For more information about creating and securing IAM users, see the following topics in the *IAM User Guide*:
+ [Creating an IAM user in your Amazon Web Services account](https://docs.amazonaws.cn//IAM/latest/UserGuide/id_users_create.html)
+ [Access management for Amazon resources](https://docs.amazonaws.cn/IAM/latest/UserGuide/access.html)
+ [Example IAM identity-based policies](https://docs.amazonaws.cn/IAM/latest/UserGuide/access_policies_examples.html)
## Create roles and storage
SageMaker Edge Manager needs access to your Amazon S3 bucket URI. To facilitate this, create an IAM role that can run SageMaker AI and has permission to access Amazon S3. Using this role, SageMaker AI can run under your account and access to your Amazon S3 bucket.
You can create an IAM role by using the IAM console, Amazon SDK for Python (Boto3), or Amazon CLI. The following is an example of how to create an IAM role, attach the necessary policies with the IAM console, and create an Amazon S3 bucket.
1. **Create an IAM role for Amazon SageMaker AI.**
1. Sign in to the Amazon Web Services Management Console and open the IAM console at [https://console.amazonaws.cn/iam/](https://console.amazonaws.cn/iam/).
1. In the navigation pane of the IAM console, choose **Roles**, and then choose **Create role**.
1. For **Select type of trusted entity**, choose **Amazon service**.
1. Choose the service that you want to allow to assume this role. In this case, choose **SageMaker AI**. Then choose **Next: Permissions**.
+ This automatically creates an IAM policy that grants access to related services such as Amazon S3, Amazon ECR, and CloudWatch Logs.
1. Choose **Next: Tags**.
1. (Optional) Add metadata to the role by attaching tags as key–value pairs. For more information about using tags in IAM, see [Tagging IAM resources](https://docs.amazonaws.cn//IAM/latest/UserGuide/id_tags.html).
1. Choose **Next: Review**.
1. Type in a **Role name**.
1. If possible, type a role name or role name suffix. Role names must be unique within your Amazon account. They are not distinguished by case. For example, you cannot create roles named both `PRODROLE` and `prodrole`. Because other Amazon resources might reference the role, you cannot edit the name of the role after it has been created.
1. (Optional) For **Role description**, type a description for the new role.
1. Review the role and then choose **Create role**.
Note the SageMaker AI Role ARN, which you use to create a compilation job with SageMaker Neo and a packaging job with Edge Manager. To find out the role ARN using the console, do the following:
1. Go to the IAMconsole: [https://console.amazonaws.cn/iam/](https://console.amazonaws.cn/iam/)
1. Select **Roles**.
1. Search for the role you just created by typing in the name of the role in the search field.
1. Select the role.
1. The role ARN is at the top of the **Summary** page.
1. **Create an IAM role for Amazon IoT.**
The Amazon IoT IAM role you create is used to authorize your thing objects. You also use the IAM role ARN to create and register device fleets with a SageMaker AI client object.
Configure an IAM role in your Amazon account for the credentials provider to assume on behalf of the devices in your device fleet. Then, attach a policy to authorize your devices to interact with Amazon IoT services.
Create a role for Amazon IoT either programmatically or with the IAM console, similar to what you did when you created a role for SageMaker AI.
1. Sign in to the Amazon Web Services Management Console and open the IAM console at [https://console.amazonaws.cn/iam/](https://console.amazonaws.cn/iam/).
1. In the navigation pane of the IAM console, choose **Roles**, and then choose **Create role**.
1. For **Select type of trusted entity**, choose **Amazon service**.
1. Choose the service that you want to allow to assume this role. In this case, choose **IoT**. Select **IoT** as the **Use Case**.
1. Choose **Next: Permissions**.
1. Choose **Next: Tags**.
1. (Optional) Add metadata to the role by attaching tags as key–value pairs. For more information about using tags in IAM, see [Tagging IAM resources](https://docs.amazonaws.cn//IAM/latest/UserGuide/id_tags.html).
1. Choose **Next: Review**.
1. Type in a **Role name**. The role name must start with `SageMaker AI`.
1. (Optional) For **Role description**, type a description for the new role.
1. Review the role and then choose **Create role**.
1. Once the role is created, choose **Roles** in the IAM console. Search for the role you created by typing in role name in the **Search** field.
1. Choose your role.
1. Next, choose **Attach Policies**.
1. Search for `AmazonSageMakerEdgeDeviceFleetPolicy` in the **Search** field. Select `AmazonSageMakerEdgeDeviceFleetPolicy`.
1. Choose **Attach policy**.
1. Add the following policy statement to the trust relationship:
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {"Service": "credentials.iot.amazonaws.com"},
"Action": "sts:AssumeRole"
},
{
"Effect": "Allow",
"Principal": {"Service": "sagemaker.amazonaws.com"},
"Action": "sts:AssumeRole"
}
]
}
```
------
A trust policy is a [JSON policy document](https://docs.amazonaws.cn//IAM/latest/UserGuide/reference_policies_grammar) in which you define the principals that you trust to assume the role. For more information about trust policies, see [Roles terms and concepts](https://docs.amazonaws.cn/IAM/latest/UserGuide/id_roles_terms-and-concepts.html).
1. Note the Amazon IoT role ARN. You use the Amazon IoT Role ARN to create and register the device fleet. To find the IAM role ARN with the console:
1. Go to the IAM console: [https://console.amazonaws.cn/iam/](https://console.amazonaws.cn/iam/)
1. Choose **Roles**.
1. Search for the role you created by typing in the name of the role in the **Search** field.
1. Select the role.
1. The role ARN is on the Summary page.
1. **Create an Amazon S3 bucket.**
SageMaker Neo and Edge Manager access your pre-compiled model and compiled model from an Amazon S3 bucket. Edge Manager also stores sample data from your device fleet in Amazon S3.
1. Open the Amazon S3 console at [https://console.amazonaws.cn/s3/](https://console.amazonaws.cn/s3/).
1. Choose **Create bucket**.
1. In **Bucket name**, enter a name for your bucket.
1. In **Region**, choose the Amazon Region where you want the bucket to reside.
1. In **Bucket settings for Block Public Access**, choose the settings that you want to apply to the bucket.
1. Choose **Create bucket**.
For more information about creating Amazon S3 buckets, see [Getting started with Amazon S3](https://docs.amazonaws.cn/AmazonS3/latest/userguide/GetStartedWithS3.html).
# Prepare Your Model for Deployment
In this section you will create SageMaker AI and Amazon IoT client objects, download a pre-trained machine learning model, upload your model to your Amazon S3 bucket, compile your model for your target device with SageMaker Neo, and package your model so that it can be deployed with the Edge Manager agent.
1. **Import libraries and create client objects.**
This tutorial uses the Amazon SDK for Python (Boto3) to create clients to interact with SageMaker AI, Amazon S3, and Amazon IoT.
Import Boto3, specify your Region, and initialize the client objects you need as shown in the following example:
```
import boto3
import json
import time
AWS_REGION = 'us-west-2'# Specify your Region
bucket = 'bucket-name'
sagemaker_client = boto3.client('sagemaker', region_name=AWS_REGION)
iot_client = boto3.client('iot', region_name=AWS_REGION)
```
Define variables and assign them the role ARN you created for SageMaker AI and Amazon IoT as strings:
```
# Replace with the role ARN you created for SageMaker
sagemaker_role_arn = "arn:aws:iam:::role/*"
# Replace with the role ARN you created for Amazon IoT.
# Note: The name must start with 'SageMaker'
iot_role_arn = "arn:aws:iam:::role/SageMaker*"
```
1. **Train a machine learning model.**
See [Train a Model with Amazon SageMaker](https://docs.amazonaws.cn/sagemaker/latest/dg/how-it-works-training.html) for more information on how to train a machine learning model using SageMaker AI. You can optionally upload your locally trained model directly into an Amazon S3 URI bucket.
If you do not have a model yet, you can use a pre-trained model for the next steps in this tutorial. For example, you can save the MobileNet V2 models from the TensorFlow framework. MobileNet V2 is an image classification model optimized for mobile applications. For more information about MobileNet V2, see the [MobileNet GitHub README](https://github.com/tensorflow/models/tree/master/research/slim/nets/mobilenet).
Type the following into your Jupyter Notebook to save the pre-trained MobileNet V2 model:
```
# Save the MobileNet V2 model to local storage
import tensorflow as tf
model = tf.keras.applications.MobileNetV2()
model.save(“mobilenet_v2.h5”)
```
**Note**
If you do not have TensorFlow installed, you can do so by running `pip install tensorflow=2.4`
Use TensorFlow version 2.4 or lower for this tutorial.
The model will be saved into the `mobilenet_v2.h5` file. Before packaging the model, you will need to first compile your model using SageMaker Neo. See [Supported Frameworks, Devices, Systems, and Architectures](neo-supported-devices-edge.md) to check if your version of TensorFlow (or other framework of choice) is currently supported by SageMaker Neo.
SageMaker Neo requires models to be stored as a compressed TAR file. Repackage it as a compressed TAR file (\$1.tar.gz):
```
# Package MobileNet V2 model into a TAR file
import tarfile
tarfile_name='mobilenet-v2.tar.gz'
with tarfile.open(tarfile_name, mode='w:gz') as archive:
archive.add('mobilenet-v2.h5')
```
1. **Upload your model to Amazon S3.**
Once you have a machine learning model, store it in an Amazon S3 bucket. The following example uses an Amazon CLI command to upload the model the to the Amazon S3 bucket you created earlier in a directory called *models*. Type in the following into your Jupyter Notebook:
```
!aws s3 cp mobilenet-v2.tar.gz s3://{bucket}/models/
```
1. **Compile your model with SageMaker Neo.**
Compile your machine learning model with SageMaker Neo for an edge device. You need to know your Amazon S3 bucket URI where you stored the trained model, the machine learning framework you used to train your model, the shape of your model’s input, and your target device.
For the MobileNet V2 model, use the following:
```
framework = 'tensorflow'
target_device = 'jetson_nano'
data_shape = '{"data":[1,3,224,224]}'
```
SageMaker Neo requires a specific model input shape and model format based on the deep learning framework you use. For more information about how to save your model, see [What input data shapes does SageMaker Neo expect?](neo-compilation-preparing-model.md#neo-job-compilation-expected-inputs). For more information about devices and frameworks supported by Neo, see [Supported Frameworks, Devices, Systems, and Architectures](neo-supported-devices-edge.md).
Use the `CreateCompilationJob` API to create a compilation job with SageMaker Neo. Provide a name to the compilation job, the SageMaker AI Role ARN, the Amazon S3 URI where your model is stored, the input shape of the model, the name of the framework, the Amazon S3 URI where you want SageMaker AI to store your compiled model, and your edge device target.
```
# Specify the path where your model is stored
model_directory = 'models'
s3_model_uri = 's3://{}/{}/{}'.format(bucket, model_directory, tarfile_name)
# Store compiled model in S3 within the 'compiled-models' directory
compilation_output_dir = 'compiled-models'
s3_output_location = 's3://{}/{}/'.format(bucket, compilation_output_dir)
# Give your compilation job a name
compilation_job_name = 'getting-started-demo'
sagemaker_client.create_compilation_job(CompilationJobName=compilation_job_name,
RoleArn=sagemaker_role_arn,
InputConfig={
'S3Uri': s3_model_uri,
'DataInputConfig': data_shape,
'Framework' : framework.upper()},
OutputConfig={
'S3OutputLocation': s3_output_location,
'TargetDevice': target_device},
StoppingCondition={'MaxRuntimeInSeconds': 900})
```
1. **Package your compiled model.**
Packaging jobs take SageMaker Neo–compiled models and make any changes necessary to deploy the model with the inference engine, Edge Manager agent. To package your model, create an edge packaging job with the `create_edge_packaging` API or the SageMaker AI console.
You need to provide the name that you used for your Neo compilation job, a name for the packaging job, a role ARN (see [Setting Up](edge-getting-started-step1.md) section), a name for the model, a model version, and the Amazon S3 bucket URI for the output of the packaging job. Note that Edge Manager packaging job names are case-sensitive. The following is an example of how to create a packaging job using the API.
```
edge_packaging_name='edge-packaging-demo'
model_name="sample-model"
model_version="1.1"
```
Define the Amazon S3 URI where you want to store the packaged model.
```
# Output directory where you want to store the output of the packaging job
packaging_output_dir = 'packaged_models'
packaging_s3_output = 's3://{}/{}'.format(bucket, packaging_output_dir)
```
Use `CreateEdgePackagingJob` to package your Neo-compiled model. Provide a name for your edge packaging job and the name you provided for your compilation job (in this example, it was stored in the variable `compilation_job_name`). Also provide a name for your model, a version for your model (this is used to help you keep track of what model version you are using), and the S3 URI where you want SageMaker AI to store the packaged model.
```
sagemaker_client.create_edge_packaging_job(
EdgePackagingJobName=edge_packaging_name,
CompilationJobName=compilation_job_name,
RoleArn=sagemaker_role_arn,
ModelName=model_name,
ModelVersion=model_version,
OutputConfig={
"S3OutputLocation": packaging_s3_output
}
)
```
# Register and Authenticate Your Device Fleet
In this section you will create your Amazon IoT thing object, create a device fleet, register your device fleet so it can interact with the cloud, create X.509 certificates to authenticate your devices to Amazon IoT Core, associate the role alias with Amazon IoT that was generated when you created your fleet, get your Amazon account-specific endpoint for the credentials provider, get an official Amazon Root CA file, and upload the Amazon CA file to Amazon S3.
1. **Create Amazon IoT things.**
SageMaker Edge Manager takes advantage of the Amazon IoT Core services to facilitate the connection between the edge devices and endpoints in the Amazon cloud. You can take advantage of existing Amazon IoT functionality after you set up your devices to work with Edge Manager.
To connect your device to Amazon IoT, you need to create Amazon IoT *thing objects*, create and register a client certificate with Amazon IoT, and create and configure the IAM role for your devices.
First, create Amazon IoT thing objects with the Amazon IoT client (`iot_client`) you created earlier with Boto3. The following example shows how to create two thing objects:
```
iot_thing_name = 'sample-device'
iot_thing_type = 'getting-started-demo'
iot_client.create_thing_type(
thingTypeName=iot_thing_type
)
# Create an Amazon IoT thing objects
iot_client.create_thing(
thingName=iot_thing_name,
thingTypeName=iot_thing_type
)
```
1. **Create your device fleet.**
Create a device fleet with the SageMaker AI client object defined in a previous step. You can also use the SageMaker AI console to create a device fleet.
```
import time
device_fleet_name="demo-device-fleet" + str(time.time()).split('.')[0]
device_name="sagemaker-edge-demo-device" + str(time.time()).split('.')[0]
```
Specify your IoT role ARN. This lets Amazon IoT grant temporary credentials to devices.
```
device_model_directory='device_output'
s3_device_fleet_output = 's3://{}/{}'.format(bucket, device_model_directory)
sagemaker_client.create_device_fleet(
DeviceFleetName=device_fleet_name,
RoleArn=iot_role_arn, # IoT Role ARN specified in previous step
OutputConfig={
'S3OutputLocation': s3_device_fleet_output
}
)
```
An Amazon IoT role alias is created when you create a device fleet. This role alias is associated with Amazon IoT using the `iot_client` object in a later step.
1. **Register your device fleet.**
To interact with the cloud, you need to register your device with SageMaker Edge Manager. In this example, you register a single device with the fleet you created. To register the device, you need to provide a device name and the Amazon IoT thing name as shown in the following example:
```
# Device name should be 36 characters
device_name = "sagemaker-edge-demo-device" + str(time.time()).split('.')[0]
sagemaker_client.register_devices(
DeviceFleetName=device_fleet_name,
Devices=[
{
"DeviceName": device_name,
"IotThingName": iot_thing_name
}
]
)
```
1. **Create X.509 certificates.**
After creating the Amazon IoT thing object, you must create a X.509 device certificate for your thing object. This certificate authenticates your device to Amazon IoT Core.
Use the following to create a private key, public key, and a X.509 certificate file using the Amazon IoT client defined (`iot_client`) earlier.
```
# Creates a 2048-bit RSA key pair and issues an X.509 # certificate
# using the issued public key.
create_cert = iot_client.create_keys_and_certificate(
setAsActive=True
)
# Get certificate from dictionary object and save in its own
with open('./device.pem.crt', 'w') as f:
for line in create_cert['certificatePem'].split('\n'):
f.write(line)
f.write('\n')
# Get private key from dictionary object and save in its own
with open('./private.pem.key', 'w') as f:
for line in create_cert['keyPair']['PrivateKey'].split('\n'):
f.write(line)
f.write('\n')
# Get a private key from dictionary object and save in its own
with open('./public.pem.key', 'w') as f:
for line in create_cert['keyPair']['PublicKey'].split('\n'):
f.write(line)
f.write('\n')
```
1. **Associate the role alias with Amazon IoT.**
When you create a device fleet with SageMaker AI (`sagemaker_client.create_device_fleet()`), a role alias is generated for you. An Amazon IoT role alias provides a mechanism for connected devices to authenticate to Amazon IoT using X.509 certificates, and then obtain short-lived Amazon credentials from an IAM role that is associated with an Amazon IoT role alias. The role alias allows you to change the role of the device without having to update the device. Use `DescribeDeviceFleet` to get the role alias name and ARN.
```
# Print Amazon Resource Name (ARN) and alias that has access
# to Amazon Internet of Things (IoT).
sagemaker_client.describe_device_fleet(DeviceFleetName=device_fleet_name)
# Store iot role alias string in a variable
# Grabs role ARN
full_role_alias_name = sagemaker_client.describe_device_fleet(DeviceFleetName=device_fleet_name)['IotRoleAlias']
start_index = full_role_alias_name.find('SageMaker') # Find beginning of role name
role_alias_name = full_role_alias_name[start_index:]
```
Use the `iot_client` to facilitate associating the role alias generated from creating the device fleet with Amazon IoT:
```
role_alias = iot_client.describe_role_alias(
roleAlias=role_alias_name)
```
For more information about IAM role alias, see [Role alias allows access to unused services](https://docs.amazonaws.cn/iot/latest/developerguide/audit-chk-role-alias-unused-svcs.html) .
You created and registered a certificate with Amazon IoT earlier for successful authentication of your device. Now, you need to create and attach a policy to the certificate to authorize the request for the security token.
```
alias_policy = {
"Version": "2012-10-17",
"Statement": {
"Effect": "Allow",
"Action": "iot:AssumeRoleWithCertificate",
"Resource": role_alias['roleAliasDescription']['roleAliasArn']
}
}
policy_name = 'aliaspolicy-'+ str(time.time()).split('.')[0]
aliaspolicy = iot_client.create_policy(policyName=policy_name,
policyDocument=json.dumps(alias_policy))
# Attach policy
iot_client.attach_policy(policyName=policy_name,
target=create_cert['certificateArn'])
```
1. **Get your Amazon account-specific endpoint for the credentials provider.**
Edge devices need an endpoint in order to assume credentials. Obtain your Amazon account-specific endpoint for the credentials provider.
```
# Get the unique endpoint specific to your Amazon account that is making the call.
iot_endpoint = iot_client.describe_endpoint(
endpointType='iot:CredentialProvider'
)
endpoint="https://{}/role-aliases/{}/credentials".format(iot_endpoint['endpointAddress'],role_alias_name)
```
1. **Get the official Amazon root CA file and upload it to the Amazon S3 bucket.**
Use the following in your Jupyter Notebook or Amazon CLI (if you use your terminal, remove the ‘\$1’ magic function):
```
!wget https://www.amazontrust.com/repository/AmazonRootCA1.pem
```
Use the endpoint to make an HTTPS request to the credentials provider to return a security token. The following example command uses `curl`, but you can use any HTTP client.
```
!curl --cert device.pem.crt --key private.pem.key --cacert AmazonRootCA1.pem $endpoint
```
If the certificate is verified, upload the keys and certificate to your Amazon S3 bucket URI:
```
!aws s3 cp private.pem.key s3://{bucket}/authorization-files/
!aws s3 cp device.pem.crt s3://{bucket}/authorization-files/
!aws s3 cp AmazonRootCA1.pem s3://{bucket}/authorization-files/
```
Clean your working directory by moving your keys and certificate to a different directory:
```
# Optional - Clean up working directory
!mkdir authorization-files
!mv private.pem.key device.pem.crt AmazonRootCA1.pem authorization-files/
```
# Download and Set Up Edge Manager
The Edge Manager agent is an inference engine for your edge devices. Use the agent to make predictions with models loaded onto your edge devices. The agent also collects model metrics and captures data at specific intervals.
In this section you will set up your device with the agent. To do so, first copy a release artifact and signing root certificate from the release bucket locally to your machine. After you unzip the release artifact, upload it to Amazon S3. Next, define and save a configuration file for the agent. A template is provided for you to copy and paste. Finally, copy the release artifacts, configuration file, and credentials to your device.
1. **Download the SageMaker Edge Manager agent.**
The agent is released in binary format for supported operating systems. This example runs inference on a Jetson Nano which uses a Linux operating system and has an ARM64 architecture. For more information about what operating system and architecture supported devices use, see [Supported Devices, Chip Architectures, and Systems](neo-supported-devices-edge-devices.md).
Fetch the latest version of binaries from the SageMaker Edge Manager release bucket from the us-west-2 Region.
```
!aws s3 ls s3://sagemaker-edge-release-store-us-west-2-linux-armv8/Releases/ | sort -r
```
This returns release artifacts sorted by their version.
```
PRE 1.20210512.96da6cc/
PRE 1.20210305.a4bc999/
PRE 1.20201218.81f481f/
PRE 1.20201207.02d0e97/
```
The version has the following format: `..`. It consists of three components:
+ ``: The release version. The release version is currently set to `1`.
+ ``: The time stamp of the artifact release.
+ : The repository commit ID from which the release is built.
Copy the zipped TAR file locally or to your device directly. The following example shows how to copy the latest release artifact at the time this document was released.
```
!aws s3 cp s3://sagemaker-edge-release-store-us-west-2-linux-x64/Releases/1.20201218.81f481f/1.20201218.81f481f.tgz ./
```
Once you have the artifact, unzip the zipped TAR file. The following unzips the TAR file and stores it in a directory called `agent_demo`:
```
!mkdir agent_demo
!tar -xvzf 1.20201218.81f481f.tgz -C ./agent_demo
```
Upload the agent release artifacts to your Amazon S3 bucket. The following code example copies the content within `agent_demo` and uploads it to a directory within your Amazon S3 bucket called `agent_demo`:
```
!aws s3 cp --recursive ./agent_demo s3://{bucket}/agent_demo
```
You also need the signing root certificates from the release bucket:
```
!aws s3 cp s3://sagemaker-edge-release-store-us-west-2-linux-x64/Certificates/us-west-2/us-west-2.pem ./
```
Upload the signing root certificate to your Amazon S3 bucket:
```
!aws s3 cp us-west-2.pem s3://{bucket}/authorization-files/
```
1. **Define a SageMaker Edge Manager agent configuration file.**
First, define the agent configuration file as follows:
```
sagemaker_edge_config = {
"sagemaker_edge_core_device_name": "device_name",
"sagemaker_edge_core_device_fleet_name": "device_fleet_name",
"sagemaker_edge_core_capture_data_buffer_size": 30,
"sagemaker_edge_core_capture_data_push_period_seconds": 4,
"sagemaker_edge_core_folder_prefix": "demo_capture",
"sagemaker_edge_core_region": "us-west-2",
"sagemaker_edge_core_root_certs_path": "/agent_demo/certificates",
"sagemaker_edge_provider_aws_ca_cert_file": "/agent_demo/iot-credentials/AmazonRootCA1.pem",
"sagemaker_edge_provider_aws_cert_file": "/agent_demo/iot-credentials/device.pem.crt",
"sagemaker_edge_provider_aws_cert_pk_file": "/agent_demo/iot-credentials/private.pem.key",
"sagemaker_edge_provider_aws_iot_cred_endpoint": "endpoint",
"sagemaker_edge_provider_provider": "Aws",
"sagemaker_edge_provider_s3_bucket_name": bucket,
"sagemaker_edge_core_capture_data_destination": "Cloud"
}
```
Replace the following:
+ `"device_name"` with the name of your device (this string was stored in an earlier step in a variable named `device_name`).
+ `"device_fleet_name`" with the name of your device fleet (this string was stored an earlier step in a variable named `device_fleet_name`)
+ `"endpoint"` with your Amazon account-specific endpoint for the credentials provider (this string was stored in an earlier step in a variable named `endpoint`).
Next, save it as a JSON file:
```
edge_config_file = open("sagemaker_edge_config.json", "w")
json.dump(sagemaker_edge_config, edge_config_file, indent = 6)
edge_config_file.close()
```
Upload the configuration file to your Amazon S3 bucket:
```
!aws s3 cp sagemaker_edge_config.json s3://{bucket}/
```
1. **Copy the release artifacts, configuration file, and credentials to your device.**
The following instructions are performed on the edge device itself.
**Note**
You must first install Python, the Amazon SDK for Python (Boto3), and the Amazon CLI on your edge device.
Open a terminal on your device. Create a folder to store the release artifacts, your credentials, and the configuration file.
```
mkdir agent_demo
cd agent_demo
```
Copy the contents of the release artifacts that you stored in your Amazon S3 bucket to your device:
```
# Copy release artifacts
aws s3 cp s3:///agent_demo/ ./ --recursive
```
(The contents of the release artifact was stored in a directory called `agent_demo` in a previous step). Replace `` and `agent_demo` with the name of your Amazon S3 bucket and the file path to your release artifacts, respectively.
Go the `/bin` directory and make the binary files executable:
```
cd bin
chmod +x sagemaker_edge_agent_binary
chmod +x sagemaker_edge_agent_client_example
cd agent_demo
```
Make a directory to store your Amazon IoT credentials and copy your credentials from your Amazon S3 bucket to your edge device (use the same on you define in the variable `bucket`:
```
mkdir iot-credentials
cd iot-credentials
aws s3 cp s3:///authorization-files/AmazonRootCA1.pem ./
aws s3 cp s3:///authorization-files/device.pem.crt ./
aws s3 cp s3:///authorization-files/private.pem.key ./
cd ../
```
Make a directory to store your model signing root certificates:
```
mkdir certificates
cd certificates
aws s3 cp s3:///authorization-files/us-west-2.pem ./
cd agent_demo
```
Copy your configuration file to your device:
```
#Download config file from S3
aws s3 cp s3:///sagemaker_edge_config.json ./
cd agent_demo
```
Your `agent_demo` directory on your edge device should look similar to the following:
```
├──agent_demo
| ├── bin
| ├── sagemaker_edge_agent_binary
| └── sagemaker_edge_agent_client_example
| ├── sagemaker_edge_config.json
| ├── certificates
| └──us-west-2.pem
| ├── iot-credentials
| ├── AmazonRootCA1.pem
| ├── device.pem.crt
| └── private.pem.key
| ├── docs
| ├── api
| └── examples
| ├── ATTRIBUTIONS.txt
| ├── LICENSE.txt
| └── RELEASE_NOTES.md
```
# Run Agent
In this section you will run the agent as a binary using gRPC, and check that both your device and fleet are working and collecting sample data.
1. **Launch the agent.**
The SageMaker Edge Manager agent can be run as a standalone process in the form of an Executable and Linkable Format (ELF) executable binary or can be linked against as a Dynamic Shared Object (.dll). Running as a standalone executable binary is the preferred mode and is supported on Linux.
This example uses gRPC to run the agent. gRPC is an open source high-performance Remote Procedure Call (RPC) framework that can run in any environment. For more information about gRPC, see the [gRPC documentation](https://grpc.io/docs/).
To use gRPC, perform the following steps:
1. Define a service in a .proto file.
1. Generate server and client code using the protocol buffer compiler.
1. Use the Python (or other languages supported by gRPC) gRPC API to write the server for your service.
1. Use the Python (or other languages supported by gRPC) gRPC API to write a client for your service.
The release artifact you downloaded contains a gRPC application ready for you to run the agent. The example is located within the `/bin` directory of your release artifact. The `sagemaker_edge_agent_binary` binary executable is in this directory.
To run the agent with this example, provide the path to your socket file (.sock) and JSON .config file:
```
./bin/sagemaker_edge_agent_binary -a /tmp/sagemaker_edge_agent_example.sock -c sagemaker_edge_config.json
```
1. **Check your device.**
Check that your device is connected and sampling data. Making periodic checks, manually or automatically, allows you to check that your device or fleet is working properly.
Provide the name of the fleet to which the device belongs and the unique device identifier. From your local machine, run the following:
```
sagemaker_client.describe_device(
DeviceName=device_name,
DeviceFleetName=device_fleet_name
)
```
For the given model, you can see the name, model version, latest sample time, and when the last inference was made.
```
{
"DeviceName": "sample-device",
"DeviceFleetName": "demo-device-fleet",
"IoTThingName": "sample-thing-name-1",
"RegistrationTime": 1600977370,
"LatestHeartbeat": 1600977370,
"Models":[
{
"ModelName": "mobilenet_v2.tar.gz",
"ModelVersion": "1.1",
"LatestSampleTime": 1600977370,
"LatestInference": 1600977370
}
]
}
```
The timestamp provided by `LastetHeartbeat` indicates the last signal that was received from the device. `LatestSampleTime` and `LatestInference` describe the time stamp of the last data sample and inference, respectively.
1. **Check your fleet.**
Check that your fleet is working with `GetDeviceFleetReport`. Provide the name of the fleet the device belongs to.
```
sagemaker_client.get_device_fleet_report(
DeviceFleetName=device_fleet_name
)
```
For a given model, you can see the name, model version, latest sample time, and when the last inference was made, along with the Amazon S3 bucket URI where the data samples are stored.
```
# Sample output
{
"DeviceFleetName": "sample-device-fleet",
"DeviceFleetArn": "arn:aws:sagemaker:us-west-2:9999999999:device-fleet/sample-fleet-name",
"OutputConfig": {
"S3OutputLocation": "s3://fleet-bucket/package_output",
},
"AgentVersions":[{"Version": "1.1", "AgentCount": 2}]}
"DeviceStats": {"Connected": 2, "Registered": 2},
"Models":[{
"ModelName": "sample-model",
"ModelVersion": "1.1",
"OfflineDeviceCount": 0,
"ConnectedDeviceCount": 2,
"ActiveDeviceCount": 2,
"SamplingDeviceCount": 100
}]
}
```
# Setup for Devices and Fleets in SageMaker Edge Manager
Fleets are collections of logically grouped devices you can use to collect and analyze data. You can use SageMaker Edge Manager to operate machine learning models on a fleet of smart cameras, smart speakers, robots, and other edge devices.
Create a fleet and register your devices either programmatically with the Amazon SDK for Python (Boto3) or through the SageMaker AI console.
**Topics**
+ [Create a Fleet](edge-device-fleet-create.md)
+ [Register a Device](edge-device-fleet-register.md)
+ [Check Status](edge-device-fleet-check-status.md)
# Create a Fleet
You can create a fleet programmatically with the Amazon SDK for Python (Boto3) or through the SageMaker AI console [https://console.amazonaws.cn/sagemaker](https://console.amazonaws.cn/sagemaker/).
## Create a Fleet (Boto3)
Use the `CreateDeviceFleet` API to create a fleet. Specify a name for the fleet, your Amazon IoT Role ARN for the `RoleArn` field, as well as an Amazon S3 URI where you want the device to store sampled data.
You can optionally include a description of the fleet, tags, and an Amazon KMS Key ID.
```
import boto3
# Create SageMaker client so you can interact and manage SageMaker resources
sagemaker_client = boto3.client("sagemaker", region_name="aws-region")
sagemaker_client.create_device_fleet(
DeviceFleetName="sample-fleet-name",
RoleArn="arn:aws:iam::999999999:role/rolename", # IoT Role ARN
Description="fleet description",
OutputConfig={
S3OutputLocation="s3://bucket/",
KMSKeyId: "1234abcd-12ab-34cd-56ef-1234567890ab",
},
Tags=[
{
"Key": "string",
"Value" : "string"
}
],
)
```
An Amazon IoT Role Alias is created for you when you create a device fleet. The Amazon IoT role alias provides a mechanism for connected devices to authenticate to Amazon IoT using X.509 certificates and then obtain short-lived Amazon credentials from an IAM role that is associated with the Amazon IoT role alias.
Use `DescribeDeviceFleet` to get the role alias name and ARN.
```
# Print Amazon Resource Name (ARN) and alias that has access
# to Amazon Internet of Things (IoT).
sagemaker_client.describe_device_fleet(DeviceFleetName=device_fleet_name)['IotRoleAlias']
```
Use `DescribeDeviceFleet` API to get a description of fleets you created.
```
sagemaker_client.describe_device_fleet(
DeviceFleetName="sample-fleet-name"
)
```
By default, it returns the name of the fleet, the device fleet ARN, the Amazon S3 bucket URI, the IAM role, the role alias created in Amazon IoT, a timestamp of when the fleet was created, and a timestamp of when the fleet was last modified.
```
{ "DeviceFleetName": "sample-fleet-name",
"DeviceFleetArn": "arn:aws:sagemaker:us-west-2:9999999999:device-fleet/sample-fleet-name",
"IAMRole": "arn:aws:iam::999999999:role/rolename",
"Description": "this is a sample fleet",
"IoTRoleAlias": "arn:aws:iot:us-west-2:9999999999:rolealias/SagemakerEdge-sample-fleet-name"
"OutputConfig": {
"S3OutputLocation": "s3://bucket/folder",
"KMSKeyId": "1234abcd-12ab-34cd-56ef-1234567890ab"
},
"CreationTime": "1600977370",
"LastModifiedTime": "1600977370"}
```
## Create a Fleet (Console)
You can create a Edge Manager packaging job using the Amazon SageMaker AI console at [https://console.amazonaws.cn/sagemaker](https://console.amazonaws.cn/sagemaker/).
1. In the SageMaker AI console, choose **Edge Manager** and then choose **Edge device fleets**.
1. Choose **Create device fleet**.
![\[The locaiton of the Create device fleet in the console.\]](http://docs.amazonaws.cn/en_us/sagemaker/latest/dg/images/smith/create-device-button-edited.png)
1. Enter a name for the device fleet in the **Device fleet name** field. Choose **Next**.
![\[The location of the Next button in the Device fleet properties section in the console.\]](http://docs.amazonaws.cn/en_us/sagemaker/latest/dg/images/smith/create-device-fleet-filled.png)
1. On the **Output configuration** page, specify the Amazon S3 bucket URI where you want to store sample data from your device fleet. You can optionally add an encryption key as well by electing an existing Amazon KMS key from the dropdown list or by entering a key’s ARN. Choose **Submit**.
![\[Example Output configuration page in the console.\]](http://docs.amazonaws.cn/en_us/sagemaker/latest/dg/images/smith/create-device-fleet-output-filled.png)
1. Choose the name of your device fleet to be redirected to the device fleet details. This page displays the name of the device fleet, ARN, description (if you provided one), date the fleet was created, last time the fleet was modified, Amazon S3 bucket URI, Amazon KMS key ID (if provided), Amazon IoT alias (if provided), and IAM role. If you added tags, they appear in the **Device fleet tags** section.
# Register a Device
**Important**
Device registration is required to use any part of SageMaker Edge Manager.
You can create a fleet programmatically with the Amazon SDK for Python (Boto3) or through the SageMaker AI console at [https://console.amazonaws.cn/sagemaker](https://console.amazonaws.cn/sagemaker/).
## Register a Device (Boto3)
To register your device, first create and register an Amazon IoT thing object and configure an IAM role. SageMaker Edge Manager takes advantage of the Amazon IoT Core services to facilitate the connection between the edge devices and the cloud. You can take advantage of existing Amazon IoT functionality after you set up your devices to work with Edge Manager.
To connect your device to Amazon IoT you need to create Amazon IoT thing objects, create and register a client certificate with Amazon IoT, and create and configure IAM role for your devices.
See the [Getting Started Guide](https://docs.amazonaws.cn/sagemaker/latest/dg/edge-manager-getting-started.html) for an in-depth example or the [Explore Amazon IoT Core services in hands-on tutorial](https://docs.amazonaws.cn/iot/latest/developerguide/iot-gs-first-thing.html).
Use the `RegisterDevices` API to register your device. Provide the name of the fleet of which you want the devices to be a part, as well as a name for the device. You can optionally add a description to the device, tags, and Amazon IoT thing name associated with the device.
```
sagemaker_client.register_devices(
DeviceFleetName="sample-fleet-name",
Devices=[
{
"DeviceName": "sample-device-1",
"IotThingName": "sample-thing-name-1",
"Description": "Device #1"
}
],
Tags=[
{
"Key": "string",
"Value" : "string"
}
],
)
```
## Register a Device (Console)
You can register your device using the SageMaker AI console at [https://console.amazonaws.cn/sagemaker](https://console.amazonaws.cn/sagemaker/).
1. In the SageMaker AI console, choose **Edge Inference** and then choose **Edge devices**.
1. Choose **Register devices**.
![\[Location of Register devices in the Edge Devices section of the console.\]](http://docs.amazonaws.cn/en_us/sagemaker/latest/dg/images/smith/register-device-button.png)
1. In the **Device properties** section, enter the name of the fleet the device belongs to under the **Device fleet name** field. Choose **Next**.
![\[The Device properties section in the console.\]](http://docs.amazonaws.cn/en_us/sagemaker/latest/dg/images/smith/register-devices-empty.png)
1. In the **Device source** section, add your devices one by one. You must include a **Device Name** for each device in your fleet. You can optionally provide a description (in the **Description** field) and an Internet of Things (IoT) object name (in the **IoT name** field). Choose **Submit** once you have added all your devices.
![\[The Device source section in the console.\]](http://docs.amazonaws.cn/en_us/sagemaker/latest/dg/images/smith/register-devices-device-source.png)
The **Devices** page displays the name of the device you have added, the fleet to which it belongs, when it was registered, the last heartbeat, and the description and Amazon IoT name, if you provided one.
Choose a device to view the device’s details, including the device name, fleet, ARN, description, IoT Thing name, when the device was registered, and the last heartbeat.
# Check Status
Check that your device or fleet is connected and sampling data. Making periodic checks, manually or automatically, allows you to check that your device or fleet is working properly.
Use the Amazon S3 console at [https://console.amazonaws.cn/s3/](https://console.amazonaws.cn/s3/) to interactively choose a fleet for a status check. You can also use the Amazon SDK for Python (Boto3). The following describes different APIs from Boto3 you can use to check the status of your device or fleet. Use the API that best fits your use case.
+ **Check an individual device.**
To check the status of an individual device, use `DescribeDevice` API. A list containing one or more models is provided if a models have been deployed to the device.
```
sagemaker_client.describe_device(
DeviceName="sample-device-1",
DeviceFleetName="sample-fleet-name"
)
```
Running `DescribeDevice` returns:
```
{ "DeviceName": "sample-device".
"Description": "this is a sample device",
"DeviceFleetName": "sample-device-fleet",
"IoTThingName": "SampleThing",
"RegistrationTime": 1600977370,
"LatestHeartbeat": 1600977370,
"Models":[
{
"ModelName": "sample-model",
"ModelVersion": "1.1",
"LatestSampleTime": 1600977370,
"LatestInference": 1600977370
}
]
}
```
+ **Check a fleet of devices.**
To check the status of the fleet, use the `GetDeviceFleetReport` API. Provide the name of the device fleet to get a summary of the fleet.
```
sagemaker_client.get_device_fleet_report(
DeviceFleetName="sample-fleet-name"
)
```
+ **Check for a heartbeat.**
Each device within a fleet periodically generates a signal, or “heartbeat”. The heartbeat can be used to check that the device is communicating with Edge Manager. If the timestamp of the last heartbeat is not being updated, the device may be failing.
Check the last heartbeat with made by a device with the `DescribeDevice` API. Specify the name of the device and the fleet to which the edge device belongs.
```
sagemaker_client.describe_device(
DeviceName="sample-device-1",
DeviceFleetName="sample-fleet-name"
)
```
# How to Package Model
SageMaker Edge Manager packaging jobs take Amazon SageMaker Neo–compiled models and make any changes necessary to deploy the model with the inference engine, Edge Manager agent.
**Topics**
+ [Complete prerequisites](edge-packaging-job-prerequisites.md)
+ [Package a Model (Amazon SageMaker AI Console)](edge-packaging-job-console.md)
+ [Package a Model (Boto3)](edge-packaging-job-boto3.md)
# Complete prerequisites
To package a model, you must do the following:
1. **Compile your machine learning model with SageMaker AI Neo.**
If you have not already done so, compile your model with SageMaker Neo. For more information on how to compile your model, see [Compile and Deploy Models with Neo](https://docs.amazonaws.cn/sagemaker/latest/dg/neo.html). If you are first-time user of SageMaker Neo, go through [Getting Started with Neo Edge Devices](https://docs.amazonaws.cn/sagemaker/latest/dg/neo-getting-started-edge.html).
1. **Get the name of your compilation job.**
Provide the name of the compilation job name you used when you compiled your model with SageMaker Neo. Open the SageMaker AI console at [https://console.amazonaws.cn/sagemaker/](https://console.amazonaws.cn/sagemaker/) and choose **Compilation jobs** to find a list of compilations that have been submitted to your Amazon account. The names of submitted compilation jobs are in the **Name** column.
1. **Get your IAM ARN.**
You need an Amazon Resource Name (ARN) of an IAM role that you can use to download and upload the model and contact SageMaker Neo.
Use one of the following methods to get your IAM ARN:
+ **Programmatically with the SageMaker AI Python SDK**
```
import sagemaker
# Initialize SageMaker Session object so you can interact with Amazon resources
sess = sagemaker.Session()
# Get the role ARN
role = sagemaker.get_execution_role()
print(role)
>> arn:aws:iam:::role/
```
For more information about using the SageMaker Python SDK, see the [SageMaker AI Python SDK API](https://sagemaker.readthedocs.io/en/stable/index.html).
+ **Using the Amazon Identity and Access Management (IAM) console**
Navigate to the IAM console at [https://console.amazonaws.cn/iam/](https://console.amazonaws.cn/iam/). In the IAM **Resources** section, choose **Roles** to view a list of roles in your Amazon account. Select or create a role that has `AmazonSageMakerFullAccess`, `AWSIoTFullAccess`, and `AmazonS3FullAccess`.
For more information on IAM, see [What is IAM?](https://docs.amazonaws.cn/IAM/latest/UserGuide/introduction.html)
1. **Have an S3 bucket URI.**
You need to have at least one Amazon Simple Storage Service (Amazon S3) bucket URI to store your Neo-compiled model, the output of the Edge Manager packaging job, and sample data from your device fleet.
Use one of the following methods to create an Amazon S3 bucket:
+ **Programmatically with the SageMaker AI Python SDK**
You can use the default Amazon S3 bucket during a session. A default bucket is created based on the following format: `sagemaker-{region}-{aws-account-id}`. To create a default bucket with the SageMaker Python SDK, use the following:
```
import sagemaker
session=sagemaker.create_session()
bucket=session.default_bucket()
```
+ **Using the Amazon S3 console**
Open the Amazon S3 console at [https://console.amazonaws.cn/s3/](https://console.amazonaws.cn/s3/) and see [How do I create an S3 Bucket?](https://docs.amazonaws.cn/AmazonS3/latest/user-guide/create-bucket.html) for step-by-step instructions.
# Package a Model (Amazon SageMaker AI Console)
You can create a SageMaker Edge Manager packaging job using the SageMaker AI console at [https://console.amazonaws.cn/sagemaker/](https://console.amazonaws.cn/sagemaker/). Before continuing, make sure you have satisfied the [Complete prerequisites](edge-packaging-job-prerequisites.md).
1. In the SageMaker AI console, choose **Edge Inference** and then choose **Create edge packaging jobs**, as shown in the following image.
![\[Location of Create edge packaging jobs in the console.\]](http://docs.amazonaws.cn/en_us/sagemaker/latest/dg/images/smith/pre-edge-packaging-button-edited.png)
1. On the **Job properties** page, enter a name for your packaging job under **Edge packaging job name**. Note that Edge Manager packaging job names are case-sensitive. Name your model and give it a version: enter this under **Model name** and **Model version**, respectively.
1. Next, select an **IAM role**. You can chose a role or let Amazon create a role for you. You can optionally specify a **resource key ARN** and **job tags**.
1. Choose **Next**.
![\[Example of the Job properties section in the console.\]](http://docs.amazonaws.cn/en_us/sagemaker/latest/dg/images/smith/create-edge-packaging-job-filled.png)
1. Specify the name of the compilation job you used when compiling your model with SageMaker Neo in the **Compilation job name** field. Choose **Next**.
![\[Example of the Model source section in the console.\]](http://docs.amazonaws.cn/en_us/sagemaker/latest/dg/images/smith/create-edge-packaging-job-model-source-filled.png)
1. On the **Output configuration** page, enter the Amazon S3 bucket URI in which you want to store the output of the packaging job.
![\[Example Output configuration page in the console.\]](http://docs.amazonaws.cn/en_us/sagemaker/latest/dg/images/smith/create-device-fleet-output-filled.png)
The **Status** column on the **Edge packaging** jobs page should read **IN PROGRESS**. Once the packaging job is complete, the status updates to **COMPLETED**.
Selecting a packaging job directs you to that job's settings. The **Job settings** section displays the job name, ARN, status, creation time, last modified time, duration of the packaging job, and role ARN.
The **Input configuration** section displays the location of the model artifacts, the data input configuration, and the machine learning framework of the model.
The **Output configuration** section displays the output location of the packaging job, the target device for which the model was compiled, and any tags you created.
1. Choose the name of your device fleet to be redirected to the device fleet details. This page displays the name of the device fleet, ARN, description (if you provided one), date the fleet was created, last time the fleet was modified, Amazon S3 bucket URI, Amazon KMS key ID (if provided), Amazon IoT alias (if provided), and IAM role. If you added tags, they appear in the **Device fleet tags** section.
# Package a Model (Boto3)
You can create a SageMaker Edge Manager packaging job with the Amazon SDK for Python (Boto3). Before continuing, make sure you have satisfied the [Complete prerequisites](edge-packaging-job-prerequisites.md).
To request an edge packaging job, use `CreateEdgePackagingJob`. You need to provide a name to your edge packaging job, the name of your SageMaker Neo compilation job, your role Amazon resource name (ARN), a name for your model, a version for your model, and the Amazon S3 bucket URI where you want to store the output of your packaging job. Note that Edge Manager packaging job names and SageMaker Neo compilation job names are case-sensitive.
```
# Import Amazon SDK for Python (Boto3)
import boto3
# Create Edge client so you can submit a packaging job
sagemaker_client = boto3.client("sagemaker", region_name='aws-region')
sagemaker_client.create_edge_packaging_job(
EdgePackagingJobName="edge-packaging-name",
CompilationJobName="neo-compilation-name",
RoleArn="arn:aws:iam::99999999999:role/rolename",
ModelName="sample-model-name",
ModelVersion="model-version",
OutputConfig={
"S3OutputLocation": "s3://your-bucket/",
}
)
```
You can check the status of an edge packaging job using `DescribeEdgePackagingJob` and providing the case-sensitive edge packaging job name:
```
response = sagemaker_client.describe_edge_packaging_job(
EdgePackagingJobName="edge-packaging-name")
```
This returns a dictionary that can be used to poll the status of the packaging job:
```
# Optional - Poll every 30 sec to check completion status
import time
while True:
response = sagemaker_client.describe_edge_packaging_job(
EdgePackagingJobName="edge-packaging-name")
if response['EdgePackagingJobStatus'] == 'Completed':
break
elif response['EdgePackagingJobStatus'] == 'Failed':
raise RuntimeError('Packaging job failed')
print('Packaging model...')
time.sleep(30)
print('Done!')
```
For a list of packaging jobs, use `ListEdgePackagingJobs`. You can use this API to search for a specific packaging job. Provide a partial name to filter packaging job names for `NameContains`, a partial name for `ModelNameContains` to filter for jobs in which the model name contains the name you provide. Also specify with which column to sort with `SortBy`, and by which direction to sort for `SortOrder` (either `Ascending` or `Descending`).
```
sagemaker_client.list_edge_packaging_jobs(
"NameContains": "sample",
"ModelNameContains": "sample",
"SortBy": "column-name",
"SortOrder": "Descending"
)
```
To stop a packaging job, use `StopEdgePackagingJob` and provide the name of your edge packaging job.
```
sagemaker_client.stop_edge_packaging_job(
EdgePackagingJobName="edge-packaging-name"
)
```
For a full list of Edge Manager APIs, see the [Boto3 documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html).
# The Edge Manager Agent
The Edge Manager agent is an inference engine for your edge devices. Use the agent to make predictions with models loaded onto your edge devices. The agent also collects model metrics and captures data at specific intervals. Sample data is stored in your Amazon S3 bucket.
There are two methods of installing and deploying the Edge Manager agent onto your edge devices:
1. Download the agent as a binary from the Amazon S3 release bucket. For more information, see [Download and Set Up the Edge Manager Agent Manually](edge-device-fleet-manual.md).
1. Use the Amazon IoT Greengrass V2 console or the Amazon CLI to deploy `aws.greengrass.SageMakerEdgeManager`. See [Create the Amazon IoT Greengrass V2 Components](edge-greengrass-custom-component.md).
# Download and Set Up the Edge Manager Agent Manually
Download the Edge Manager agent based on your operating system, architecture, and Amazon Region. The agent is periodically updated, so you have the option to choose your agent based on release dates and versions. Once you have the agent, create a JSON configuration file. Specify the device IoT thing name, fleet name, device credentials, and other key-value pairs. See [Running the Edge Manager agent](#edge-device-fleet-running-agent) for full a list of keys you must specify in the configuration file. You can run the agent as an executable binary or link against it as a dynamic shared object (DSO).
## How the agent works
The agent runs on the CPU of your devices. The agent runs inference on the framework and hardware of the target device you specified during the compilation job. For example, if you compiled your model for the Jetson Nano, the agent supports the GPU in the provided [Deep Learning Runtime](https://github.com/neo-ai/neo-ai-dlr) (DLR).
The agent is released in binary format for supported operating systems. Check that your operating system is supported and meets the minimum OS requirement in the following table:
------
#### [ Linux ]
**Version:** Ubuntu 18.04
**Supported Binary Formats:** x86-64 bit (ELF binary) and ARMv8 64 bit (ELF binary)
------
#### [ Windows ]
**Version:** Windows 10 version 1909
**Supported Binary Formats:** x86-32 bit (DLL) and x86-64 bit (DLL)
------
## Installing the Edge Manager agent
To use the Edge Manager agent, you first must obtain the release artifacts and a root certificate. The release artifacts are stored in an Amazon S3 bucket in the `us-west-2` Region. To download the artifacts, specify your operating system (``) and the ``.
Based on your operating system, replace `` with one of the following:
| Windows 32-bit | Windows 64-bit | Linux x86-64 | Linux ARMv8 |
| --- | --- | --- | --- |
| windows-x86 | windows-x64 | linux-x64 | linux-armv8 |
The `VERSION` is broken into three components: `.-`, where:
+ ``: The release version. The release version is currently set to `1`.
+ ``: The time stamp of the artifacts release.
+ ``: The repository commit ID from which the release is built.
You must provide the `` and the time stamp in `YYYY-MM-DD` format. We suggest you use the latest artifact release time stamp.
Run the following in your command line to get the latest time stamp. Replace `` with your operating system:
```
aws s3 ls s3://sagemaker-edge-release-store-us-west-2-/Releases/ | sort -r
```
For example, if you have a Windows 32-bit OS, run:
```
aws s3 ls s3://sagemaker-edge-release-store-us-west-2-windows-x86/Releases/ | sort -r
```
This returns:
```
2020-12-01 23:33:36 0
PRE 1.20201218.81f481f/
PRE 1.20201207.02d0e97/
```
The return output in this example shows two release artifacts. The first release artifact file notes that the release version has a major release version of `1`, a time stamp of `20201218` (in YYYY-MM-DD format), and a `81f481f` SHA-7 commit ID.
**Note**
The preceding command assumes you have configured the Amazon Command Line Interface. For more information, about how to configure the settings that the Amazon CLI uses to interact with Amazon, see [Configuring the Amazon CLI](https://docs.amazonaws.cn//cli/latest/userguide/cli-chap-configure.html).
Based on your operating system, use the following commands to install the artifacts:
------
#### [ Windows 32-bit ]
```
aws s3 cp s3://sagemaker-edge-release-store-us-west-2-windows-x86/Releases//.zip .
aws s3 cp s3://sagemaker-edge-release-store-us-west-2-windows-x86/Releases//sha256_hex.shasum .
```
------
#### [ Windows 64-bit ]
```
aws s3 cp s3://sagemaker-edge-release-store-us-west-2-windows-x64/Releases//.zip .
aws s3 cp s3://sagemaker-edge-release-store-us-west-2-windows-x64/Releases//sha256_hex.shasum .
```
------
#### [ Linux x86-64 ]
```
aws s3 cp s3://sagemaker-edge-release-store-us-west-2-linux-x64/Releases//.tgz .
aws s3 cp s3://sagemaker-edge-release-store-us-west-2-linux-x64/Releases//sha256_hex.shasum .
```
------
#### [ Linux ARMv8 ]
```
aws s3 cp s3://sagemaker-edge-release-store-us-west-2-linux-armv8/Releases//.tgz .
aws s3 cp s3://sagemaker-edge-release-store-us-west-2-linux-armv8/Releases//sha256_hex.shasum .
```
------
You also must download a root certificate. This certificate validates model artifacts signed by Amazon before loading them onto your edge devices.
Replace `` corresponding to your platform from the list of supported operation systems and replace `` with your Amazon Region.
```
aws s3 cp s3://sagemaker-edge-release-store-us-west-2-/Certificates//.pem .
```
## Running the Edge Manager agent
You can run the SageMaker AI Edge Manager agent as a standalone process in the form of an Executable and Linkable Format (ELF) executable binary or you can link against it as a dynamic shared object (.dll). Linux supports running it as a standalone executable binary and is the preferred mode. Windows supports running it as a shared object (.dll).
On Linux, we recommend that you run the binary via a service that’s a part of your initialization (`init`) system. If you want to run the binary directly, you can do so in a terminal as shown in the following example. If you have a modern OS, there are no other installations necessary prior to running the agent, since all the requirements are statically built into the executable. This gives you flexibility to run the agent on the terminal, as a service, or within a container.
To run the agent, first create a JSON configuration file. Specify the following key-value pairs:
+ `sagemaker_edge_core_device_name`: The name of the device. This device name needs to be registered along with the device fleet in the SageMaker Edge Manager console.
+ `sagemaker_edge_core_device_fleet_name`: The name of the fleet to which the device belongs.
+ `sagemaker_edge_core_region`: The Amazon Region associated with the device, the fleet and the Amazon S3 buckets. This corresponds to the Region where the device is registered and where the Amazon S3 bucket is created (they are expected to be the same). The models themselves can be compiled with SageMaker Neo in a different Region, this configuration is not related to model compilation Region.
+ `sagemaker_edge_core_root_certs_path`: The absolute folder path to root certificates. This is used to validate the device with the relevant Amazon account.
+ `sagemaker_edge_provider_aws_ca_cert_file`: The absolute path to Amazon Root CA certificate (AmazonRootCA1.pem). This is used to validate the device with the relevant Amazon account. `AmazonCA` is a certificate owned by Amazon.
+ `sagemaker_edge_provider_aws_cert_file`: The absolute path to Amazon IoT signing root certificate (`*.pem.crt`).
+ `sagemaker_edge_provider_aws_cert_pk_file`: The absolute path to Amazon IoT private key. (`*.pem.key`).
+ `sagemaker_edge_provider_aws_iot_cred_endpoint`: The Amazon IoT credentials endpoint (*identifier*.iot.*region*.amazonaws.com). This endpoint is used for credential validation. See [Connecting devices to Amazon IoT](https://docs.amazonaws.cn/iot/latest/developerguide/iot-connect-devices.html) for more information.
+ `sagemaker_edge_provider_provider`: This indicates the implementation of the provider interface being used. The provider interface communicates with the end network services for uploads, heartbeats and registration validation. By default this is set to `"Aws"`. We allow custom implementations of the provider interface. It can be set to `None` for no provider or `Custom` for custom implementation with the relevant shared object path provided.
+ `sagemaker_edge_provider_provider_path`: Provides the absolute path to the provider implementation shared object. (.so or .dll file). The `"Aws"` provider .dll or .so file is provided with the agent release. This field is mandatory.
+ `sagemaker_edge_provider_s3_bucket_name`: The name of your Amazon S3 bucket (not the Amazon S3 bucket URI). The bucket must have a `sagemaker` string within its name.
+ `sagemaker_edge_log_verbose` (Boolean.): Optional. This sets the debug log. Select either `True` or `False`.
+ `sagemaker_edge_telemetry_libsystemd_path`: For Linux only, `systemd` implements the agent crash counter metric. Set the absolute path of libsystemd to turn on the crash counter metric. You can find the default libsystemd path can be found by running `whereis libsystemd` in the device terminal.
+ `sagemaker_edge_core_capture_data_destination`: The destination for uploading capture data. Choose either `"Cloud"` or `"Disk"`. The default is set to `"Disk"`. Setting it to `"Disk"` writes the input and output tensor(s) and auxiliary data to the local file system at your preferred location of. When writing to `"Cloud"` use the Amazon S3 bucket name provided in the `sagemaker_edge_provider_s3_bucket_name` configuration.
+ `sagemaker_edge_core_capture_data_disk_path`: Set the absolute path in the local file system, into which capture data files are written when `"Disk"` is the destination. This field is not used when `"Cloud"` is specified as the destination.
+ `sagemaker_edge_core_folder_prefix`: The parent prefix in Amazon S3 where captured data is stored when you specify `"Cloud"` as the capture data destination (`sagemaker_edge_core_capture_data_disk_path)`. Captured data is stored in a sub-folder under `sagemaker_edge_core_capture_data_disk_path` if `"Disk"` is set as the data destination.
+ `sagemaker_edge_core_capture_data_buffer_size` (Integer value) : The capture data circular buffer size. It indicates the maximum number of requests stored in the buffer.
+ `sagemaker_edge_core_capture_data_batch_size` (Integer value): The capture data batch size. It indicates the size of a batch of requests that are handled from the buffer. This value must to be less than `sagemaker_edge_core_capture_data_buffer_size`. A maximum of half the size of the buffer is recommended for batch size.
+ `sagemaker_edge_core_capture_data_push_period_seconds` (Integer value): The capture data push period in seconds. A batch of requests in the buffer is handled when there are batch size requests in the buffer, or when this time period has completed (whichever comes first). This configuration sets that time period.
+ `sagemaker_edge_core_capture_data_base64_embed_limit`: The limit for uploading capture data in bytes. Integer value.
Your configuration file should look similar to the following example(with your specific values specified). This example uses the default Amazon provider(`"Aws"`) and does not specify a periodic upload.
```
{
"sagemaker_edge_core_device_name": "device-name",
"sagemaker_edge_core_device_fleet_name": "fleet-name",
"sagemaker_edge_core_region": "region",
"sagemaker_edge_core_root_certs_path": "",
"sagemaker_edge_provider_provider": "Aws",
"sagemaker_edge_provider_provider_path" : "/path/to/libprovider_aws.so",
"sagemaker_edge_provider_aws_ca_cert_file": "/AmazonRootCA1.pem",
"sagemaker_edge_provider_aws_cert_file": "/device.pem.crt",
"sagemaker_edge_provider_aws_cert_pk_file": "/private.pem.key",
"sagemaker_edge_provider_aws_iot_cred_endpoint": "https://",
"sagemaker_edge_core_capture_data_destination": "Cloud",
"sagemaker_edge_provider_s3_bucket_name": "sagemaker-bucket-name",
"sagemaker_edge_core_folder_prefix": "Amazon S3 folder prefix",
"sagemaker_edge_core_capture_data_buffer_size": 30,
"sagemaker_edge_core_capture_data_batch_size": 10,
"sagemaker_edge_core_capture_data_push_period_seconds": 4000,
"sagemaker_edge_core_capture_data_base64_embed_limit": 2,
"sagemaker_edge_log_verbose": false
}
```
The release artifact includes a binary executable called `sagemaker_edge_agent_binary` in the `/bin` directory. To run the binary, use the `-a` flag to create a socket file descriptor (.sock) in a directory of your choosing and specify the path of the agent JSON config file you created with the `-c` flag.
```
./sagemaker_edge_agent_binary -a -c
```
The following example shows the code snippet with a directory and file path specified:
```
./sagemaker_edge_agent_binary -a /tmp/sagemaker_edge_agent_example.sock -c sagemaker_edge_config.json
```
In this example, a socket file descriptor named `sagemaker_edge_agent_example.sock` is created in the `/tmp` directory and points to a configuration file that is in the same working directory as the agent called `sagemaker_edge_config.json`.
# Model Package and Edge Manager Agent Deployment with Amazon IoT Greengrass
SageMaker Edge Manager integrates Amazon IoT Greengrass version 2 to simplify accessing, maintaining, and deploying the Edge Manager agent and model to your devices. Without Amazon IoT Greengrass V2, setting up your devices and fleets to use SageMaker Edge Manager requires you to manually copy the Edge Manager agent from an Amazon S3 release bucket. You use the agent to make predictions with models loaded onto your edge devices. With Amazon IoT Greengrass V2 and SageMaker Edge Manager integration, you can use Amazon IoT Greengrass V2 components. Components are pre-built software modules that can connect your edge devices to Amazon services or third-party service via Amazon IoT Greengrass.
You must install the Amazon IoT Greengrass Core software onto your device(s) if you want to use Amazon IoT Greengrass V2 to deploy the Edge Manager agent and your model. For more information about device requirements and how to set up your devices, see [Setting up Amazon IoT Greengrass core devices](https://docs.amazonaws.cn/greengrass/v2/developerguide/setting-up.html) in the Amazon IoT Greengrass documentation.
You use the following three components to deploy the Edge Manager agent:
+ *A pre-built public component*: SageMaker AI maintains the public Edge Manager component.
+ *A autogenerated private component*: The private component is autogenerated when you package your machine learning model with the [https://docs.amazonaws.cn/sagemaker/latest/APIReference/API_CreateEdgePackagingJob.html](https://docs.amazonaws.cn/sagemaker/latest/APIReference/API_CreateEdgePackagingJob.html) API and specify `GreengrassV2Component` for the Edge Manager API field `PresetDeploymentType`.
+ *A custom component*: This is the inference application that is responsible for preprocessing and making inferences on your device. You must create this component. See either [Create a Hello World custom component](edge-greengrass-custom-component.md#edge-greengrass-create-custom-component-how) in the SageMaker Edge Manager documentation or [Create custom Amazon IoT Greengrass components](https://docs.amazonaws.cn/greengrass/v2/developerguide/create-components.html) in the Amazon IoT Greengrass documentation for more information on how to create custom components.
# Complete prerequisites to deploy the Edge Manager agent
SageMaker Edge Manager uses Amazon IoT Greengrass V2 to simplify the deployment of the Edge Manager agent, your machine learning models, and your inference application to your devices with the use of components. To make it easier to maintain your Amazon IAM roles, Edge Manager allows you to reuse your existing Amazon IoT role alias. If you do not have one yet, Edge Manager generates a role alias as part of the Edge Manager packaging job. You no longer need to associate a role alias generated from the SageMaker Edge Manager packaging job with your Amazon IoT role.
Before you start, you must complete the following prerequisites:
1. Install the Amazon IoT Greengrass Core software. For detailed information, see [Install the Amazon IoT Greengrass Core software](https://docs.amazonaws.cn/greengrass/v2/developerguide/getting-started.html#install-greengrass-v2).
1. Set up Amazon IoT Greengrass V2. For more information, see [Install Amazon IoT Greengrass Core software with manual resource provisioning](https://docs.amazonaws.cn/greengrass/v2/developerguide/manual-installation.html).
**Note**
Make sure the Amazon IoT thing name is all lowercase and does not contain characters except (optionally) dashes (`‐`).
The IAM Role must start with `SageMaker*`
1. Attach the following permission and inline policy to the IAM role created during Amazon IoT Greengrass V2 setup.
+ Navigate to the IAM console [https://console.amazonaws.cn/iam/](https://console.amazonaws.cn/iam/).
+ Search for the role you created by typing in rhe role name in the **Search** field.
+ Choose your role.
+ Next, choose **Attach policies**.
+ Search for **AmazonSageMakerEdgeDeviceFleetPolicy**.
+ Select **AmazonSageMakerFullAccess** (This is an optional step that makes it easier for you to reuse this IAM role in model compilation and packaging).
+ Add required permissions to a role's permissions policy, don't attach inline policies to IAM users.
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement":[
{
"Sid":"GreengrassComponentAccess",
"Effect":"Allow",
"Action":[
"greengrass:CreateComponentVersion",
"greengrass:DescribeComponent"
],
"Resource":"*"
}
]
}
```
------
+ Choose **Attach policy**.
+ Choose **Trust relationship**.
+ Choose **Edit trust relationship**.
+ Replace the content with the following.
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "credentials.iot.amazonaws.com"
},
"Action": "sts:AssumeRole"
},
{
"Effect": "Allow",
"Principal": {
"Service": "sagemaker.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
```
------
1. Create an Edge Manager device fleet. For information on how to create a fleet, see [Setup for Devices and Fleets in SageMaker Edge Manager](edge-device-fleet.md).
1. Register your device with the same name as your Amazon IoT thing name created during the Amazon IoT Greengrass V2 setup.
1. Create at least one custom private Amazon IoT Greengrass component. This component is the application that runs inference on the device. For more information, see [Create a Hello World custom component](edge-greengrass-custom-component.md#edge-greengrass-create-custom-component-how)
**Note**
The SageMaker Edge Manager and Amazon IoT Greengrass integration only works for Amazon IoT Greengrass v2.
Both your Amazon IoT thing name and Edge Manager device name must be the same.
SageMaker Edge Manager does not load local Amazon IoT certificates and call the Amazon IoT credential provider endpoint directly. Instead, SageMaker Edge Manager uses the Amazon IoT Greengrass v2 TokenExchangeService and it fetches a temporary credential from a TES endpoint.
# Create the Amazon IoT Greengrass V2 Components
Amazon IoT Greengrass uses *components*, a software module that is deployed to and runs on a Amazon IoT Greengrass core device. You need (at a minimum) three components:
1. *A public Edge Manager Agent Amazon IoT Greengrass component* which deploys the Edge Manager agentbinary.
1. *A model component* that is autogenerated when you package your machine learning model with either the Amazon SDK for Python (Boto3) API or with the SageMaker AI console. For information, see [Create an autogenerated component](#edge-greengrass-autogenerate-component-how).
1. *A private, custom component* to implement the Edge Manager agent client application, and do any preprocessing and post-processing of the inference results. For more information about how to create a custom component, see [Create an autogenerated component](#edge-greengrass-autogenerate-component-how) or [Create custom Amazon IoT Greengrass components](https://docs.amazonaws.cn/greengrass/v2/developerguide/create-components.html).
## Create an autogenerated component
Generate the model component with the [https://docs.amazonaws.cn/sagemaker/latest/APIReference/API_CreateEdgePackagingJob.html](https://docs.amazonaws.cn/sagemaker/latest/APIReference/API_CreateEdgePackagingJob.html) API and specify `GreengrassV2Component` for the SageMaker Edge Manager packaging job API field `PresetDeploymentType`. When you call the `CreateEdgePackagingJob` API, Edge Manager takes your SageMaker AI Neo–compiled model in Amazon S3 and creates a model component. The model component is automatically stored in your account. You can view any of your components by navigating to the Amazon IoT console [https://console.amazonaws.cn/iot/](https://console.amazonaws.cn/greengrass/). Select **Greengrass** and then select **Core** devices. The page has a list of Amazon IoT Greengrass core devices associated with your account. If a model component name is not specified in `PresetDeploymentConfig`, the default name generated consists of `"SagemakerEdgeManager"` and the name of your Edge Manager agent packaging job. The following example demonstrates how to specify to Edge Manager to create a Amazon IoT Greengrass V2 component with the `CreateEdgePackagingJob` API.
```
import sagemaker
import boto3
# Create a SageMaker client object to make it easier to interact with other AWS services.
sagemaker_client = boto3.client('sagemaker', region=)
# Replace with your IAM Role ARN
sagemaker_role_arn = "arn:aws:iam:::role/*"
# Replace string with the name of your already created S3 bucket.
bucket = 'amzn-s3-demo-bucket-edge-manager'
# Specify a name for your edge packaging job.
edge_packaging_name = "edge_packag_job_demo"
# Replace the following string with the name you used for the SageMaker Neo compilation job.
compilation_job_name = "getting-started-demo"
# The name of the model and the model version.
model_name = "sample-model"
model_version = "1.1"
# Output directory in S3 where you want to store the packaged model.
packaging_output_dir = 'packaged_models'
packaging_s3_output = 's3://{}/{}'.format(bucket, packaging_output_dir)
# The name you want your Greengrass component to have.
component_name = "SagemakerEdgeManager" + edge_packaging_name
sagemaker_client.create_edge_packaging_job(
EdgePackagingJobName=edge_packaging_name,
CompilationJobName=compilation_job_name,
RoleArn=sagemaker_role_arn,
ModelName=model_name,
ModelVersion=model_version,
OutputConfig={
"S3OutputLocation": packaging_s3_output,
"PresetDeploymentType":"GreengrassV2Component",
"PresetDeploymentConfig":"{\"ComponentName\":\"sample-component-name\", \"ComponentVersion\":\"1.0.2\"}"
}
)
```
You can also create the autogenerated component with the SageMaker AI console. Follow steps 1-6 in [Package a Model (Amazon SageMaker AI Console)](edge-packaging-job-console.md)
Enter the Amazon S3 bucket URI where you want to store the output of the packaging job and optional encrytion key.
Complete the following to create the model component:
1. Choose **Preset deployment**.
1. Specify the name of the component for the **Component name** field.
1. Optionally, provide a description of the component, a component version, the platform OS, or the platform architecture for the **Component description**, **Component version**, **Platform OS**, and **Platform architecture**, respectively.
1. Choose **Submit**.
## Create a Hello World custom component
The custom application component is used to perform inference on the edge device. The component is responsible for loading models to SageMaker Edge Manager, invoking the Edge Manager agent for inference, and unloading the model when the component is shut down. Before you create your component, ensure the agent and application can communicate with Edge Manager. To do this, configure [gRPC](https://grpc.io/). The Edge Manager agent uses methods defined in Protobuf Buffers and the gRPC server to establish communication with the client application on the edge device and the cloud.
To use gRPC, you must:
1. Create a gRPC stub using the .proto file provided when you download the Edge Manager agent from Amazon S3 release bucket.
1. Write client code with the language you prefer.
You do not need to define the service in a .proto file. The service .proto files are included in the compressed TAR file when you download the Edge Manager agent release binary from the Amazon S3 release bucket.
Install gRPC and other necessary tools on your host machine and create the gRPC stubs `agent_pb2_grpc.py` and `agent_pb2.py` in Python. Make sure you have `agent.proto` in your local directory.
```
%%bash
pip install grpcio
pip install grpcio-tools
python3 -m grpc_tools.protoc --proto_path=. --python_out=. --grpc_python_out=. agent.proto
```
The preceding code generates the gRPC client and server interfaces from your .proto service definition. In other words, it creates the gRPC model in Python. The API directory contains the Protobuf specification for communicating with the agent.
Next, use the gRPC API to write a client and server for your service (2). The following example script, `edge_manager_python_example.py`, uses Python to load, list, and unload a `yolov3` model to the edge device.
```
import grpc
from PIL import Image
import agent_pb2
import agent_pb2_grpc
import os
model_path = ''
agent_socket = 'unix:///tmp/aws.greengrass.SageMakerEdgeManager.sock'
agent_channel = grpc.insecure_channel(agent_socket, options=(('grpc.enable_http_proxy', 0),))
agent_client = agent_pb2_grpc.AgentStub(agent_channel)
def list_models():
return agent_client.ListModels(agent_pb2.ListModelsRequest())
def list_model_tensors(models):
return {
model.name: {
'inputs': model.input_tensor_metadatas,
'outputs': model.output_tensor_metadatas
}
for model in list_models().models
}
def load_model(model_name, model_path):
load_request = agent_pb2.LoadModelRequest()
load_request.url = model_path
load_request.name = model_name
return agent_client.LoadModel(load_request)
def unload_model(name):
unload_request = agent_pb2.UnLoadModelRequest()
unload_request.name = name
return agent_client.UnLoadModel(unload_request)
def predict_image(model_name, image_path):
image_tensor = agent_pb2.Tensor()
image_tensor.byte_data = Image.open(image_path).tobytes()
image_tensor_metadata = list_model_tensors(list_models())[model_name]['inputs'][0]
image_tensor.tensor_metadata.name = image_tensor_metadata.name
image_tensor.tensor_metadata.data_type = image_tensor_metadata.data_type
for shape in image_tensor_metadata.shape:
image_tensor.tensor_metadata.shape.append(shape)
predict_request = agent_pb2.PredictRequest()
predict_request.name = model_name
predict_request.tensors.append(image_tensor)
predict_response = agent_client.Predict(predict_request)
return predict_response
def main():
try:
unload_model('your-model')
except:
pass
print('LoadModel...', end='')
try:
load_model('your-model', model_path)
print('done.')
except Exception as e:
print()
print(e)
print('Model already loaded!')
print('ListModels...', end='')
try:
print(list_models())
print('done.')
except Exception as e:
print()
print(e)
print('List model failed!')
print('Unload model...', end='')
try:
unload_model('your-model')
print('done.')
except Exception as e:
print()
print(e)
print('unload model failed!')
if __name__ == '__main__':
main()
```
Ensure `model_path` points to the name of the Amazon IoT Greengrass component containing the model if you use the same client code example.
You can create your Amazon IoT Greengrass V2 Hello World component once you have generated your gRPC stubs and you have your Hello World code ready. To do so:
+ Upload your `edge_manager_python_example.py`, `agent_pb2_grpc.py`, and `agent_pb2.py` to your Amazon S3 bucket and note down their Amazon S3 path.
+ Create a private component in the Amazon IoT Greengrass V2 console and define the recipe for your component. Specify the Amazon S3 URI to your Hello World application and gRPC stub in the following recipe.
```
---
RecipeFormatVersion: 2020-01-25
ComponentName: com.sagemaker.edgePythonExample
ComponentVersion: 1.0.0
ComponentDescription: Sagemaker Edge Manager Python example
ComponentPublisher: Amazon Web Services, Inc.
ComponentDependencies:
aws.greengrass.SageMakerEdgeManager:
VersionRequirement: '>=1.0.0'
DependencyType: HARD
Manifests:
- Platform:
os: linux
architecture: "/amd64|x86/"
Lifecycle:
install: |-
apt-get install python3-pip
pip3 install grpcio
pip3 install grpcio-tools
pip3 install protobuf
pip3 install Pillow
run:
script: |-
python3 {artifacts:path}/edge_manager_python_example.py
Artifacts:
- URI:
- URI:
- URI:
```
For detailed information about creating a Hello World recipe, see [Create your first component](https://docs.amazonaws.cn/greengrass/v2/developerguide/getting-started.html#create-first-component) in the Amazon IoT Greengrass documentation.
# Deploy the components to your device
Deploy your components with the Amazon IoT console or with the Amazon CLI.
## To deploy your components (console)
Deploy your Amazon IoT Greengrass components with the Amazon IoT console.
1. In the Amazon IoT Greengrass console at [https://console.amazonaws.cn/iot/](https://console.amazonaws.cn/greengrass/) navigation menu, choose **Deployments**.
1. On the **Components** page, on the **Public components** tab, choose `aws.greengrass.SageMakerEdgeManager`.
1. On the `aws.greengrass.SageMakerEdgeManager` page, choose **Deploy**.
1. From `Add to deployment`, choose one of the following:
1. To merge this component to an existing deployment on your target device, choose **Add to existing deployment**, and then select the deployment that you want to revise.
1. To create a new deployment on your target device, choose **Create new deployment**. If you have an existing deployment on your device, choosing this step replaces the existing deployment.
1. On the **Specify target** page, do the following:
1. Under **Deployment information**, enter or modify the friendly name for your deployment.
1. Under **Deployment targets**, select a target for your deployment, and choose **Next**. You cannot change the deployment target if you are revising an existing deployment.
1. On the **Select components** page, under **My components**, choose:
+ com.**
+ `aws.greengrass.SageMakerEdgeManager`
+ SagemakerEdgeManager.**
1. On the **Configure components** page, choose **com.greengrass.SageMakerEdgeManager**, and do the following.
1. Choose **Configure component**.
1. Under **Configuration update**, in **Configuration to merge**, enter the following configuration.
```
{
"DeviceFleetName": "device-fleet-name",
"BucketName": "bucket-name"
}
```
Replace *`device-fleet-name`* with the name of the edge device fleet that you created, and replace *`bucket-name`* with the name of the Amazon S3 bucket that is associated with your device fleet.
1. Choose **Confirm**, and then choose **Next**.
1. On the **Configure advanced settings** page, keep the default configuration settings, and choose **Next**.
1. On the **Review** page, choose **Deploy**.
## To deploy your components (Amazon CLI)
1. Create a ` deployment.json` file to define the deployment configuration for your SageMaker Edge Manager components. This file should look like the following example.
```
{
"targetArn":"targetArn",
"components": {
"aws.greengrass.SageMakerEdgeManager": {
"componentVersion": 1.0.0,
"configurationUpdate": {
"merge": {
"DeviceFleetName": "device-fleet-name",
"BucketName": "bucket-name"
}
}
},
"com.greengrass.SageMakerEdgeManager.ImageClassification": {
"componentVersion": 1.0.0,
"configurationUpdate": {
}
},
"com.greengrass.SageMakerEdgeManager.ImageClassification.Model": {
"componentVersion": 1.0.0,
"configurationUpdate": {
}
},
}
}
```
+ In the `targetArn` field, replace *`targetArn`* with the Amazon Resource Name (ARN) of the thing or thing group to target for the deployment, in the following format:
+ Thing: `arn:aws:iot:region:account-id:thing/thingName`
+ Thing group: `arn:aws:iot:region:account-id:thinggroup/thingGroupName`
+ In the `merge` field, replace *`device-fleet-name`* with the name of the edge device fleet that you created, and replace *`bucket-name`* with the name of the Amazon S3 bucket that is associated with your device fleet.
+ Replace the component versions for each component with the latest available version.
1. Run the following command to deploy the components on the device:
```
aws greengrassv2 create-deployment \
--cli-input-json file://path/to/deployment.json
```
The deployment can take several minutes to complete. In the next step, check the component log to verify that the deployment completed successfully and to view the inference results.
For more information about deploying components to individual devices or groups of devices, see [Deploy Amazon IoT Greengrass components to devices](https://docs.amazonaws.cn/greengrass/v2/developerguide/manage-deployments.html).
# Deploy the Model Package Directly with SageMaker Edge Manager Deployment API
SageMaker Edge Manager provides a deployment API that you can use to deploy models to device targets without Amazon IoT Greengrass. It is useful in situations where you want to update models independently of firmware updates or application deployment mechanisms. You can use the API to integrate your edge deployments into a CI/CD workflow to automatically deploy models once you have validated your model for accuracy. The API also has convenient rollback and staged rollout options for you to ensure models work well in a particular environment before wider rollout.
To use the Edge Manager deployment API first compile and package your model. For information on how to compile and package your model, see [Prepare Your Model for Deployment](edge-getting-started-step2.md). The following sections of this guide show how you can create edge deployments using SageMaker API, after you have compiled and packaged your models.
**Topics**
+ [Create an edge deployment plan](#create-edge-deployment-plan)
+ [Start the edge deployment](#start-edge-deployment-stage)
+ [Check the status of the deployment](#describe-edge-deployment-status)
## Create an edge deployment plan
You can create an edge deployment plan with the [https://docs.amazonaws.cn/sagemaker/latest/APIReference/API_CreateEdgeDeploymentPlan.html](https://docs.amazonaws.cn/sagemaker/latest/APIReference/API_CreateEdgeDeploymentPlan.html) API. The deployment plan can have multiple stages. You can configure each stage to rollout the deployment to a subset of edge devices (by percentage, or by device name). You can also configure how rollout failures are handled at each stage.
The following code snippet shows how you can create an edge deployment plan with 1 stage to deploy a compiled and package model to 2 specific edge devices:
```
import boto3
client = boto3.client("sagemaker")
client.create_edge_deployment_plan(
EdgeDeploymentPlanName="edge-deployment-plan-name",
DeviceFleetName="device-fleet-name",
ModelConfigs=[
{
"EdgePackagingJobName": "edge-packaging-job-name",
"ModelHandle": "model-handle"
}
],
Stages=[
{
"StageName": "stage-name",
"DeviceSelectionConfig": {
"DeviceSubsetType": "SELECTION",
"DeviceNames": ["device-name-1", "device-name-2"]
},
"DeploymentConfig": {
"FailureHandlingPolicy": "ROLLBACK_ON_FAILURE"
}
}
]
)
```
Instead of specific devices, if you want to deploy to the model to a percentage of devices in your fleet, then set the value of `DeviceSubsetType` to `"PERCENTAGE"` and replace `"DeviceNames": ["device-name-1", "device-name-2"]` with `"Percentage": desired-percentage` in the above example.
Stages can be added after the deployment plan has been created with the [CreateEdgeDeploymentStage](https://docs.amazonaws.cn/sagemaker/latest/APIReference/API_CreateEdgeDeploymentStage.html) API, in case you want to start rolling out new stages after validating your test rollout success. For more information about deployment stages see [DeploymentStage.](https://docs.amazonaws.cn/sagemaker/latest/APIReference/API_DeploymentStage.html)
## Start the edge deployment
After creating the deployment plan and the deployment stages, you can start the deployment with the [https://docs.amazonaws.cn/sagemaker/latest/APIReference/API_StartEdgeDeploymentStage.html](https://docs.amazonaws.cn/sagemaker/latest/APIReference/API_StartEdgeDeploymentStage.html) API.
```
client.start_edge_deployment_stage(
EdgeDeploymentPlanName="edge-deployment-plan-name",
StageName="stage-name"
)
```
## Check the status of the deployment
You can check the status of the edge deployment with the [DescribeEdgeDeploymentPlan](https://docs.amazonaws.cn/sagemaker/latest/APIReference/API_DescribeEdgeDeploymentPlan.html) API.
```
client.describe_edge_deployment_plan(
EdgeDeploymentPlanName="edge-deployment-plan-name"
)
```
# Manage Model
The Edge Manager agent can load multiple models at a time and make inference with loaded models on edge devices. The number of models the agent can load is determined by the available memory on the device. The agent validates the model signature and loads into memory all the artifacts produced by the edge packaging job. This step requires all the required certificates described in previous steps to be installed along with rest of the binary installation. If the model’s signature cannot be validated, then loading of the model fails with appropriate return code and reason.
SageMaker Edge Manager agent provides a list of Model Management APIs that implement control plane and data plane APIs on edge devices. Along with this documentation, we recommend going through the sample client implementation which shows canonical usage of the below described APIs.
The `proto` file is available as a part of the release artifacts (inside the release tarball). In this doc, we list and describe the usage of APIs listed in this `proto` file.
**Note**
There is one-to-one mapping for these APIs on Windows release and a sample code for an application implement in C\$1 is shared with the release artifacts for Windows. Below instructions are for running the Agent as a standalone process, applicable for to the release artifacts for Linux.
Extract the archive based on your OS. Where `VERSION` is broken into three components: `.-`. See [Installing the Edge Manager agent](edge-device-fleet-manual.md#edge-device-fleet-installation) for information on how to obtain the release version (``), time stamp of the release artifact (``), and the repository commit ID (`SHA-7`)
------
#### [ Linux ]
The zip archive can be extracted with the command:
```
tar -xvzf .tgz
```
------
#### [ Windows ]
The zip archive can be extracted with the UI or command:
```
unzip .tgz
```
------
The release artifact hierarchy (after extracting the `tar/zip` archive) is shown below. The agent `proto` file is available under `api/`.
```
0.20201205.7ee4b0b
├── bin
│ ├── sagemaker_edge_agent_binary
│ └── sagemaker_edge_agent_client_example
└── docs
├── api
│ └── agent.proto
├── attributions
│ ├── agent.txt
│ └── core.txt
└── examples
└── ipc_example
├── CMakeLists.txt
├── sagemaker_edge_client.cc
├── sagemaker_edge_client_example.cc
├── sagemaker_edge_client.hh
├── sagemaker_edge.proto
├── README.md
├── shm.cc
├── shm.hh
└── street_small.bmp
```
**Topics**
+ [Load Model](#edge-manage-model-loadmodel)
+ [Unload Model](#edge-manage-model-unloadmodel)
+ [List Models](#edge-manage-model-listmodels)
+ [Describe Model](#edge-manage-model-describemodel)
+ [Capture Data](#edge-manage-model-capturedata)
+ [Get Capture Status](#edge-manage-model-getcapturedata)
+ [Predict](#edge-manage-model-predict)
## Load Model
The Edge Manager agent supports loading multiple models. This API validates the model signature and loads into memory all the artifacts produced by the `EdgePackagingJob` operation. This step requires all the required certificates to be installed along with rest of the agent binary installation. If the model’s signature cannot be validated then this step fails with appropriate return code and error messages in the log.
```
// perform load for a model
// Note:
// 1. currently only local filesystem paths are supported for loading models.
// 2. multiple models can be loaded at the same time, as limited by available device memory
// 3. users are required to unload any loaded model to load another model.
// Status Codes:
// 1. OK - load is successful
// 2. UNKNOWN - unknown error has occurred
// 3. INTERNAL - an internal error has occurred
// 4. NOT_FOUND - model doesn't exist at the url
// 5. ALREADY_EXISTS - model with the same name is already loaded
// 6. RESOURCE_EXHAUSTED - memory is not available to load the model
// 7. FAILED_PRECONDITION - model is not compiled for the machine.
//
rpc LoadModel(LoadModelRequest) returns (LoadModelResponse);
```
------
#### [ Input ]
```
//
// request for LoadModel rpc call
//
message LoadModelRequest {
string url = 1;
string name = 2; // Model name needs to match regex "^[a-zA-Z0-9](-*[a-zA-Z0-9])*$"
}
```
------
#### [ Output ]
```
//
//
// response for LoadModel rpc call
//
message LoadModelResponse {
Model model = 1;
}
//
// Model represents the metadata of a model
// url - url representing the path of the model
// name - name of model
// input_tensor_metadatas - TensorMetadata array for the input tensors
// output_tensor_metadatas - TensorMetadata array for the output tensors
//
// Note:
// 1. input and output tensor metadata could empty for dynamic models.
//
message Model {
string url = 1;
string name = 2;
repeated TensorMetadata input_tensor_metadatas = 3;
repeated TensorMetadata output_tensor_metadatas = 4;
}
```
------
## Unload Model
Unloads a previously loaded model. It is identified via the model alias which was provided during `loadModel`. If the alias is not found or model is not loaded then returns error.
```
//
// perform unload for a model
// Status Codes:
// 1. OK - unload is successful
// 2. UNKNOWN - unknown error has occurred
// 3. INTERNAL - an internal error has occurred
// 4. NOT_FOUND - model doesn't exist
//
rpc UnLoadModel(UnLoadModelRequest) returns (UnLoadModelResponse);
```
------
#### [ Input ]
```
//
// request for UnLoadModel rpc call
//
message UnLoadModelRequest {
string name = 1; // Model name needs to match regex "^[a-zA-Z0-9](-*[a-zA-Z0-9])*$"
}
```
------
#### [ Output ]
```
//
// response for UnLoadModel rpc call
//
message UnLoadModelResponse {}
```
------
## List Models
Lists all the loaded models and their aliases.
```
//
// lists the loaded models
// Status Codes:
// 1. OK - unload is successful
// 2. UNKNOWN - unknown error has occurred
// 3. INTERNAL - an internal error has occurred
//
rpc ListModels(ListModelsRequest) returns (ListModelsResponse);
```
------
#### [ Input ]
```
//
// request for ListModels rpc call
//
message ListModelsRequest {}
```
------
#### [ Output ]
```
//
// response for ListModels rpc call
//
message ListModelsResponse {
repeated Model models = 1;
}
```
------
## Describe Model
Describes a model that is loaded on the agent.
```
//
// Status Codes:
// 1. OK - load is successful
// 2. UNKNOWN - unknown error has occurred
// 3. INTERNAL - an internal error has occurred
// 4. NOT_FOUND - model doesn't exist at the url
//
rpc DescribeModel(DescribeModelRequest) returns (DescribeModelResponse);
```
------
#### [ Input ]
```
//
// request for DescribeModel rpc call
//
message DescribeModelRequest {
string name = 1;
}
```
------
#### [ Output ]
```
//
// response for DescribeModel rpc call
//
message DescribeModelResponse {
Model model = 1;
}
```
------
## Capture Data
Allows the client application to capture input and output tensors in Amazon S3 bucket, and optionally the auxiliary. The client application is expected to pass a unique capture ID along with each call to this API. This can be later used to query status of the capture.
```
//
// allows users to capture input and output tensors along with auxiliary data.
// Status Codes:
// 1. OK - data capture successfully initiated
// 2. UNKNOWN - unknown error has occurred
// 3. INTERNAL - an internal error has occurred
// 5. ALREADY_EXISTS - capture initiated for the given capture_id
// 6. RESOURCE_EXHAUSTED - buffer is full cannot accept any more requests.
// 7. OUT_OF_RANGE - timestamp is in the future.
// 8. INVALID_ARGUMENT - capture_id is not of expected format.
//
rpc CaptureData(CaptureDataRequest) returns (CaptureDataResponse);
```
------
#### [ Input ]
```
enum Encoding {
CSV = 0;
JSON = 1;
NONE = 2;
BASE64 = 3;
}
//
// AuxilaryData represents a payload of extra data to be capture along with inputs and outputs of inference
// encoding - supports the encoding of the data
// data - represents the data of shared memory, this could be passed in two ways:
// a. send across the raw bytes of the multi-dimensional tensor array
// b. send a SharedMemoryHandle which contains the posix shared memory segment id and
// offset in bytes to location of multi-dimensional tensor array.
//
message AuxilaryData {
string name = 1;
Encoding encoding = 2;
oneof data {
bytes byte_data = 3;
SharedMemoryHandle shared_memory_handle = 4;
}
}
//
// Tensor represents a tensor, encoded as contiguous multi-dimensional array.
// tensor_metadata - represents metadata of the shared memory segment
// data_or_handle - represents the data of shared memory, this could be passed in two ways:
// a. send across the raw bytes of the multi-dimensional tensor array
// b. send a SharedMemoryHandle which contains the posix shared memory segment
// id and offset in bytes to location of multi-dimensional tensor array.
//
message Tensor {
TensorMetadata tensor_metadata = 1; //optional in the predict request
oneof data {
bytes byte_data = 4;
// will only be used for input tensors
SharedMemoryHandle shared_memory_handle = 5;
}
}
//
// request for CaptureData rpc call
//
message CaptureDataRequest {
string model_name = 1;
string capture_id = 2; //uuid string
Timestamp inference_timestamp = 3;
repeated Tensor input_tensors = 4;
repeated Tensor output_tensors = 5;
repeated AuxilaryData inputs = 6;
repeated AuxilaryData outputs = 7;
}
```
------
#### [ Output ]
```
//
// response for CaptureData rpc call
//
message CaptureDataResponse {}
```
------
## Get Capture Status
Depending on the models loaded the input and output tensors can be large (for many edge devices). Capture to the cloud can be time consuming. So the `CaptureData()` is implemented as an asynchronous operation. A capture ID is a unique identifier that the client provides during capture data call, this ID can be used to query the status of the asynchronous call.
```
//
// allows users to query status of capture data operation
// Status Codes:
// 1. OK - data capture successfully initiated
// 2. UNKNOWN - unknown error has occurred
// 3. INTERNAL - an internal error has occurred
// 4. NOT_FOUND - given capture id doesn't exist.
//
rpc GetCaptureDataStatus(GetCaptureDataStatusRequest) returns (GetCaptureDataStatusResponse);
```
------
#### [ Input ]
```
//
// request for GetCaptureDataStatus rpc call
//
message GetCaptureDataStatusRequest {
string capture_id = 1;
}
```
------
#### [ Output ]
```
enum CaptureDataStatus {
FAILURE = 0;
SUCCESS = 1;
IN_PROGRESS = 2;
NOT_FOUND = 3;
}
//
// response for GetCaptureDataStatus rpc call
//
message GetCaptureDataStatusResponse {
CaptureDataStatus status = 1;
}
```
------
## Predict
The `predict` API performs inference on a previously loaded model. It accepts a request in the form of a tensor that is directly fed into the neural network. The output is the output tensor (or scalar) from the model. This is a blocking call.
```
//
// perform inference on a model.
//
// Note:
// 1. users can chose to send the tensor data in the protobuf message or
// through a shared memory segment on a per tensor basis, the Predict
// method with handle the decode transparently.
// 2. serializing large tensors into the protobuf message can be quite expensive,
// based on our measurements it is recommended to use shared memory of
// tenors larger than 256KB.
// 3. SMEdge IPC server will not use shared memory for returning output tensors,
// i.e., the output tensor data will always send in byte form encoded
// in the tensors of PredictResponse.
// 4. currently SMEdge IPC server cannot handle concurrent predict calls, all
// these call will be serialized under the hood. this shall be addressed
// in a later release.
// Status Codes:
// 1. OK - prediction is successful
// 2. UNKNOWN - unknown error has occurred
// 3. INTERNAL - an internal error has occurred
// 4. NOT_FOUND - when model not found
// 5. INVALID_ARGUMENT - when tenors types mismatch
//
rpc Predict(PredictRequest) returns (PredictResponse);
```
------
#### [ Input ]
```
// request for Predict rpc call
//
message PredictRequest {
string name = 1;
repeated Tensor tensors = 2;
}
//
// Tensor represents a tensor, encoded as contiguous multi-dimensional array.
// tensor_metadata - represents metadata of the shared memory segment
// data_or_handle - represents the data of shared memory, this could be passed in two ways:
// a. send across the raw bytes of the multi-dimensional tensor array
// b. send a SharedMemoryHandle which contains the posix shared memory segment
// id and offset in bytes to location of multi-dimensional tensor array.
//
message Tensor {
TensorMetadata tensor_metadata = 1; //optional in the predict request
oneof data {
bytes byte_data = 4;
// will only be used for input tensors
SharedMemoryHandle shared_memory_handle = 5;
}
}
//
// Tensor represents a tensor, encoded as contiguous multi-dimensional array.
// tensor_metadata - represents metadata of the shared memory segment
// data_or_handle - represents the data of shared memory, this could be passed in two ways:
// a. send across the raw bytes of the multi-dimensional tensor array
// b. send a SharedMemoryHandle which contains the posix shared memory segment
// id and offset in bytes to location of multi-dimensional tensor array.
//
message Tensor {
TensorMetadata tensor_metadata = 1; //optional in the predict request
oneof data {
bytes byte_data = 4;
// will only be used for input tensors
SharedMemoryHandle shared_memory_handle = 5;
}
}
//
// TensorMetadata represents the metadata for a tensor
// name - name of the tensor
// data_type - data type of the tensor
// shape - array of dimensions of the tensor
//
message TensorMetadata {
string name = 1;
DataType data_type = 2;
repeated int32 shape = 3;
}
//
// SharedMemoryHandle represents a posix shared memory segment
// offset - offset in bytes from the start of the shared memory segment.
// segment_id - shared memory segment id corresponding to the posix shared memory segment.
// size - size in bytes of shared memory segment to use from the offset position.
//
message SharedMemoryHandle {
uint64 size = 1;
uint64 offset = 2;
uint64 segment_id = 3;
}
```
------
#### [ Output ]
**Note**
The `PredictResponse` only returns `Tensors` and not `SharedMemoryHandle`.
```
// response for Predict rpc call
//
message PredictResponse {
repeated Tensor tensors = 1;
}
```
------
# SageMaker Edge Manager end of life
Starting in April 26, 2024, you can no longer access Amazon SageMaker Edge Manager through the Amazon management console, make edge packaging jobs, and manage edge device fleets.
## FAQs
Use the following sections to get answers to commonly asked questions about the SageMaker Edge Manager end of life (EOL).
### Q: What happens to my Amazon SageMaker Edge Manager after the EOL date?
A: After April 26, 2024, all references to edge packaging jobs, devices, and device fleets are deleted from the Edge Manager service. You can no longer discover or access the Edge Manager service from your Amazon console and applications that call on the Edge Manager service APIs no longer work.
### Q: Will I be billed for Edge Manager resources remaining in my account after the EOL date?
A: Resources created by Edge Manager, such as edge packages inside Amazon S3 buckets, Amazon IoT things, and Amazon IAM roles, continue to exist on their respective services after April 26, 2024. To avoid being billed after Edge Manager is no longer supported, delete your resources. For more information on deleting your resources, see [Delete Edge Manager resources](#edge-eol-delete-resources).
### Q: How do I delete my Amazon SageMaker Edge Manager resources?
A: Resources created by Edge Manager, such as edge packages inside Amazon S3 buckets, Amazon IoT things, and Amazon IAM roles, continue to exist on their respective services after April 26, 2024. To avoid being billed after Edge Manager is no longer supported, delete your resources. For more information on deleting your resources, see [Delete Edge Manager resources](#edge-eol-delete-resources).
### Q: How can I continue deploying models on the edge?
A: We suggest you try one the following machine learning tools. For a cross-platform edge runtime, use [ONNX](https://onnxruntime.ai/). ONNX is a popular, well-maintained open-source solution that translates your models into instructions that many types of hardware can run, and is compatible with the latest ML frameworks. ONNX can be integrated into your SageMaker AI workflows as an automated step for your edge deployments.
For edge deployments and monitoring use Amazon IoT Greengrass V2. Amazon IoT Greengrass V2 has an extensible packaging and deployment mechanism that can fit models and applications at the edge. You can use the built-in MQTT channels to send model telemetry back for Amazon SageMaker Model Monitor or use the built-in permissions system to send data captured from the model back to Amazon Simple Storage Service (Amazon S3). If you don't or can't use Amazon IoT Greengrass V2, we suggest using MQTT and IoT Jobs (C/C\$1\$1 library) to create a lightweight OTA mechanism to deliver models.
We have prepared [sample code available at this GitHub repository](https://github.com/aws-samples/ml-edge-getting-started) to help you transition to these suggested tools.
## Delete Edge Manager resources
Resources created by Edge Manager continue to exist after April 26, 2024. To avoid billing, delete these resources.
To delete Amazon IoT Greengrass resources, do the following:
1. In the Amazon IoT Core console, choose **Greengrass devices** under **Manage**.
1. Choose **Components**.
1. Under **My components**, Edge Manager created components are in the format * SageMaker AIEdge (EdgePackagingJobName)*. Select the component you want to delete.
1. Then choose **Delete version**.
To delete a Amazon IoT role alias, do the following:
1. In the Amazon IoT Core console, choose **Security** under **Manage**.
1. Choose **Role aliases**.
1. Edge Manager created role aliases are in the format *SageMaker AIEdge-\$1DeviceFleetName\$1*. Select the role you want to delete.
1. Choose **Delete**.
To delete packaging jobs in Amazon S3 buckets, do the following:
1. In the SageMaker AI console, choose **Edge Inference**.
1. Choose **Edge packaging jobs**.
1. Select one of the edge packaging jobs. Copy the Amazon S3 URI under **Model artifact** in the **Output configuration** section.
1. In the Amazon S3 console, navigate to the corresponding location, and check if you need to delete the model artifact. To delete the model artifact, select the Amazon S3 object and choose **Delete**.