Troubleshooting Amazon IoT Greengrass V2 - Amazon IoT Greengrass
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Troubleshooting Amazon IoT Greengrass V2

Use the troubleshooting information and solutions in this section to help resolve issues with Amazon IoT Greengrass Version 2.

View Amazon IoT Greengrass Core software and component logs

The Amazon IoT Greengrass Core software writes logs to the local file system that you can use to view real-time information about the core device. You can also configure core devices to write logs to CloudWatch Logs, so you can remotely troubleshoot core devices. These logs can help you identify issues with components, deployments, and core devices. For more information, see Monitor Amazon IoT Greengrass logs.

Amazon IoT Greengrass Core software issues

Troubleshoot Amazon IoT Greengrass Core software issues.

Topics

Unable to set up core device

If the Amazon IoT Greengrass Core software installer fails and you aren't able to set up a core device, you might need to uninstall the software and try again. For more information, see Uninstall the Amazon IoT Greengrass Core software.

Unable to start the Amazon IoT Greengrass Core software as a system service

If the Amazon IoT Greengrass Core software fails to start, check the system service logs to identify the issue. One common issue is where Java isn't available on the PATH environment variable (Linux) or PATH system variable (Windows).

Unable to set up nucleus as a system service

You might see this error when the Amazon IoT Greengrass Core software installer fails to set up Amazon IoT Greengrass as a system service. On Linux devices, this error typically occurs if the core device doesn't have the systemd init system. The installer can successfully set up the Amazon IoT Greengrass Core software even if it fails to set up the system service.

Do one of the following:

Unable to connect to Amazon IoT Core

You might see this error when the Amazon IoT Greengrass Core software can't connect to Amazon IoT Core to retrieve deployment jobs, for example. Do the following:

  • Check that your core device can connect to the internet and Amazon IoT Core. For more information about the Amazon IoT Core endpoint to which your device connects, see Configure the Amazon IoT Greengrass Core software.

  • Check that your core device's Amazon IoT thing uses a certificate that allows the iot:Connect, iot:Publish, iot:Receive, and iot:Subscribe permissions.

  • If your core device uses a network proxy, check that your core device has a device role and that its role allows the iot:Connect, iot:Publish, iot:Receive, and iot:Subscribe permissions.

Out of memory error

This error typically occurs if your device doesn't have sufficient memory to allocate an object in the Java heap. On devices with limited memory, you might need to specify a maximum heap size to control memory allocation. For more information, see Control memory allocation with JVM options.

Unable to install Greengrass CLI

You might see the following console message when you use the --deploy-dev-tools argument in your installation command for Amazon IoT Greengrass Core.

Thing group exists, it could have existing deployment and devices, hence NOT creating deployment for Greengrass first party dev tools, please manually create a deployment if you wish to

This occurs when the Greengrass CLI component is not installed because your core device is a member of a thing group that has an existing deployment. If you see this message, you can manually deploy the Greengrass CLI component (aws.greengrass.Cli) to the device to install the Greengrass CLI. For more information, see Install the Greengrass CLI.

User root is not allowed to execute

You might see this error when the user that runs the Amazon IoT Greengrass Core software (typically root) doesn't have permission to run sudo with any user and any group. For the default ggc_user system user, this error looks like the following:

Sorry, user root is not allowed to execute <command> as ggc_user:ggc_group.

Check that your /etc/sudoers file gives the user permission to run sudo as other groups. The permission for the user in /etc/sudoers should look like the following example.

root ALL=(ALL:ALL) ALL

com.aws.greengrass.lifecyclemanager.GenericExternalService: Could not determine user/group to run with

You might see this error when the core device tries to run a component, and the Greengrass nucleus doesn't specify a default system user to use to run components.

To fix this issue, configure the Greengrass nucleus to specify the default system user that runs components. For more information, see Configure the user that runs components and Configure the default component user.

Failed to map segment from shared object: operation not permitted

You might see this error when the Amazon IoT Greengrass Core software fails to start because the /tmp folder is mounted with noexec permissions. The Amazon Common Runtime (CRT) library uses the /tmp folder by default.

Do one of the following:

  • Run the following command to remount the /tmp folder with exec permissions and try again.

    sudo mount -o remount,exec /tmp
  • If you run Greengrass nucleus v2.5.0 or later, you can set a JVM option to change the folder that the Amazon CRT library uses. You can specify the jvmOptions parameter in the Greengrass nucleus component configuration in a deployment or when you install the Amazon IoT Greengrass Core software. Replace /path/to/use with the path to a folder that the Amazon CRT library can use.

    { "jvmOptions": "-Daws.crt.lib.dir=\"/path/to/use\"" }

Failed to set up Windows service

You might see this error if you install the Amazon IoT Greengrass Core software on a Microsoft Windows 2016 device. The Amazon IoT Greengrass Core software is not supported on Windows 2016, for a list of supported operating systems, see Supported platforms.

If you must use Windows 2016, you can do the following:

  1. Unzip the downloaded Amazon IoT Greengrass Core installation archive

  2. In the Greengrass directory open the bin/greengrass.xml.template file.

  3. Add the <autoRefresh> tag to the end of the file just before the </service> tag.

    </log> <autoRefresh>false</autoRefresh> </service>

com.aws.greengrass.util.exceptions.TLSAuthException: Failed to get trust manager

You might see this error when you install the Amazon IoT Greengrass Core software without a root certificate authority (CA) file.

2022-06-05T10:00:39.556Z [INFO] (main) com.aws.greengrass.lifecyclemanager.Kernel: service-loaded. {serviceName=DeploymentService} 2022-06-05T10:00:39.943Z [WARN] (main) com.aws.greengrass.componentmanager.ClientConfigurationUtils: configure-greengrass-mutual-auth. Error during configure greengrass client mutual auth. {} com.aws.greengrass.util.exceptions.TLSAuthException: Failed to get trust manager

Check that you specify a valid root CA file with the rootCaPath parameter in the configuration file that you provide to the installer. For more information, see Install the Amazon IoT Greengrass Core software.

com.aws.greengrass.deployment.IotJobsHelper: No connection available during subscribing to Iot Jobs descriptions topic. Will retry in sometime

You might see this warning message when the core device can't connect to Amazon IoT Core to subscribe to deployment job notifications. Do the following:

  • Check that the core device is connected to the internet and can reach the Amazon IoT data endpoint that you configured. For more information about endpoints that core devices use, see Allow device traffic through a proxy or firewall.

  • Check the Greengrass logs for other errors that reveal other root causes.

software.amazon.awssdk.services.iam.model.IamException: The security token included in the request is invalid

You might see this error when you install the Amazon IoT Greengrass Core software with automatic provisioning, and the installer uses an Amazon session token that isn't valid. Do the following:

  • If you use temporary security credentials, check that the session token is correct and that you are copying and pasting the complete session token.

  • If you use long-term security credentials, check that the device doesn't have a session token from a time where you previously used temporary credentials. Do the following:

    1. Run the following command to unset the session token environment variable.

      Linux or Unix
      unset AWS_SESSION_TOKEN
      Windows Command Prompt (CMD)
      set AWS_SESSION_TOKEN=
      PowerShell
      Remove-Item Env:\AWS_SESSION_TOKEN
    2. Check if the Amazon credentials file, ~/.aws/credentials, contains a session token, aws_session_token. If so, remove that line from the file.

      aws_session_token = AQoEXAMPLEH4aoAH0gNCAPyJxz4BlCFFxWNE1OPTgk5TthT+FvwqnKwRcOIfrRh3c/LTo6UDdyJwOOvEVPvLXCrrrUtdnniCEXAMPLE/IvU1dYUg2RVAJBanLiHb4IgRmpRV3zrkuWJOgQs8IZZaIv2BXIa2R4Olgk

You can also install the Amazon IoT Greengrass Core software without providing Amazon credentials. For more information, see Install Amazon IoT Greengrass Core software with manual resource provisioning or Install Amazon IoT Greengrass Core software with Amazon IoT fleet provisioning.

software.amazon.awssdk.services.iot.model.IotException: User: <user> is not authorized to perform: iot:GetPolicy

You might see this error when you install the Amazon IoT Greengrass Core software with automatic provisioning, and the installer uses Amazon credentials that don't have the required permissions. For more information about the permissions that are required, see Minimal IAM policy for installer to provision resources.

Check the permissions for the credentials' IAM identity, and grant the IAM identity any required permissions that are missing.

Error: com.aws.greengrass.shadowmanager.sync.model.FullShadowSyncRequest: Could not execute cloud shadow get request

You might see this error when you use the shadow manager component to sync device shadows with Amazon IoT Core. The HTTP 403 status code indicates that this error occurred because the core device's Amazon IoT policy doesn't grant permission to call GetThingShadow.

com.aws.greengrass.shadowmanager.sync.model.FullShadowSyncRequest: Could not execute cloud shadow get request. {thing name=MyGreengrassCore, shadow name=MyShadow} 2021-07-14T21:09:02.456Z [ERROR] (pool-2-thread-109) com.aws.greengrass.shadowmanager.sync.SyncHandler: sync. Skipping sync request. {thing name=MyGreengrassCore, shadow name=MyShadow} com.aws.greengrass.shadowmanager.exception.SkipSyncRequestException: software.amazon.awssdk.services.iotdataplane.model.IotDataPlaneException: null (Service: IotDataPlane, Status Code: 403, Request ID: f6e713ba-1b01-414c-7b78-5beb3f3ad8f6, Extended Request ID: null)

To sync local shadows with Amazon IoT Core, the core device's Amazon IoT policy must grant the following permissions:

  • iot:GetThingShadow

  • iot:UpdateThingShadow

  • iot:DeleteThingShadow

Check the core device's Amazon IoT policy, and add any required permissions that are missing. For more information, see the following:

Operation aws.greengrass#<operation> is not supported by Greengrass

You might see this error when you use an interprocess communication (IPC) operation in a custom Greengrass component, and the required Amazon-provided component isn't installed on the core device.

To fix this issue, add the required component as a dependency in your component recipe, so the Amazon IoT Greengrass Core software installed the required component when you deploy your component.

java.io.FileNotFoundException: <stream-manager-store-root-dir>/stream_manager_metadata_store (Permission denied)

You might see this error in the stream manager log file (aws.greengrass.StreamManager.log) when you configure stream manager to use a root folder that doesn't exist or have the correct permissions. For more information about how to configure this folder, see stream manager configuration.

com.aws.greengrass.security.provider.pkcs11.PKCS11CryptoKeyService: Private key or certificate with label <label> does not exist

This error occurs when the PKCS#11 provider component can't find or load the private key or certificate that you specify when you configure the Amazon IoT Greengrass Core software to use a hardware security module (HSM). Do the following:

  • Check that the private key and certificate are stored in the HSM using the slot, user PIN, and object label that you configure the Amazon IoT Greengrass Core software to use.

  • Check that the private key and certificate use the same object label in the HSM.

  • If your HSM supports object IDs, check that the private key and certificate use the same object ID in the HSM.

Check the documentation for your HSM to learn how to query details about the security tokens in the HSM. If you need to change the slot, object label, or object ID for a security token, check the documentation for your HSM to learn how to do so.

software.amazon.awssdk.services.secretsmanager.model.SecretsManagerException: User: <user> is not authorized to perform: secretsmanager:GetSecretValue on resource: <arn>

This error can occur when you use the secret manager component to deploy an Amazon Secrets Manager secret. If the core device's token exchange IAM role doesn't grant permission to get the secret, the deployment fails and the Greengrass logs include this error.

To authorize a core device to download a secret
  1. Add the secretsmanager:GetSecretValue permission to the core device's token exchange role. The following example policy statement grants permission to get the value of a secret.

    { "Effect": "Allow", "Action": [ "secretsmanager:GetSecretValue" ], "Resource": [ "arn:aws-cn:secretsmanager:us-west-2:123456789012:secret:MyGreengrassSecret-abcdef" ] }

    For more information, see Authorize core devices to interact with Amazon services.

  2. Reapply the deployment to the core device. Do one of the following:

    • Revise the deployment without any changes. The core device tries to download the secret again when it receives the revised deployment. For more information, see Revise deployments.

    • Restart the Amazon IoT Greengrass Core software to retry the deployment. For more information, see Run the Amazon IoT Greengrass Core software

    The deployment succeeds if secret manager downloads the secret successfully.

software.amazon.awssdk.services.secretsmanager.model.SecretsManagerException: Access to KMS is not allowed

This error can occur when you use the secret manager component to deploy an Amazon Secrets Manager secret that is encrypted by an Amazon Key Management Service key. If the core device's token exchange IAM role doesn't grant permission to decrypt the secret, the deployment fails and the Greengrass logs include this error.

To fix the issue, add the kms:Decrypt permission to the core device's token exchange role. For more information, see the following:

java.lang.NoClassDefFoundError: com/aws/greengrass/security/CryptoKeySpi

You might see this error when you try to install the Amazon IoT Greengrass Core software with hardware security and you use an earlier Greengrass nucleus version that doesn't support hardware security integration. To use hardware security integration, you must use Greengrass nucleus v2.5.3 or later.

com.aws.greengrass.security.provider.pkcs11.PKCS11CryptoKeyService: CKR_OPERATION_NOT_INITIALIZED

You might see this error when you use the TPM2 library when running Amazon IoT Greengrass Core as a system service.

This error indicates that you need to add an environment variable that provides the location of the PKCS#11 store in the Amazon IoT Greengrass Core systemd service file.

For more information, see the Requirements section of the PKCS#11 provider component documentation.

Greengrass core device stuck on nucleus v2.12.3

If your Greengrass core device won't revise your deployment from nucleus version 2.12.3, you might need to download and replace the Greengrass.jar file with Greengrass nucleus version 2.12.2. Do the following:

  1. On your Greengrass core device, run the following command to stop the Greengrass Core software.

    Linux or Unix
    sudo systemctl stop greengrass
    Windows Command Prompt (CMD)
    sc stop "greengrass"
    PowerShell
    Stop-Service -Name "greengrass"
  2. On your core device, download the Amazon IoT Greengrass software to a file named greengrass-2.12.2.zip.

    Linux or Unix
    curl -s https://d2s8p88vqu9w66.cloudfront.net/releases/greengrass-2.12.2.zip > greengrass-2.12.2.zip
    Windows Command Prompt (CMD)
    curl -s https://d2s8p88vqu9w66.cloudfront.net/releases/greengrass-2.12.2.zip > greengrass-2.12.2.zip
    PowerShell
    iwr -Uri https://d2s8p88vqu9w66.cloudfront.net/releases/greengrass-2.12.2.zip -OutFile greengrass-2.12.2.zip
  3. Unzip the Amazon IoT Greengrass Core software to a folder on your device. Replace GreengrassInstaller with the folder that you want to use.

    Linux or Unix
    unzip greengrass-2.12.2.zip -d GreengrassInstaller && rm greengrass-2.12.2.zip
    Windows Command Prompt (CMD)
    mkdir GreengrassInstaller && tar -xf greengrass-2.12.2.zip -C GreengrassInstaller && del greengrass-2.12.2.zip
    PowerShell
    Expand-Archive -Path greengrass-2.12.2.zip -DestinationPath .\\GreengrassInstaller rm greengrass-2.12.2.zip
  4. Run the following command to override the nucleus version 2.12.3 Greengrass JAR file with the nucleus version 2.12.2 Greengrass JAR file.

    Linux or Unix
    sudo cp ./GreengrassInstaller/lib/Greengrass.jar /greengrass/v2/packages/artifacts-unarchived/aws.greengrass.Nucleus/2.12.3/aws.greengrass.nucleus/lib
    Windows Command Prompt (CMD)
    robocopy ./GreengrassInstaller/lib/Greengrass.jar /greengrass/v2/packages/artifacts-unarchived/aws.greengrass.Nucleus/2.12.3/aws.greengrass.nucleus/lib /E
    PowerShell
    cp -Path ./GreengrassInstaller/lib/Greengrass.jar -Destination /greengrass/v2/packages/artifacts-unarchived/aws.greengrass.Nucleus/2.12.3/aws.greengrass.nucleus/lib
  5. Run the following command to start the Greengrass Core software.

    Linux or Unix
    sudo systemctl start greengrass
    Windows Command Prompt (CMD)
    sc start "greengrass"
    PowerShell
    Start-Service -Name "greengrass"

Amazon IoT Greengrass cloud issues

Use the following information to troubleshoot issues with the Amazon IoT Greengrass console and API. Each entry corresponds to an error message that you might see when you perform an action.

An error occurred (AccessDeniedException) when calling the CreateComponentVersion operation: User: arn:aws-cn:iam::123456789012:user/<username> is not authorized to perform: null

You might see this error when you create a component version from the Amazon IoT Greengrass console or with the CreateComponentVersion operation.

This error indicates that your recipe isn't valid JSON or YAML. Check the syntax of your recipe, fix any syntax issues, and try again. You can use an online JSON or YAML syntax checker to identify syntax issues in your recipe.

Invalid Input: Encountered following errors in Artifacts: {<s3ArtifactUri> = Specified artifact resource cannot be accessed}

You might see this error when you create a component version from the Amazon IoT Greengrass console or with the CreateComponentVersion operation. This error indicates that an S3 artifact in the component recipe isn't valid.

Do the following:

  • Check that the S3 bucket is in the same Amazon Web Services Region where you create the component. Amazon IoT Greengrass doesn't support cross-Region requests for component artifacts.

  • Check that the artifact URI is a valid S3 object URL, and check the artifact exists at that S3 object URL.

  • Check that your Amazon Web Services account has permission to access the artifact at its S3 object URL.

INACTIVE deployment status

You might get an INACTIVE deployment status when you call the ListDeployments API without the required dependent Amazon IoT policies. You must have the necessary permissions in order to get an accurate deployment status. You can find the dependent actions by looking in the Actions defined by Amazon IoT Greengrass V2 and following the permissions needed for ListDeployments. Without the required dependent Amazon IoT permissions, you will still see the deployment status but you might see an inaccurate deployment status of INACTIVE.

Core device deployment issues

Troubleshoot deployment issues on Greengrass core devices. Each entry corresponds to a log message that you might see on your core device.

Topics

Error: com.aws.greengrass.componentmanager.exceptions.PackageDownloadException: Failed to download artifact

You might see this error when the Amazon IoT Greengrass Core software fails to download a component artifact when the core device applies a deployment. The deployment fails as a result of this error.

When you receive this error, the log also includes a stack trace that you can use to identify the specific issue. Each of the following entries corresponds to a message that you might see in the stack trace of the Failed to download artifact error message.

software.amazon.awssdk.services.s3.model.S3Exception: null (Service: S3, Status Code: 403, Request ID: null, ...)

The PackageDownloadException error might include this stack trace in the following cases:

  • The component artifact isn't available at the S3 object URL that you specify in the component's recipe. Check that you uploaded the artifact to the S3 bucket and that the artifact URI matches the S3 object URL of the artifact in the bucket.

  • The core device's token exchange role doesn't allow the Amazon IoT Greengrass Core software to download the component artifact from the S3 object URL that you specify in the component's recipe. Check that the token exchange role allows s3:GetObject for the S3 object URL where the artifact is available.

software.amazon.awssdk.services.s3.model.S3Exception: Access Denied (Service: S3, Status Code: 403, Request ID: <requestID>

The PackageDownloadException error might include this stack trace when the core device doesn't have permission to call s3:GetBucketLocation. The error message also includes the following message.

reason: Failed to determine S3 bucket location

Check that the core device's token exchange role allows s3:GetBucketLocation for the S3 bucket where the artifact is available.

Error: com.aws.greengrass.componentmanager.exceptions.ArtifactChecksumMismatchException: Integrity check for downloaded artifact failed. Probably due to file corruption.

You might see this error when the Amazon IoT Greengrass Core software fails to download a component artifact when the core device applies a deployment. The deployment fails because the downloaded artifact file's checksum doesn't match the checksum that Amazon IoT Greengrass calculated when you created the component.

Do the following:

  • Check if the artifact file changed in the S3 bucket where you host it. If the file changed since you created the component, restore it to the previous version that the core device expects. If you can't restore the file to its previous version, or if you want to use the new version of the file, create a new version of the component with the artifact file.

  • Check your core device's internet connection. This error can occur if the artifact file becomes corrupted while it downloads. Create a new deployment and try again.

Error: com.aws.greengrass.componentmanager.exceptions.NoAvailableComponentVersionException: Failed to negotiate component <name> version with cloud and no local applicable version satisfying requirement <requirements>

You might see this error when a core device can't find a component version that meets the requirements of the deployments for that core device. The core device checks for the component in the Amazon IoT Greengrass service and on the local device. The error message includes each deployment's target and that deployment's version requirements for the component. The deployment target can be a thing, a thing group, or LOCAL_DEPLOYMENT, which represents the local deployment on the core device.

This issue can occur in the following cases:

  • The core device is the target of multiple deployments that have conflicting component version requirements. For example, the core device might be the target of multiple deployments that include a com.example.HelloWorld component, where one deployment requires version 1.0.0 and the other requires version 1.0.1. It's impossible to have a component that meets both requirements, so the deployment fails.

  • The component version doesn't exist in the Amazon IoT Greengrass service or on the local device. The component might have been deleted, for example.

  • There exists component versions that meet the version requirements, but none are compatible with the core device's platform.

  • The core device's Amazon IoT policy doesn't grant the greengrass:ResolveComponentCandidates permission. Look for Status Code: 403 in the error log to identify this issue. To resolve this issue, add the greengrass:ResolveComponentCandidates permission to the core device's Amazon IoT policy. For more information, see Minimal Amazon IoT policy for Amazon IoT Greengrass V2 core devices.

To resolve this issue, revise the deployments to include compatible component versions or remove incompatible ones. For more information about how to revise cloud deployments, see Revise deployments. For more information about how to revise local deployments, see the Amazon IoT Greengrass CLI deployment create command.

software.amazon.awssdk.services.greengrassv2data.model.ResourceNotFoundException: The latest version of Component <componentName> doesn't claim platform <coreDevicePlatform> compatibility

You might see this error when you deploy a component to a core device, and the component doesn't list a platform that is compatible with the core device's platform. Do one of the following:

com.aws.greengrass.componentmanager.exceptions.PackagingException: The deployment attempts to update the nucleus from aws.greengrass.Nucleus-<version> to aws.greengrass.Nucleus-<version> but no component of type nucleus was included as target component

You might see this error when you deploy a component that depends on the Greengrass nucleus, and the core device runs an earlier Greengrass nucleus version than the latest minor version available. This error occurs because the Amazon IoT Greengrass Core software tries to automatically update components to the latest compatible version. However, the Amazon IoT Greengrass Core software prevents the Greengrass nucleus from updating to a new minor version, because several Amazon-provided components depend on specific minor versions of the Greengrass nucleus. For more information, see Greengrass nucleus update behavior.

You must revise the deployment to specify the Greengrass nucleus version that you want to use. Do one of the following:

  • Revise the deployment to specify the Greengrass nucleus version that the core device currently runs.

  • Revise the deployment to specify a later minor version of the Greengrass nucleus. If you choose this option, you must also update the versions of all Amazon-provided components that depend on specific minor versions of the Greengrass nucleus. For more information, see Amazon-provided components.

Error: com.aws.greengrass.deployment.exceptions.DeploymentException: Unable to process deployment. Greengrass launch directory is not set up or Greengrass is not set up as a system service

You might see this error when you move a Greengrass device from one thing group to another, and then back to the original group with deployments that require Greengrass to restart.

To resolve this issue, recreate the launch directory for the device. We also strongly recommend upgrading to version 2.9.6 or later of the Greengrass nucleus.

The following is a Linux script to recreate the launch directory. Save the script in a file called fix_directory.sh.

#!/bin/bash set -e GG_ROOT=$1 GG_VERSION=$2 CURRENT="$GG_ROOT/alts/current" if [ ! -L "$CURRENT" ]; then mkdir -p $GG_ROOT/alts/directory_fix echo "Relinking $GG_ROOT/alts/directory_fix to $CURRENT" ln -sf $GG_ROOT/alts/directory_fix $CURRENT fi TARGET=$(readlink $CURRENT) if [[ ! -d "$TARGET" ]]; then echo "Creating directory: $TARGET" mkdir -p "$TARGET" fi DISTRO_LINK="$TARGET/distro" DISTRO="$GG_ROOT/packages/artifacts-unarchived/aws.greengrass.Nucleus/$GG_VERSION/aws.greengrass.nucleus/" echo "Relinking Nucleus artifacts to $DISTRO_LINK" ln -sf $DISTRO $DISTRO_LINK

To run the script, execute the following command:

[root@ip-172-31-27-165 ~]# ./fix_directory.sh /greengrass/v2 2.9.5 Relinking /greengrass/v2/alts/directory_fix to /greengrass/v2/alts/current Relinking Nucleus artifacts to /greengrass/v2/alts/directory_fix/distro

Info: com.aws.greengrass.deployment.exceptions.RetryableDeploymentDocumentDownloadException: Greengrass Cloud Service returned an error when getting full deployment configuration

You might see this error when the core device receives a large deployment document, which is a deployment document larger than 7 KB (for deployments that target things) or 31 KB (for deployments that target thing groups). To retrieve a large deployment document, a core device's Amazon IoT policy must allow the greengrass:GetDeploymentConfiguration permission. This error can occur when the core device doesn't have this permission. When this error occurs, the deployment retries indefinitely, and its status is In progress (IN_PROGRESS).

To resolve this issue, add the greengrass:GetDeploymentConfiguration permission to the core device's Amazon IoT policy. For more information, see Update a core device's Amazon IoT policy.

Warn: com.aws.greengrass.deployment.DeploymentService: Failed to get thing group hierarchy

You might see this warning when the core device receives a deployment and the core device's Amazon IoT policy doesn't allow the greengrass:ListThingGroupsForCoreDevice permission. When you create a deployment, the core device uses this permission to identify its thing groups and remove components for any thing groups from which you removed the core device. If the core device runs Greengrass nucleus v2.5.0, the deployment fails. If the core device runs Greengrass nucleus v2.5.1 or later, the deployment proceeds but doesn't remove components. For more information about thing group removal behavior, see Deploy Amazon IoT Greengrass components to devices.

To update the core device's behavior to remove components for thing groups from which you remove the core device, add the greengrass:ListThingGroupsForCoreDevice permission to the core device's Amazon IoT policy. For more information, see Update a core device's Amazon IoT policy.

Info: com.aws.greengrass.deployment.DeploymentDocumentDownloader: Calling Greengrass cloud to get full deployment configuration

You might see this information message printed multiple times without an error, because the core device logs the error at the DEBUG log level. This issue can occur when the core device receives a large deployment document. When this issue occurs, the deployment retries indefinitely, and its status is In progress (IN_PROGRESS). For more information about how to resolve this issue, see this troubleshooting entry.

Caused by: software.amazon.awssdk.services.greengrassv2data.model.GreengrassV2DataException: null (Service: GreengrassV2Data, Status Code: 403, Request ID: <some_request_id>, Extended Request ID: null)

You might see this error when a dataplane API doesn't have iot:Connect permission. If you don't have the correct policy, you'll receive a GreengrassV2DataException: 403. To create a permission policy, follow these instructions: Create an Amazon IoT policy.

Core device component issues

Troubleshoot Greengrass component issues on core devices.

Warn: '<command>' is not recognized as an internal or external command

You might see this error in a Greengrass component's logs when the Amazon IoT Greengrass Core software fails to run a command in the component's lifecycle script. The component's state becomes BROKEN as a result of this error. This error can occur if the system user that runs the component, such as ggc_user, can't find the command's executable in the folders in the PATH.

On Windows devices, check that the folder that contains the executable is in the PATH for the system user that runs the component. If it's missing from the PATH, do one of the following:

  • Add the executable's folder to the PATH system variable, which is available to all users. Then, restart the component.

    If you run Greengrass nucleus 2.5.0, after you update the PATH system variable, you must restart the Amazon IoT Greengrass Core software to run components with the updated PATH. If the Amazon IoT Greengrass Core software doesn't use the updated PATH after you restart the software, restart the device and try again. For more information, see Run the Amazon IoT Greengrass Core software.

  • Add the executable's folder to the PATH user variable for the system user that runs the component.

Python script doesn't log messages

Greengrass core devices collect logs that you can use to identify issues with components. If your Python script's stdout and stderr messages don't appear in your component logs, you might need to flush the buffer or disable buffering for these standard output streams in Python. Do any of the following:

  • Run Python with the -u argument to disable buffering on stdout and stderr.

    Linux or Unix
    python3 -u hello_world.py
    Windows
    py -3 -u hello_world.py
  • Use Setenv in your component's recipe to set the PYTHONUNBUFFERED environment variable to a non-empty string. This environment variable disables buffering on stdout and stderr.

  • Flush the buffer for the stdout or stderr streams. Do one of the following:

    • Flush a message when you print.

      import sys print('Hello, error!', file=sys.stderr, flush=True)
    • Flush a message after you print. You can send multiple messages before you flush the stream.

      import sys print('Hello, error!', file=sys.stderr) sys.stderr.flush()

For more information about how to verify that your Python script outputs log messages, see Monitor Amazon IoT Greengrass logs.

Component configuration doesn't update when changing default configuration

When you change the DefaultConfiguration in a component's recipe, the new default configuration won't replace the component's existing configuration during a deployment. To apply the new default configuration, you must reset the component's configuration to its default settings. When you deploy the component, specify a single empty string as the reset update.

Console
Reset paths
[""]
Amazon CLI

The following command creates a deployment to a core device.

aws greengrassv2 create-deployment --cli-input-json file://reset-configuration-deployment.json

The reset-configuration-deployment.json file contains the following JSON document.

{ "targetArn": "arn:aws-cn:iot:us-west-2:123456789012:thing/MyGreengrassCore", "deploymentName": "Deployment for MyGreengrassCore", "components": { "com.example.HelloWorld": { "componentVersion": "1.0.0", "configurationUpdate": {, "reset": [""] } } } }
Greengrass CLI

The following Greengrass CLI command creates a local deployment on a core device.

sudo greengrass-cli deployment create \ --recipeDir recipes \ --artifactDir artifacts \ --merge "com.example.HelloWorld=1.0.0" \ --update-config reset-configuration-deployment.json

The reset-configuration-deployment.json file contains the following JSON document.

{ "com.example.HelloWorld": { "RESET": [""] } }

awsiot.greengrasscoreipc.model.UnauthorizedError

You might see this error in a Greengrass component's logs when the component doesn't have permission to perform an IPC operation on a resource. To grant a component permission to call an IPC operation, define an IPC authorization policy in the component's configuration. For more information, see Authorize components to perform IPC operations.

Tip

If you change the DefaultConfiguration in a component's recipe, you must reset the component's configuration to its new default configuration. When you deploy the component, specify a single empty string as the reset update. For more information, see Component configuration doesn't update when changing default configuration.

com.aws.greengrass.authorization.exceptions.AuthorizationException: Duplicate policy ID "<id>" for principal "<componentList>"

You might see this error if multiple IPC authorization policies, including across all components on the core device, use the same policy ID.

Check your components' IPC authorization policies, fix any duplicates, and try again. To create unique policy IDs, we recommend that you combine the component name, IPC service name, and a counter. For more information, see Authorize components to perform IPC operations.

Tip

If you change the DefaultConfiguration in a component's recipe, you must reset the component's configuration to its new default configuration. When you deploy the component, specify a single empty string as the reset update. For more information, see Component configuration doesn't update when changing default configuration.

com.aws.greengrass.tes.CredentialRequestHandler: Error in retrieving AwsCredentials from TES (HTTP 400)

You might see this error when a core device can't get Amazon credentials from the token exchange service. The HTTP 400 status code indicates that this error occurred because the core device's token exchange IAM role doesn't exist or doesn't have a trust relationship that allows the Amazon IoT credentials provider to assume it.

Do the following:

  1. Identify the token exchange role that the core device uses. The error message includes the core device's Amazon IoT role alias, which points to the token exchange role. Run the following command on your development computer, and replace MyGreengrassCoreTokenExchangeRoleAlias with the name of the Amazon IoT role alias from the error message.

    aws iot describe-role-alias --role-alias MyGreengrassCoreTokenExchangeRoleAlias

    The response includes the Amazon Resource Name (ARN) of the token exchange IAM role.

    { "roleAliasDescription": { "roleAlias": "MyGreengrassCoreTokenExchangeRoleAlias", "roleAliasArn": "arn:aws-cn:iot:us-west-2:123456789012:rolealias/MyGreengrassCoreTokenExchangeRoleAlias", "roleArn": "arn:aws-cn:iam::123456789012:role/MyGreengrassV2TokenExchangeRole", "owner": "123456789012", "credentialDurationSeconds": 3600, "creationDate": "2021-02-05T16:46:18.042000-08:00", "lastModifiedDate": "2021-02-05T16:46:18.042000-08:00" } }
  2. Check that the role exists. Run the following command, and replace MyGreengrassV2TokenExchangeRole with the name of the token exchange role.

    aws iam get-role --role-name MyGreengrassV2TokenExchangeRole

    If the command returns a NoSuchEntity error, the role doesn't exist, and you must create it. For more information about how to create and configure this role, see Authorize core devices to interact with Amazon services.

  3. Check that the role has a trust relationship that allows the Amazon IoT credentials provider to assume it. The response from the previous step contains an AssumeRolePolicyDocument, which defines the role's trust relationships. The role must define a trust relationship that allows credentials.iot.amazonaws.com to assume it. This document should look similar to the following example.

    { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "credentials.iot.amazonaws.com" }, "Action": "sts:AssumeRole" } ] }

    If the role's trust relationships don't allow credentials.iot.amazonaws.com to assume it, you must add this trust relationship to the role. For more information, see Modifying a role in the Amazon Identity and Access Management User Guide.

com.aws.greengrass.tes.CredentialRequestHandler: Error in retrieving AwsCredentials from TES (HTTP 403)

You might see this error when a core device can't get Amazon credentials from the token exchange service. The HTTP 403 status code indicates that this error occurred because the core device's Amazon IoT policies don't grant the iot:AssumeRoleWithCertificate permission for the core device's Amazon IoT role alias.

Review the core device's Amazon IoT policies, and add the iot:AssumeRoleWithCertificate permission for the core device's Amazon IoT role alias. The error message includes the core device's current Amazon IoT role alias. For more information about this permission and how to update the core device's Amazon IoT policies, see Minimal Amazon IoT policy for Amazon IoT Greengrass V2 core devices and Update a core device's Amazon IoT policy.

com.aws.greengrass.tes.CredentialsProviderError: Could not load credentials from any providers

You might see this error when the component tries to request Amazon credentials and can't connect to the token exchange service.

Do the following:

  • Check that the component declares a dependency on the token exchange service component, aws.greengrass.TokenExchangeService. If it doesn't, add the dependency and redeploy the component.

  • If the component runs in docker, ensure that you apply the right network settings and environment variables, according to Use Amazon credentials in Docker container components (Linux).

  • If the component is written in NodeJS, set dns.setDefaultResultOrder to ipv4first.

  • Inspect /etc/hosts for an entry that starts with ::1 and contains localhost. Remove the entry to see if it caused the component to connect to the token exchange service at the wrong address.

Received error when attempting to retrieve ECS metadata: Could not connect to the endpoint URL: "<tokenExchangeServiceEndpoint>"

You might see this error when the component doesn't run the token exchange service and a component tries to request Amazon credentials.

Do the following:

  • Check that the component declares a dependency on the token exchange service component, aws.greengrass.TokenExchangeService. If it doesn't, add the dependency and redeploy the component.

  • Check whether the component uses Amazon credentials in its install lifecycle. Amazon IoT Greengrass doesn't guarantee the availability of the token exchange service during the install lifecycle. Update the component to move the code that uses Amazon credentials into the startup or run lifecycle, then redeploy the component.

copyFrom: <configurationPath> is already a container, not a leaf

You might see this error when you change a configuration value from a container type (a list or object) to a non-container type (a string, number, or Boolean). Do the following:

  1. Check the component's recipe to see whether its default configuration sets that configuration value to a list or an object. If so, remove or change that configuration value.

  2. Create a deployment to reset that configuration value to its default value. For more information, see Create deployments and Update component configurations.

Then, you can set that configuration value to a string, number, or Boolean.

com.aws.greengrass.componentmanager.plugins.docker.exceptions.DockerLoginException: Error logging into the registry using credentials - 'The stub received bad data.'

You might see this error in the Greengrass nucleus logs when the Docker application manager component tries to download a Docker image from a private repository in Amazon Elastic Container Registry (Amazon ECR). This error occurs if you use the wincred Docker credential helper (docker-credential-wincred). As a result, Amazon ECR is unable to store the login credentials.

Take one of the following actions:

  • If you don't use the wincred Docker credential helper, remove the docker-credential-wincred program from the core device.

  • If you use the wincred Docker credential helper, do the following:

    1. Rename the docker-credential-wincred program on the core device. Replace wincred with a new name for the Windows Docker credential helper. For example, you can rename it to docker-credential-wincredreal.

    2. Update the credsStore option in the Docker configuration file (.docker/config.json) to use the new name for the Windows Docker credential helper. For example, if you renamed the program to docker-credential-wincredreal, update the credsStore option to wincredreal.

      { "credsStore": "wincredreal" }

java.io.IOException: Cannot run program "cmd" ...: [LogonUser] The password for this account has expired.

You might see this error on a Windows core device when the system user that runs the component's processes, such as ggc_user, has an expired password. As a result, the Amazon IoT Greengrass Core software is unable to run component processes as that system user.

To update a Greengrass system user's password
  1. Run the following command as an administrator to set the user's password. Replace ggc_user with the system user, and replace password with the password to set.

    net user ggc_user password
  2. Use the PsExec utility to store the user's new password in the Credential Manager instance for the LocalSystem account. Replace password with the user's password that you set.

    psexec -s cmd /c cmdkey /generic:ggc_user /user:ggc_user /pass:password
Tip

Depending on your Windows configuration, the user's password might be set to expire at a date in the future. To ensure your Greengrass applications continue to operate, track when the password expires, and update it before it expires. You can also set the user's password to never expire.

  • To check when a user and its password expire, run the following command.

    net user ggc_user | findstr /C:expires
  • To set a user's password to never expire, run the following command.

    wmic UserAccount where "Name='ggc_user'" set PasswordExpires=False
  • If you’re using Windows 10 or later where the wmic command is deprecated, run the following PowerShell command.

    Get-CimInstance -Query "SELECT * from Win32_UserAccount WHERE name = 'ggc_user'" | Set-CimInstance -Property @{PasswordExpires="False"}

aws.greengrass.StreamManager: Instant exceeds minimum or maximum instant

When you upgrade stream manager v2.0.7 to a version between v2.0.8 and v2.0.11, you might see the following error in the stream manager component's logs if the component fails to start.

2021-07-16T00:54:58.568Z [INFO] (Copier) aws.greengrass.StreamManager: stdout. Caused by: com.fasterxml.jackson.databind.JsonMappingException: Instant exceeds minimum or maximum instant (through reference chain: com.amazonaws.iot.greengrass.streammanager.export.PersistedSuccessExportStatesV1["lastExportTime"]). {scriptName=services.aws.greengrass.StreamManager.lifecycle.startup.script, serviceName=aws.greengrass.StreamManager, currentState=STARTING} 2021-07-16T00:54:58.579Z [INFO] (Copier) aws.greengrass.StreamManager: stdout. Caused by: java.time.DateTimeException: Instant exceeds minimum or maximum instant. {scriptName=services.aws.greengrass.StreamManager.lifecycle.startup.script, serviceName=aws.greengrass.StreamManager, currentState=STARTING}

If you deployed stream manager v2.0.7 and you want to upgrade to a later version, you must upgrade to stream manager v2.0.12 directly. For more information about the stream manager component, see Stream manager.

Core device Lambda function component issues

Troubleshoot Lambda function component issues on core devices.

The following cgroup subsystems are not mounted: devices, memory

You might see this error when you run a containerized Lambda function in the following cases:

  • The core device doesn't have cgroup v1 enabled for the memory or device cgroups.

  • The core device has cgroups v2 enabled. Greengrass Lambda functions require cgroups v1, and cgroups v1 and v2 are mutually exclusive.

To enable cgroups v1, boot the device with the following Linux kernel parameters.

cgroup_enable=memory cgroup_memory=1 systemd.unified_cgroup_hierarchy=0
Tip

On a Raspberry Pi, edit the /boot/cmdline.txt file to set the device's kernel parameters.

ipc_client.py:64,HTTP Error 400:Bad Request, b'No subscription exists for the source <label-or-lambda-arn> and subject <label-or-lambda-arn>

You might see this error when you run a V1 Lambda function, which uses the Amazon IoT Greengrass Core SDK, on a V2 core device without specifying a subscription in the legacy subscription router component. To fix this issue, deploy and configure the legacy subscription router to specify the required subscriptions. For more information, see Import V1 Lambda functions.

Component version discontinued

You might see a notification on your Personal Health Dashboard (PHD) when a component version on your core device is discontinued. The component version sends this notification to your PHD within 60 minutes of being discontinued.

To see which deployments you need to revise, do the following using the Amazon Command Line Interface:

  1. Run the following command to get a list of your core devices.

    aws greengrassv2 list-core-devices
  2. Run the following command to retrieve the status of the components on each core device from Step 1. Replace coreDeviceName with the name of each core device to query.

    aws greengrassv2 list-installed-components --core-device-thing-name coreDeviceName
  3. Gather the core devices with the discontinued component version installed from the previous steps.

  4. Run the following command to retrieve the status of all deployment jobs for each core device from Step 3. Replace coreDeviceName with the name of the core device to query.

    aws greengrassv2 list-effective-deployments --core-device-thing-name coreDeviceName

    The response contains the list of deployment jobs for the core device. You can revise the deployment to choose another component version. For more information about how to revise a deployment, see Revise deployments.

Greengrass Command Line Interface issues

Troubleshoot issues with the Greengrass CLI.

java.lang.RuntimeException: Unable to create ipc client

You might see this error when you run a Greengrass CLI command and you specify a different root folder than where the Amazon IoT Greengrass Core software is installed.

Do one of the following to set the root path, and replace /greengrass/v2 with the path to your Amazon IoT Greengrass Core software installation:

  • Set the GGC_ROOT_PATH environment variable to /greengrass/v2.

  • Add the --ggcRootPath /greengrass/v2 argument to your command as shown in the following example.

    greengrass-cli --ggcRootPath /greengrass/v2 <command> <subcommand> [arguments]

Amazon Command Line Interface issues

Troubleshoot Amazon CLI issues for Amazon IoT Greengrass V2.

Error: Invalid choice: 'greengrassv2'

You might see this error when you run an Amazon IoT Greengrass V2 command with the Amazon CLI (for example, aws greengrassv2 list-core-devices).

This error indicates that you have a version of the Amazon CLI that doesn't support Amazon IoT Greengrass V2. To use Amazon IoT Greengrass V2 with the Amazon CLI, you must have one of the following versions or later:

  • Minimum Amazon CLI V1 version: v1.18.197

  • Minimum Amazon CLI V2 version: v2.1.11

Tip

You can run the following command to check the version of the Amazon CLI that you have.

aws --version

To resolve this issue, update the Amazon CLI to a later version that supports Amazon IoT Greengrass V2. For more information, see Installing, updating, and uninstalling the Amazon CLI in the Amazon Command Line Interface User Guide.