# Remediating detected GuardDuty security findings Remediating findings Amazon GuardDuty generates [findings](guardduty_findings.md) that indicate potential security findings associated with GuardDuty foundational threat detection and dedicated protection plans. The following sections describe the recommended remediation steps for these scenarios. If there are alternative remediation scenarios, they will be described in the descriptions for each finding type. You can access the full information about a finding type by selecting it from the [Active findings types](guardduty_finding-types-active.md) table. **Topics** + [ # Remediating a potentially compromised Amazon EC2 instance ](compromised-ec2.md) + [ # Remediating a potentially compromised S3 bucket ](compromised-s3.md) + [ # Remediating a potentially malicious S3 object ](compromised-s3object-malware-protection-gdu.md) + [ # Remediating a potentially compromised EBS Snapshot ](compromised-snapshot.md) + [ # Remediating a potentially compromised EC2 AMI ](compromised-ami.md) + [ # Remediating a potentially compromised EC2 Recovery Point ](compromised-ec2-recoverypoint.md) + [ # Remediating a potentially compromised S3 Recovery Point ](compromised-s3-recoverypoint.md) + [ # Remediating a potentially compromised ECS cluster ](compromised-ecs.md) + [ # Remediating potentially compromised Amazon credentials ](compromised-creds.md) + [ # Remediating a potentially compromised standalone container ](remediate-compromised-standalone-container.md) + [ # Remediating EKS Protection findings ](guardduty-remediate-kubernetes.md) + [ # Remediating Runtime Monitoring findings ](guardduty-remediate-runtime-monitoring.md) + [ # Remediating a potentially compromised database ](guardduty-remediate-compromised-database-rds.md) + [ # Remediating a potentially compromised Lambda function ](remediate-lambda-protection-finding-types.md) # Remediating a potentially compromised Amazon EC2 instance When GuardDuty generates [finding types that indicate potentially compromised Amazon EC2 resources](https://docs.amazonaws.cn/guardduty/latest/ug/guardduty_finding-types-active.html#findings-table), then your **Resource** will be **Instance**. Potential finding types could be [EC2 finding types](guardduty_finding-types-ec2.md), [GuardDuty Runtime Monitoring finding types](findings-runtime-monitoring.md), or [Malware Protection for EC2 finding types](findings-malware-protection.md). If the behavior that caused the finding was expected in your environment, then consider using [Suppression rules](findings_suppression-rule.md). Perform the following steps to remediate the potentially compromised Amazon EC2 instance: 1. **Identify the potentially compromised Amazon EC2 instance** Investigate the potentially compromised instance for malware and remove any discovered malware. You may use [On-demand malware scan in GuardDuty](on-demand-malware-scan.md) to identify malware in the potentially compromised EC2 instance, or check [Amazon Web Services Marketplace](http://www.amazonaws.cn/marketplace) to see if there are helpful partner products to identify and remove malware. 1. **Isolate the potentially compromised Amazon EC2 instance** If possible, use the following steps to isolate the potentially compromised instance: 1. Create a dedicated **Isolation** security group. An isolation security group should only have inbound and outbound access from specific IP addresses. Make sure that there is no inbound or outbound rule that allows traffic for `0.0.0.0/0 (0-65535)`. 1. Associate the **Isolation** security group with this instance. 1. Remove all security group associations other than the newly created **Isolation** security group from the potentially compromised instance. **Note** The existing tracked connections won't be terminated as a result of changing security groups - only future traffic will be effectively blocked by the new security group. For information about blocking further traffic from suspicious existing connections, see [Enforce NACLs based on network IoCs to prevent further traffic](https://github.com/aws-samples/aws-customer-playbook-framework/blob/main/docs/Ransom_Response_EC2_Linux.md#enforce-nacls-based-on-network-iocs-to-prevent-further-traffic) in the *Incident Response Playbook*. 1. **Identify the source of the suspicious activity** If malware is detected, then based on the finding type in your account, identify and stop the potentially unauthorized activity on your EC2 instance. This may require actions such as closing any open ports, changing access policies, and upgrading applications to correct vulnerabilities. If you are unable to identify and stop unauthorized activity on your potentially compromised EC2 instance, we recommend that you terminate the compromised EC2 instance and replace it with a new instance as needed. The following are additional resources for securing your EC2 instances: + Security and Networking sections in [Best practices for Amazon EC2](https://docs.amazonaws.cn/AWSEC2/latest/UserGuide/ec2-best-practices.html) + [Amazon EC2 security groups for Linux instances](https://docs.amazonaws.cn/AWSEC2/latest/UserGuide/using-network-security.html). + [Security in Amazon EC2](https://docs.amazonaws.cn/AWSEC2/latest/UserGuide/ec2-security.html) + [Tips for securing your EC2 instances (Linux)](https://www.amazonaws.cn/articles/tips-for-securing-your-ec2-instance/). + [Amazon security best practices](http://www.amazonaws.cn//architecture/security-identity-compliance/) + [Amazon Security Incident Response Technical Guide](https://docs.amazonaws.cn/security-ir/latest/userguide/security-incident-response-guide.html). 1. **Browse Amazon Web Services re:Post** Browse [Amazon Web Services re:Post](https://repost.aws/) for further assistance. 1. **Submit a technical support request** If you are a premium support package subscriber, you can submit a [technical support](https://console.amazonaws.cn/support/home#/case/create?issueType=technical) request. # Remediating a potentially compromised S3 bucket When GuardDuty generates [GuardDuty S3 Protection finding types](guardduty_finding-types-s3.md), it indicates that your Amazon S3 buckets have been compromised. If the behavior that caused the finding was expected in your environment, then consider creating [Suppression rules](findings_suppression-rule.md). If this behavior was not expected, then follow these recommended steps to remediate a potentially compromised Amazon S3 bucket in your Amazon environment: 1. **Identify the potentially compromised S3 resource.** A GuardDuty finding for S3 will list the associated S3 bucket, its Amazon Resource Name (ARN), and its owner in the finding details. 1. **Identify the source of the suspicious activity and the API call used.** The API call used will be listed as `API` in the finding details. The source will be an IAM principal (either an IAM role, user, or account) and identifying details will be listed in the finding. Depending on the source type, Remote IP address or source domain info will be available and can help you evaluate whether the source was authorized. If the finding involved credentials from an Amazon EC2 instance the details for that resource will also be included. 1. **Determine whether the call source was authorized to access the identified resource. ** For example consider the following: + If an IAM user was involved, is it possible that their credentials have been potentially compromised? For more information, see [Remediating potentially compromised Amazon credentials](compromised-creds.md). + If an API was invoked from a principal that has no prior history of invoking this type of API, does this source need access permissions for this operation? Can the bucket permissions be further restricted? + If the access was seen from the **user name** `ANONYMOUS_PRINCIPAL` with **user type** of `AWSAccount` this indicates the bucket is public and was accessed. Should this bucket be public? If not, review the security recommendations below for alternative solutions to sharing S3 resources. + If the access was though a successful `PreflightRequest` call seen from the **user name **`ANONYMOUS_PRINCIPAL` with **user type** of `AWSAccount` this indicates the bucket has a cross-origin resource sharing (CORS) policy set. Should this bucket have a CORS policy? If not, ensure the bucket is not inadvertently public and review the security recommendations below for alternative solutions to sharing S3 resources. For more information on CORS see [Using cross-origin resource sharing (CORS)](https://docs.amazonaws.cn/AmazonS3/latest/userguide/cors.html) in the S3 user guide. 1. **Determine whether the S3 bucket contains sensitive data.** Use [Amazon Macie](https://docs.amazonaws.cn/macie/latest/user/what-is-macie.html) to determine whether the S3 bucket contains sensitive data, such as personally identifiable information (PII), financial data, or credentials. If automated sensitive data discovery is enabled for your Macie account, review the S3 bucket's details to gain a better understanding of your S3 bucket's contents. If this feature is disabled for your Macie account, we recommend turning it on to expedite your assessment. Alternatively, you can create and run a sensitive data discovery job to inspect the S3 bucket's objects for sensitive data. For more information, see [Discovering sensitive data with Macie](https://docs.amazonaws.cn/macie/latest/user/data-classification.html). If the access was authorized, you can ignore the finding. The [https://console.amazonaws.cn/guardduty/](https://console.amazonaws.cn/guardduty/) console allows you to set up rules to entirely suppress individual findings so that they no longer appear. For more information, see [Suppression rules in GuardDuty](findings_suppression-rule.md). If you determine that your S3 data has been exposed or accessed by an unauthorized party, review the following S3 security recommendations to tighten permissions and restrict access. Appropriate remediation solutions will depend on the needs of your specific environment. ## Recommendations based on specific S3 bucket access needs **The following list provides recommendations based on specific Amazon S3 bucket access needs:** + For a centralized way to limit public access to your S3 data use, S3 block public access. Block public access settings can be enabled for access points, buckets, and Amazon Accounts through four different settings to control granularity of access. For more information see [Block public access settings](https://docs.amazonaws.cn/AmazonS3/latest/userguide/access-control-block-public-access.html#access-control-block-public-access-options) in the *Amazon S3 User Guide*. + Amazon Access policies can be used to control how IAM users can access your resources or how your buckets can be accessed. For more information see [Using bucket policies and user policies](https://docs.amazonaws.cn/AmazonS3/latest/userguide/security_iam_service-with-iam.html) in the *Amazon S3 User Guide*. Additionally you can use Virtual Private Cloud (VPC) endpoints with S3 bucket policies to restrict access to specific VPC endpoints. For more information see [Controlling access from VPC endpoints with bucket policies](https://docs.amazonaws.cn/AmazonS3/latest/userguide/example-bucket-policies-vpc-endpoint.html) in the *Amazon S3 User Guide* + To temporarily allow access to your S3 objects to trusted entities outside your account you can create a Presigned URL through S3. This access is created using your account credentials and depending on the credentials used can last 6 hours to 7 days. For more information see [Using presigned URLs to download and upload objects](https://docs.amazonaws.cn/AmazonS3/latest/userguide/using-presigned-url.html) in the *Amazon S3 User Guide*. + For use cases that require that sharing of S3 objects between different sources you can use S3 Access Points to create permission sets that restrict access to only those within your private network. For more information see [Managing access to shared datasets with access points](https://docs.amazonaws.cn/AmazonS3/latest/userguide/access-points.html) in the *Amazon S3 User Guide*. + To securely grant access to your S3 resources to other Amazon Accounts you can use an access control list (ACL), for more information see [Access control list (ACL) overview](https://docs.amazonaws.cn/AmazonS3/latest/userguide/acl-overview.html) in the *Amazon S3 User Guide*. For more information about S3 security options, see [Security best practices for Amazon S3](https://docs.amazonaws.cn/AmazonS3/latest/userguide/security-best-practices.html) in the *Amazon S3 User Guide*. # Remediating a potentially malicious S3 object When GuardDuty generates [Malware Protection for S3 finding type](gdu-malware-protection-s3-finding-types.md), it indicates that a newly uploaded object in your Amazon S3 bucket contains malware. The resource type is an **S3Object**. Use the following recommended steps to potentially remediate the generated finding: 1. Identify the potentially malicious S3 object by checking the **S3ObjectDetails** associated with the finding. 1. Isolate the impacted S3 object. If you had enabled tagging at the time of enabling Malware Protection for S3 for the associated Amazon S3 bucket, GuardDuty must have assigned a **Malicious** tag to this object. Use tag-based access control (TBAC) to restrict access to this S3 object. For more information, see [Using tag-based access control (TBAC)](tag-based-access-s3-malware-protection.md). Alternatively, if you do not need this object any longer, you can also choose to delete it or move it to an isolated S3 bucket. For information about considerations for deleting an S3 object, see [Deleting objects](https://docs.amazonaws.cn/AmazonS3/latest/userguide/DeletingObjects.html) in the *Amazon S3 User Guide*. # Remediating a potentially compromised EBS Snapshot When GuardDuty generates an Execution:EC2/MaliciousFile\$1Snapshot finding type, it indicates that malware has been detected in an Amazon EBS snapshot. Perform the following steps to remediate the potentially compromised snapshot: 1. **Identify the potentially compromised snapshot** 1. Identify the potentially compromised snapshot. A GuardDuty finding for an EBS snapshot will list the affected snapshot ID, its Amazon Resource Name (ARN), and associated malware scan details in the finding details. 1. Review recovery point details using the following command: ``` aws backup describe-recovery-point —backup-vault-name 021345abcdef6789 —recovery-point-arn "arn:aws:ec2:us-east-1::snapshot/snap-abcdef01234567890" ``` 1. **Restrict access the compromised snapshot** Review and modify backup vault access policies to restrict recovery point access and suspend any automated restore jobs that might use this snapshot. 1. Review current sharing permissions: ``` aws ec2 describe-snapshot-attribute --snapshot-id snap-abcdef01234567890 --attribute createVolumePermission ``` 1. Remove specific account access: ``` aws ec2 modify-snapshot-attribute --snapshot-id snap-abcdef01234567890 --attribute createVolumePermission --operation-type remove --user-ids 111122223333 ``` 1. For additional CLI options, see [modify-snapshot-attribute CLI documentation](https://docs.aws.amazon.com/cli/latest/reference/ec2/modify-snapshot-attribute.html). 1. **Take remediation action** + Before proceeding with deletion, ensure you have identified all dependencies and have proper backups if needed. # Remediating a potentially compromised EC2 AMI When GuardDuty generates an Execution:EC2/MaliciousFile\$1AMI finding type, it indicates that malware has been detected in an Amazon Machine Image (AMI). Perform the following steps to remediate the potentially compromised AMI: 1. **Identify the potentially compromised AMI** 1. A GuardDuty finding for AMIs will list the affected AMI ID, its Amazon Resource Name (ARN), and associated malware scan details in the finding details. 1. Review AMI source image: ``` aws ec2 describe-images --image-ids ami-021345abcdef6789 ``` 1. **Restrict access to the compromised resources** 1. Review and modify backup vault access policies to restrict recovery point access and suspend any automated restore jobs that might use this recovery point. 1. Remove Permissions from source AMI permissions First view existing permissions: ``` aws ec2 describe-image-attribute --image-id ami-abcdef01234567890 --attribute launchPermission ``` Then remove individual permissions: ``` aws ec2 modify-image-attribute --image-id ami-abcdef01234567890 --launch-permission '{"Remove":[{"UserId":"111122223333"}]}' ``` For additional CLI options, see [Share an AMI with specific accounts - Amazon Elastic Compute Cloud](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/sharingamis-explicit.html#unsharing-an-ami) 1. If source is an EC2 Instance see: [Remediating a potentially compromised Amazon EC2 instance](https://docs.aws.amazon.com/guardduty/latest/ug/compromised-ec2.html). 1. **Take remediation action** + Before proceeding with deletion, ensure you have identified all dependencies and have proper backups if needed. # Remediating a potentially compromised EC2 Recovery Point When GuardDuty generates an Execution:EC2/MaliciousFile\$1RecoveryPoint finding type, it indicates that malware has been detected in an EC2 Recovery Point Backup resource. Perform the following steps to remediate the potentially compromised recovery point: 1. **Identify the potentially compromised EC2 Recovery Point** 1. A GuardDuty finding for EC2 Recovery Point will list its Amazon Resource Name (ARN), and associated malware scan details in the finding details: ``` aws backup describe-recovery-point --backup-vault-name 021345abcdef6789 --recovery-point-arn "arn:aws:backup:us-east-1:111122223333:recovery-point:a1b2c3d4-5678-90ab-cdef-EXAMPLE11111" ``` 1. Review recovery details to look for source image: ``` aws backup get-recovery-point-restore-metadata --backup-vault-name 021345abcdef6789 --recovery-point-arn "arn:aws:backup:us-east-1:111122223333:recovery-point:a1b2c3d4-5678-90ab-cdef-EXAMPLE11111" ``` 1. **Restrict access to the compromised resources** + Review and modify backup vault access policies to restrict recovery point access and suspend any automated restore jobs that might use this recovery point. If your environment uses resource tagging, tag the recovery point appropriately to indicate it's under investigation and consider pausing scheduled backups if necessary. Example: *aws backup tag-resource -—resource-arn arn:aws:backup:us-east-1:111122223333:recovery-point:a1b2c3d4-5678-90ab-cdef-EXAMPLE11111 -—tags Investigation=Malware,DoNotDelete=True* 1. **Take remediation action** + Before proceeding with deletion, ensure you have identified all dependencies and have proper backups if needed. # Remediating a potentially compromised S3 Recovery Point When GuardDuty generates an Execution:S3/MaliciousFile\$1RecoveryPoint finding type, it indicates that malware has been detected in an S3 Recovery Point Backup resource. Perform the following steps to remediate the potentially compromised recovery point: 1. **Identify the potentially compromised S3 Recovery Point** 1. A GuardDuty finding for S3 recovery points will list the affected recovery point ARN, backup vault name, and associated malware scan details in the finding details. 1. Review recovery point details: ``` aws backup describe-recovery-point --backup-vault-name 021345abcdef6789 --recovery-point-arn arn:aws:backup:us-east-1:123456789012:recovery-point:abcdef01234567890 ``` 1. **Restrict access to the compromised resources** + Review and modify backup vault access policies to restrict recovery point access and suspend any automated restore jobs that might use this recovery point. If your environment uses resource tagging, tag the recovery point appropriately to indicate it's under investigation and consider pausing scheduled backups if necessary. Example: ``` aws backup tag-resource —resource-arn arn:aws:backup:us-east-1:111122223333:recovery-point:abcdef01234567890 —tags Investigation=Malware,DoNotDelete=True ``` For additional details see: [tag-resource CLI Command Reference](https://docs.aws.amazon.com/cli/latest/reference/backup/tag-resource.html) 1. **Take remediation action** + Before proceeding with deletion, ensure you have identified all dependencies and have proper backups if needed. # Remediating a potentially compromised ECS cluster A potentially compromised ECS cluster finding indicates suspicious or malicious activity has been detected within your Amazon ECS environment. This could include unauthorized access, malware execution, or other malicious behavior that puts your container workloads at risk. Follow these steps to remediate a potentially compromised Amazon ECS cluster: 1. **Identify the potentially compromised ECS cluster and the detected threat (findings)** Impacted ECS cluster details are listed in the GuardDuty finding details panel. 1. **Evaluate the source of threat/malware** Check for malware in container images. If malware is detected, review the container image being used. Use [https://docs.amazonaws.cn//AmazonECS/latest/APIReference/API_ListTasks.html](https://docs.amazonaws.cn//AmazonECS/latest/APIReference/API_ListTasks.html) to identify all other running tasks that use the same potentially compromised image. 1. **Isolate impacted tasks** Stop the threat by blocking all network traffic (both incoming and outgoing) to the affected tasks. This network isolation helps prevent any ongoing attacks by cutting off all connections to the compromised task. **Note**: If you determine this finding was triggered by expected/legitimate activity in your environment, you can set up a suppression rule to prevent similar findings from appearing. For additional information, see [Suppression rules in GuardDuty](findings_suppression-rule.md). # Remediating potentially compromised Amazon credentials When GuardDuty generates [IAM finding types](guardduty_finding-types-iam.md), it indicates that your Amazon credentials have been compromised. The potentially compromised **Resource** type is **AccessKey**. To remediate potentially compromised credentials in your Amazon environment, perform the following steps: 1. **Identify the potentially compromised IAM entity and the API call used.** The API call used will be listed as `API` in the finding details. The IAM entity (either an IAM role or user) and its identifying information will be listed in the **Resource** section of the finding details. The type of IAM entity involved can be determined by the **User Type** field, the name of the IAM entity will be in the **User name** field. The type of IAM entity involved in the finding can also be determined by the **Access key ID** used. For keys beginning with `AKIA`: This type of key is a long term customer-managed credential associated with an IAM user or Amazon Web Services account root user. For information about managing access keys for IAM users, see [Managing access keys for IAM users](https://docs.amazonaws.cn/IAM/latest/UserGuide/id_credentials_access-keys.html). For keys beginning with `ASIA`: This type of key is a short term temporary credential generated by Amazon Security Token Service. These keys exists for only a short time and cannot be viewed or managed in the Amazon Management Console. IAM roles will always use Amazon STS credentials, but they can also be generated for IAM Users, for more information on Amazon STS see [IAM: Temporary security credentials](https://docs.amazonaws.cn/IAM/latest/UserGuide/id_credentials_temp.html#sts-introduction). If a role was used the **User name** field will indicate the name of the role used. You can determine how the key was requested with Amazon CloudTrail by examining the `sessionIssuer` element of the CloudTrail log entry, for more information see [IAM and Amazon STS information in CloudTrail](https://docs.amazonaws.cn/IAM/latest/UserGuide/cloudtrail-integration.html#iam-info-in-cloudtrail). 1. **Review permissions for the IAM entity.** Open the IAM console. Depending on the type of the entity used, choose the **Users** or **Roles** tab, and locate the affected entity by typing the identified name into the search field. Use the **Permission** and **Access Advisor** tabs to review effective permissions for that entity. 1. **Determine whether the IAM entity credentials were used legitimately.** Contact the user of the credentials to determine if the activity was intentional. For example, find out if the user did the following: + Invoked the API operation that was listed in the GuardDuty finding + Invoked the API operation at the time that is listed in the GuardDuty finding + Invoked the API operation from the IP address that is listed in the GuardDuty finding If this activity is a legitimate use of the Amazon credentials, you can ignore the GuardDuty finding. The [https://console.amazonaws.cn/guardduty/](https://console.amazonaws.cn/guardduty/) console allows you to set up rules to entirely suppress individual findings so that they no longer appear. For more information, see [Suppression rules in GuardDuty](findings_suppression-rule.md). If you can't confirm if this activity is a legitimate use, it could be the result of a compromise to the particular access key - the IAM user's sign-in credentials, or possibly the entire Amazon Web Services account. If you suspect your credentials have been compromised, review the information in [My Amazon Web Services account may be compromised](https://repost.aws/knowledge-center/potential-account-compromise) to remediate this issue. # Remediating a potentially compromised standalone container When GuardDuty generates [finding types that indicate potentially compromised container](https://docs.amazonaws.cn/guardduty/latest/ug/guardduty_finding-types-active.html#findings-table), your **Resource type** will be **Container**. If the behavior that caused the finding was expected in your environment, then consider using [Suppression rules](findings_suppression-rule.md). To remediate potentially compromised credentials in your Amazon environment, perform the following steps: 1. **Isolate the potentially compromised container** The following steps will help you identify the potentially malicious container workload: + Open the GuardDuty console at [https://console.amazonaws.cn/guardduty/](https://console.amazonaws.cn/guardduty/). + On the **Findings** page, choose the corresponding finding to view the findings panel. + In the findings panel, under the **Resource affected** section, you can view the container's **ID** and **Name**. Isolate this container from other container workloads. 1. **Pause the container** Suspend all the processes in your container. For information about freezing your container, see [Pause a container](https://docs.docker.com/engine/api/v1.35/#tag/Container/operation/ContainerPause). **Stop the container**. If the step above fails, and the container doesn't pause, stop the container from running. If you've enabled the [Snapshots retention](malware-protection-customizations.md#mp-snapshots-retention) feature, GuardDuty will retain the snapshots of your EBS volumes that contain malware. For information about stopping the container, see [Stop a container](https://docs.docker.com/engine/api/v1.35/#tag/Container). 1. **Evaluate the presence of malware** Evaluate if malware was in the container's image. If the access was authorized, you can ignore the finding. The [https://console.amazonaws.cn/guardduty/](https://console.amazonaws.cn/guardduty/) console allows you to set up rules to entirely suppress individual findings so that they no longer appear. The GuardDuty console allows you to set up rules to entirely suppress individual findings so that they no longer appear. For more information, see [Suppression rules in GuardDuty](findings_suppression-rule.md). # Remediating EKS Protection findings Amazon GuardDuty generates [findings](guardduty_findings.md) that indicate potential Kubernetes security issues when EKS Protection is enabled for your account. For more information, see [EKS Protection](kubernetes-protection.md). The following sections describe the recommended remediation steps for these scenarios. Specific remediation actions are described in the entry for that specific finding type. You can access the full information about a finding type by selecting it from the [Active findings types table](guardduty_finding-types-active.md). If any of the EKS Protection finding types were generated expectantly, you can consider adding [Suppression rules in GuardDuty](findings_suppression-rule.md) to prevent future alerts. Different types of attacks and configuration issues can trigger GuardDuty EKS Protection findings. This guide helps you identify the root causes of GuardDuty findings against your cluster and outlines appropriate remediation guidance. The following are the primary root causes that lead to GuardDuty Kubernetes findings: + [Potential configuration issues](#compromised-kubernetes-config) + [Remediating potentially compromised Kubernetes users](#compromised-kubernetes-user) + [Remediating potentially compromised Kubernetes pods](#compromised-kubernetes-pod) + [Remediating potentially compromised Kubernetes nodes](#compromised-kubernetes-node) + [Remediating potentially compromised container images](#compromised-kubernetes-image) **Note** Before Kubernetes version 1.14, the `system:unauthenticated` group was associated to `system:discovery` and `system:basic-user` **ClusterRoles** by default. This may allow unintended access from anonymous users. Cluster updates do not revoke these permissions, which means that even if you have updated your cluster to version 1.14 or later, these permissions may still be in place. We recommend that you disassociate these permissions from the `system:unauthenticated` group. For more information about removing these permissions, see [Secure Amazon EKS clusters with best practices](https://docs.amazonaws.cn/eks/latest/userguide/security-best-practices.html) in the **Amazon EKS User Guide**. ## Potential configuration issues If a finding indicates a configuration issue, see the remediation section of that finding for guidance on resolving that particular issue. For more information, see the following finding types that indicate configuration issues: + [Policy:Kubernetes/AnonymousAccessGranted](guardduty-finding-types-eks-audit-logs.md#policy-kubernetes-anonymousaccessgranted) + [Policy:Kubernetes/ExposedDashboard](guardduty-finding-types-eks-audit-logs.md#policy-kubernetes-exposeddashboard) + [Policy:Kubernetes/AdminAccessToDefaultServiceAccount](guardduty-finding-types-eks-audit-logs.md#policy-kubernetes-adminaccesstodefaultserviceaccount) + [Policy:Kubernetes/KubeflowDashboardExposed](guardduty-finding-types-eks-audit-logs.md#policy-kubernetes-kubeflowdashboardexposed) + Any finding that ends in **SuccessfulAnonymousAccess** ## Remediating potentially compromised Kubernetes users A GuardDuty finding can indicate a compromised Kubernetes user when a user identified in the finding has performed an unexpected API action. You can identify the user in the **Kubernetes user details** section of a finding details in the console, or in the `resource.kubernetesDetails.kubernetesUserDetails` of the findings JSON. These user details include `user name`, `uid`, and the Kubernetes groups that the user belongs to. If the user was accessing the workload using an IAM entity, you can use the `Access Key details` section to identify the details of an IAM role or user. See the following user types and their remediation guidance. **Note** You can use Amazon Detective to further investigate the IAM role or user identified in the finding. While viewing the finding details in GuardDuty console, choose **Investigate in Detective**. Then select Amazon user or role from the listed items to investigate it in Detective. **Built-in Kubernetes admin** – The default user assigned by Amazon EKS to the IAM identity that created the cluster. This user type is identified by the user name `kubernetes-admin`. **To revoke access of a built-in Kubernetes admin:** + Identify the `userType` from the `Access Key details` section. + If the `userType` is **Role** and the role belongs to an EC2 instance role: + Identify that instance then follow the instructions in [Remediating a potentially compromised Amazon EC2 instance](compromised-ec2.md). + If the `userType` is **User**, or is a **Role** that was assumed by a user: 1. [Rotate the access key](https://docs.amazonaws.cn//IAM/latest/UserGuide/id_credentials_access-keys.html#Using_RotateAccessKey) of that user. 1. Rotate any secrets that user had access to. 1. Review the information in [My Amazon Web Services account may be compromised](https://repost.aws/knowledge-center/potential-account-compromise) for further details. **OIDC authenticated user** – A user granted access through an OIDC provider. Typically an OIDC user has an email address as a user name. You can check if your cluster uses OIDC with the following command: `aws eks list-identity-provider-configs --cluster-name your-cluster-name ` **To revoke access of an OIDC authenticated user:** 1. Rotate the credentials of that user in the OIDC provider. 1. Rotate any secrets that user had access to. **Amazon-Auth ConfigMap defined user** – An IAM user that was granted access through an Amazon-auth ConfigMap. For more information, see [Managing users or IAM roles for your cluster](https://docs.amazonaws.cn/eks/latest/userguide/add-user-role.html) in the **Amazon EKS User Guide**. You can review their permissions using the following command: `kubectl edit configmaps aws-auth --namespace kube-system` **To revoke access of an Amazon ConfigMap user:** 1. Use the following command to open the ConfigMap. ``` kubectl edit configmaps aws-auth --namespace kube-system ``` 1. Identify the role or user entry under the **mapRoles** or **mapUsers** section with the same user name as the one reported in the Kubernetes user details section of your GuardDuty finding. See the following example, where the admin user has been identified in a finding. ``` apiVersion: v1 data: mapRoles: | - rolearn: arn:aws:iam::444455556666:role/eksctl-my-cluster-nodegroup-standard-wo-NodeInstanceRole-1WP3NUE3O6UCF user name: system:node:EC2_PrivateDNSName groups: - system:bootstrappers - system:nodes mapUsers: | - userarn: arn:aws:iam::123456789012:user/admin username: admin groups: - system:masters - userarn: arn:aws:iam::111122223333:user/ops-user username: ops-user groups: - system:masters ``` 1. Remove that user from the ConfigMap. See the following example where the admin user has been removed. ``` apiVersion: v1 data: mapRoles: | - rolearn: arn:aws:iam::111122223333:role/eksctl-my-cluster-nodegroup-standard-wo-NodeInstanceRole-1WP3NUE3O6UCF username: system:node:{{EC2PrivateDNSName}} groups: - system:bootstrappers - system:nodes mapUsers: | - userarn: arn:aws:iam::111122223333:user/ops-user username: ops-user groups: - system:masters ``` 1. If the `userType` is **User**, or is a **Role** that was assumed by a user: 1. [Rotate the access key](https://docs.amazonaws.cn//IAM/latest/UserGuide/id_credentials_access-keys.html#Using_RotateAccessKey) of that user. 1. Rotate any secrets that user had access to. 1. Review the information in [My Amazon account may be compromised](http://www.amazonaws.cn//premiumsupport/knowledge-center/potential-account-compromise/) for further details. If the finding does not have a `resource.accessKeyDetails` section, the user is a Kubernetes service account. **Service account** – The service account provides an identity for pods and can be identified by a user name with the following format: `system:serviceaccount:namespace:service_account_name`. **To revoke access to a service account:** 1. Rotate the service account credentials. 1. Review the guidance for pod compromise in the following section. ## Remediating potentially compromised Kubernetes pods When GuardDuty specifies details of a pod or workload resource inside the `resource.kubernetesDetails.kubernetesWorkloadDetails` section, that pod or workload resource has been potentially compromised. A GuardDuty finding can indicate a single pod has been compromised or that multiple pods have been compromised through a higher-level resource. See the following compromise scenarios for guidance on how to identify the pod or pods that have been compromised. **Single pods compromise** If the `type` field inside the `resource.kubernetesDetails.kubernetesWorkloadDetails` section is **pods**, the finding identifies a single pods. The name field is the `name` of the pods and `namespace` field is its namespace. For information about identifying the worker node running the pods, see [Identify the offending pods and worker node](https://docs.amazonaws.cn/eks/latest/best-practices/incident-response-and-forensics.html#_identify_the_offending_pod_and_worker_node) in the *Amazon EKS Best Practices Guide*. **Pods compromised through workload resource** If the `type` field inside the `resource.kubernetesDetails.kubernetesWorkloadDetails` section identifies a **Workload Resource**, such as a `Deployment`, it is likely that all of the pods within that workload resource have been compromised. For information about identifying all the pods of the workload resource and the nodes on which they are running, see [Identify the offending pods and worker nodes using workload name](https://docs.amazonaws.cn/eks/latest/best-practices/incident-response-and-forensics.html#_identify_the_offending_pods_and_worker_nodes_using_workload_name) in the *Amazon EKS Best Practices Guide*. **Pods compromised through service account** If a GuardDuty finding identifies a Service Account in the `resource.kubernetesDetails.kubernetesUserDetails` section, it is likely that pods using the identified service account are compromised. The user name reported by a finding is a service account if it has the following format: `system:serviceaccount:namespace:service_account_name`. For information about identifying all the pods using the service account and the nodes on which they are running, see [Identify the offending pods and worker nodes using service account name](https://docs.amazonaws.cn/eks/latest/best-practices/incident-response-and-forensics.html#_identify_the_offending_pods_and_worker_nodes_using_service_account_name) in the *Amazon EKS Best Practices Guide*. After you have identified all the compromised pods and the nodes on which they are running, see [Isolate the pod by creating a network policy that denies all ingress and egress traffic to the pod](https://docs.amazonaws.cn/eks/latest/best-practices/incident-response-and-forensics.html#_isolate_the_pod_by_creating_a_network_policy_that_denies_all_ingress_and_egress_traffic_to_the_pod) in the *Amazon EKS Best Practices Guide*. **To remediate a potentially compromised pod:** 1. Identify the vulnerability that compromised the pods. 1. Implement the fix for that vulnerability and start new replacement pods. 1. Delete the vulnerable pods. For more information, see [Redeploy compromised pod or workload resource](https://docs.amazonaws.cn/eks/latest/best-practices/incident-response-and-forensics.html#_redeploy_compromised_pod_or_workload_resource) in the *Amazon EKS Best Practices Guide*. If the worker node has been assigned an IAM role that allows Pods to gain access to other Amazon resources, remove those roles from the instance to prevent further damage from the attack. Similarly, if the Pod has been assigned an IAM role, evaluate whether you can safely remove the IAM policies from the role without impacting other workloads. ## Remediating potentially compromised container images When a GuardDuty finding indicates a pod compromise, the image used to launch the pod could be potentially malicious or compromised. GuardDuty findings identify the container image within the `resource.kubernetesDetails.kubernetesWorkloadDetails.containers.image` field. You can determine if the image is malicious by scanning it for malware. **To remediate a potentially compromised container image:** 1. Stop using the image immediately and remove it from your image repository. 1. Identify all pods using the potentially compromised image. For more information, see [Identify pods with vulnerable or compromised images and worker nodes](https://docs.amazonaws.cn/eks/latest/best-practices/incident-response-and-forensics.html#_identify_pods_with_vulnerable_or_compromised_images_and_worker_nodes) in the *Amazon EKS Best Practices Guide*. 1. Isolate the potentially compromised pods, rotate credentials, and gather data for analysis. For more information, see [Isolate the pod by creating a network policy that denies all ingress and egress traffic to the pod](https://docs.amazonaws.cn/eks/latest/best-practices/incident-response-and-forensics.html#_isolate_the_pod_by_creating_a_network_policy_that_denies_all_ingress_and_egress_traffic_to_the_pod) in the *Amazon EKS Best Practices Guide*. 1. Delete all pods using the potentially compromised image. ## Remediating potentially compromised Kubernetes nodes A GuardDuty finding can indicate a node compromise if the user identified in the finding represents a node identity or if the finding indicates the use of a privileged container. The user identity is a worker node if the **username** field has following format: `system:node:node name`. For example, `system:node:ip-192-168-3-201.ec2.internal`. This indicates that the adversary has gained access to the node and it is using the node’s credentials to talk to the Kubernetes API endpoint. A finding indicates the use of a privileged container if one or more of the containers listed in the finding has the `resource.kubernetesDetails.kubernetesWorkloadDetails.containers.securityContext.privileged` finding field set to `True`. **To remediate a potentially compromised node:** 1. Isolate the pod, rotate its credentials, and gather data for forensic analysis. For more information, see [Isolate the pod by creating a network policy that denies all ingress and egress traffic to the pod](https://docs.amazonaws.cn/eks/latest/best-practices/incident-response-and-forensics.html#_isolate_the_pod_by_creating_a_network_policy_that_denies_all_ingress_and_egress_traffic_to_the_pod) in the *Amazon EKS Best Practices Guide*. 1. Identify the service accounts used by all of the pods running on the potentially compromised node. Review their permissions and rotate the service accounts if needed. 1. Terminate the potentially compromised node. # Remediating Runtime Monitoring findings When you enable Runtime Monitoring for your account, Amazon GuardDuty may generate [GuardDuty Runtime Monitoring finding types](findings-runtime-monitoring.md) that indicate potential security issues in your Amazon environment. The potential security issues indicate either a compromised Amazon EC2 instance, container workload, an Amazon EKS cluster, or a set of compromised credentials in your Amazon environment. The security agent monitors runtime events from multiple resource types. To identify the potentially compromised resource, view **Resource type** in the generated finding details in the GuardDuty console. The following section describes the recommended remediation steps for each resource type. ------ #### [ Instance ] If the **Resource type** in the finding details is **Instance**, it indicates that either an EC2 instance or an EKS node is potentially compromised. + To remediate a compromised EKS node, see [Remediating potentially compromised Kubernetes nodes](guardduty-remediate-kubernetes.md#compromised-kubernetes-node). + To remediate a compromised EC2 instance, see [Remediating a potentially compromised Amazon EC2 instance](compromised-ec2.md). ------ #### [ EKSCluster ] If the **Resource type** in the finding details is **EKSCluster**, it indicates that either a pod or a container inside an EKS cluster is potentially compromised. + To remediate a compromised pod, see [Remediating potentially compromised Kubernetes pods](guardduty-remediate-kubernetes.md#compromised-kubernetes-pod). + To remediate a compromised container image, see [Remediating potentially compromised container images](guardduty-remediate-kubernetes.md#compromised-kubernetes-image). ------ #### [ ECSCluster ] If the **Resource type** in the finding details is **ECSCluster**, it indicates that either an ECS task or a container inside an ECS task is potentially compromised. 1. **Identify the affected ECS cluster** The GuardDuty Runtime Monitoring finding provides the ECS cluster details in the finding's details panel or in the `resource.ecsClusterDetails` section in the finding JSON. 1. **Identify the affected ECS task** The GuardDuty Runtime Monitoring finding provides the ECS task details in the finding's details panel or in the `resource.ecsClusterDetails.taskDetails` section in the finding JSON. 1. **Isolate the affected task** Isolate the impacted task by denying all ingress and egress traffic to the task. A deny all traffic rule may help stop an attack that is already underway, by severing all connections to the task. 1. **Remediate the compromised task** 1. Identify the vulnerability that compromised the task. 1. Implement the fix for that vulnerability and start new a replacement task. 1. Stop the vulnerable task. ------ #### [ Container ] If the **Resource type** in the finding details is **Container**, it indicates that a standalone container is potentially compromised. + To remediate, see [Remediating a potentially compromised standalone container](remediate-compromised-standalone-container.md). + If the finding is generated across multiple containers using the same container image, see [Remediating potentially compromised container images](guardduty-remediate-kubernetes.md#compromised-kubernetes-image). + If the container has accessed the underlying EC2 host, its associated instance credentials may have been compromised. For more information, see [Remediating potentially compromised Amazon credentials](compromised-creds.md). + If a potentially malicious actor has accessed the underlying EKS node or an EC2 instance, see the recommended remediation under the *EKSCluster* and *Instance* tabs. ------ ## Remediating compromised container images When a GuardDuty finding indicates a task compromise, the image used to launch the task could be malicious or compromised. GuardDuty findings identify the container image within the `resource.ecsClusterDetails.taskDetails.containers.image` field. You can determine whether or not the image is malicious by scanning it for malware. **To remediate a compromised container image** 1. Stop using the image immediately and remove it from your image repository. 1. Identify all of the tasks that are using this image. 1. Stop all of the tasks that are using the compromised image. Update their task definitions so that they stop using the compromised image. # Remediating a potentially compromised database GuardDuty generates [RDS Protection finding types](findings-rds-protection.md) that indicate potentially suspicious and anomalous login behavior in your [Supported databases](rds-protection.md#rds-pro-supported-db) after you enable [RDS Protection](rds-protection.md). Using RDS login activity, GuardDuty analyzes and profiles threats by identifying unusual patterns in login attempts. **Note** You can access the full information about a finding type by selecting it from the [GuardDuty active finding types](guardduty_finding-types-active.md#findings-table). Follow these recommended steps to remediate a potentially compromised Amazon Aurora database in your Amazon environment. **Topics** + [ ## Remediating potentially compromised database with successful login events ](#gd-compromised-db-successful-attempt) + [ ## Remediating potentially compromised database with failed login events ](#gd-compromised-db-failed-attempt) + [ ## Remediating potentially compromised credentials ](#gd-rds-database-compromised-credentials) + [ ## Restrict network access ](#gd-rds-database-restrict-network-access) ## Remediating potentially compromised database with successful login events The following recommended steps can help you remediate a potentially compromised Aurora database that exhibits unusual behavior related to successful login events. 1. **Identify the affected database and user.** The generated GuardDuty finding provides the name of the affected database and the corresponding user details. For more information, see [Finding details](guardduty_findings-summary.md). 1. **Confirm whether this behavior is expected or unexpected. ** The following list specifies potential scenarios that may have caused GuardDuty to generate a finding: + A user who logs in to their database after a long time has passed. + A user who logs in to their database on an occasional basis, for example, a financial analyst who logs in each quarter. + A potentially suspicious actor who is involved in a successful login attempt potentially compromises the database. 1. **Begin this step if the behavior is unexpected.** 1. **Restrict database access** Restrict database access for the suspected accounts and the source of this login activity. For more information, see [Remediating potentially compromised credentials](#gd-rds-database-compromised-credentials) and [Restrict network access](#gd-rds-database-restrict-network-access). 1. **Assess the impact and determine what information was accessed.** + If available, review the audit logs to identify the pieces of information that might have been accessed. For more information, see [Monitoring events, logs, and streams in an Amazon Aurora DB cluster](https://docs.amazonaws.cn/AmazonRDS/latest/AuroraUserGuide/CHAP_Monitor_Logs_Events.html) in the *Amazon Aurora User Guide*. + Determine if any sensitive or protected information was accessed or modified. ## Remediating potentially compromised database with failed login events The following recommended steps can help you remediate a potentially compromised Aurora database that exhibits unusual behavior related to failed login events. 1. **Identify the affected database and user.** The generated GuardDuty finding provides the name of the affected database and the corresponding user details. For more information, see [Finding details](guardduty_findings-summary.md). 1. **Identify the source of the failed login attempts.** The generated GuardDuty finding provides the **IP address** and **ASN organization** (if it was a public connection) under the **Actor** section of the finding panel. An Autonomous System (AS) is a group of one or more IP prefixes (lists of IP addresses accessible on a network) run by one or more network operators that maintain a single, clearly-defined routing policy. Network operators need Autonomous System Numbers (ASNs) to control routing within their networks and to exchange routing information with other internet service providers (ISPs). 1. **Confirm that this behavior is unexpected.** Examine if this activity represents an attempt to gain additional unauthorized access to the database as follows: + If the source is internal, examine if an application is misconfigured and attempting a connection repeatedly. + If this is an external actor, examine whether the corresponding database is public facing or is misconfigured and thus allowing potential malicious actors to brute force common user names. 1. **Begin this step if the behavior is unexpected.** 1. **Restrict database access** Restrict database access for the suspected accounts and the source of this login activity. For more information, see [Remediating potentially compromised credentials](#gd-rds-database-compromised-credentials) and [Restrict network access](#gd-rds-database-restrict-network-access). 1. **Perform root-cause analysis and determine the steps that potentially led to this activity.** Set up an alert to get notified when an activity modifies a networking policy and creates an insecure state. For more information, see [ Firewall policies in Amazon Network Firewall](https://docs.amazonaws.cn/network-firewall/latest/developerguide/firewall-policies.html) in the *Amazon Network Firewall Developer Guide*. ## Remediating potentially compromised credentials A GuardDuty finding may indicate that the user credentials for an affected database have been compromised when the user identified in the finding has performed an unexpected database operation. You can identify the user in the **RDS DB user details** section within the finding panel in the console, or within the `resource.rdsDbUserDetails` of the findings JSON. These user details include user name, application used, database accessed, SSL version, and authentication method. + To revoke access or rotate passwords for specific users that are involved in the finding, see [Security with Amazon Aurora MySQL](https://docs.amazonaws.cn/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Security.html), or [Security with Amazon Aurora PostgreSQL](https://docs.amazonaws.cn/AmazonRDS/latest/AuroraUserGuide/AuroraPostgreSQL.Security.html) in the *Amazon Aurora User Guide*. + Use Amazon Secrets Manager to securely store and automatically rotate the secrets for Amazon Relational Database Service(RDS) databases. For more information, see [Amazon Secrets Manager tutorials](https://docs.amazonaws.cn/secretsmanager/latest/userguide/tutorials.html) in the *Amazon Secrets Manager User Guide*. + Use IAM database authentication to manage database users' access without the need for passwords. For more information, see [IAM database authentication](https://docs.amazonaws.cn/AmazonRDS/latest/AuroraUserGuide/UsingWithRDS.IAMDBAuth.html) in the *Amazon Aurora User Guide*. For more information, see [Security best practices for Amazon Relational Database Service](https://docs.amazonaws.cn/AmazonRDS/latest/UserGuide/CHAP_BestPractices.Security.html) in the *Amazon RDS User Guide*. ## Restrict network access A GuardDuty finding may indicate that a database is accessible beyond your applications, or Virtual Private Cloud (VPC). If the remote IP address in the finding is an unexpected connection source, audit the security groups. A list of security groups attached to the database is available under **Security groups** in the [https://console.amazonaws.cn/rds/](https://console.amazonaws.cn/rds/) console, or in the `resource.rdsDbInstanceDetails.dbSecurityGroups` of the findings JSON. For more information on configuring security groups, see [Controlling access with security groups](https://docs.amazonaws.cn/AmazonRDS/latest/UserGuide/Overview.RDSSecurityGroups.html) in the *Amazon RDS User Guide*. If you're using a firewall, restrict network access to the database by reconfiguring the Network Access Control Lists (NACLs). For more information, see [Firewalls in Amazon Network Firewall](https://docs.amazonaws.cn/network-firewall/latest/developerguide/firewalls.html) in the *Amazon Network Firewall Developer Guide*. # Remediating a potentially compromised Lambda function When GuardDuty generates [Lambda Protection finding types](lambda-protection-finding-types.md), your Lambda function may be compromised. If the activity that caused GuardDuty to generate this finding was expected, you can consider using [Suppression rules](findings_suppression-rule.md). We recommend completing the following steps to remediate a compromised Lambda function: **To remediate Lambda Protection findings** 1. **Identify the potentially compromised Lambda function version**. A GuardDuty finding for Lambda Protection provides the name, Amazon Resource Name (ARN), function version, and revision ID associated with the Lambda function listed in the finding details. 1. **Identify the source of the potentially suspicious activity**. 1. Review the code associated with the Lambda function version involved in the finding. 1. Review the imported libraries and layers of the Lambda function version involved in the finding. 1. If you have enabled [Scanning Amazon Lambda functions with Amazon Inspector](https://docs.amazonaws.cn/inspector/latest/user/scanning-lambda.html), review the [Amazon Inspector findings](https://docs.amazonaws.cn/inspector/latest/user/findings-understanding-locating-analyzing.html) associated with the Lambda function involved in the finding. 1. Review the Amazon CloudTrail logs to identify the principal that caused the function update and ensure that the activity was authorized or expected. 1. **Remediate the potentially compromised Lambda function**. 1. Disable the execution triggers of the Lambda function involved in the finding. For more information, see [DeleteFunctionEventInvokeConfig](https://docs.amazonaws.cn/lambda/latest/dg/API_DeleteFunctionEventInvokeConfig.html). 1. Review the Lambda code and update the libraries imports and [Lambda function layers](https://docs.amazonaws.cn/lambda/latest/dg/chapter-layers.html) to remove the potentially suspicious libraries and layers. 1. Mitigate Amazon Inspector findings related to the Lambda function involved in the finding.