Troubleshooting DB issues for Amazon RDS Custom for Oracle
The shared responsibility model of RDS Custom provides OS shell–level access and database administrator access. RDS Custom runs resources in your account, unlike Amazon RDS, which runs resources in a system account. With greater access comes greater responsibility. In the following sections, you can learn how to troubleshoot issues with Amazon RDS Custom DB instances.
Note
This section explains how to troubleshoot RDS Custom for Oracle. For troubleshooting RDS Custom for SQL Server, see Troubleshooting DB issues for Amazon RDS Custom for SQL Server.
Topics
Viewing RDS Custom events
The procedure for viewing events is the same for RDS Custom and Amazon RDS DB instances. For more information, see Viewing Amazon RDS events.
To view RDS Custom event notification using the Amazon CLI, use the describe-events
command. RDS Custom introduces several new
events. The event categories are the same as for Amazon RDS. For the list of events, see Amazon RDS event categories and event messages.
The following example retrieves details for the events that have occurred for the specified RDS Custom DB instance.
aws rds describe-events \ --source-identifier my-custom-instance \ --source-type db-instance
Viewing RDS Custom events
The procedure for subscribing to events is the same for RDS Custom and Amazon RDS DB instances. For more information, see Subscribing to Amazon RDS event notification.
To subscribe to RDS Custom event notification using the CLI, use the create-event-subscription
command. Include the
following required parameters:
-
--subscription-name
-
--sns-topic-arn
The following example creates a subscription for backup and recovery events for an RDS Custom DB instance in the current Amazon
account. Notifications are sent to an Amazon Simple Notification Service (Amazon SNS) topic, specified by --sns-topic-arn
.
aws rds create-event-subscription \ --subscription-name my-instance-events \ --source-type db-instance \ --event-categories '["backup","recovery"]' \ --sns-topic-arn arn:aws:sns:us-east-1:123456789012:interesting-events
Troubleshooting custom engine version creation for RDS Custom for Oracle
When CEV creation fails, RDS Custom issues RDS-EVENT-0198
with the message Creation failed for custom engine
version
, and includes details about the failure. For
example, the event prints missing files.major-engine-version.cev_name
CEV creation might fail because of the following issues:
-
The Amazon S3 bucket containing your installation files isn't in the same Amazon Region as your CEV.
-
When you request CEV creation in an Amazon Web Services Region for the first time, RDS Custom creates an S3 bucket for storing RDS Custom resources (such as CEV artifacts, Amazon CloudTrail logs, and transaction logs).
CEV creation fails if RDS Custom can't create the S3 bucket. Either the caller doesn't have S3 permissions as described in Step 4: Grant required permissions to your IAM user or role, or the number of S3 buckets has reached the limit.
-
The caller doesn't have permissions to get files from your S3 bucket that contains the installation media files. These permissions are described in Step 7: Add necessary IAM permissions.
-
Your IAM policy has an
aws:SourceIp
condition. Make sure to follow the recommendations in Amazon Denies access to Amazon based on the source IP in the Amazon Identity and Access Management User Guide. Also make sure that the caller has the S3 permissions described in Step 4: Grant required permissions to your IAM user or role. -
Installation media files listed in the CEV manifest aren't in your S3 bucket.
-
The SHA-256 checksums of the installation files are unknown to RDS Custom.
Confirm that the SHA-256 checksums of the provided files match the SHA-256 checksum on the Oracle website. If the checksums match, contact Amazon Support
and provide the failed CEV name, file name, and checksum. -
The OPatch version is incompatible with your patch files. You might get the following message:
OPatch is lower than minimum required version. Check that the version meets the requirements for all patches, and try again
. To apply an Oracle patch, you must use a compatible version of the OPatch utility. You can find the required version of the Opatch utility in the readme file for the patch. Download the most recent OPatch utility from My Oracle Support, and try creating your CEV again. -
The patches specified in the CEV manifest are in the wrong order.
You can view RDS events either on the RDS console (in the navigation pane, choose Events) or by using the
describe-events
Amazon CLI command. The default duration is 60 minutes. If no events are returned, specify a longer
duration, as shown in the following example.
aws rds describe-events --duration 360
Currently, the MediaImport service that imports files from Amazon S3 to create CEVs isn't integrated with Amazon CloudTrail. Therefore,
if you turn on data logging for Amazon RDS in CloudTrail, calls to the MediaImport service such as the
CreateCustomDbEngineVersion
event aren't logged.
However, you might see calls from the API gateway that accesses your Amazon S3 bucket. These calls come from the MediaImport
service for the CreateCustomDbEngineVersion
event.
Fixing unsupported configurations in RDS Custom for Oracle
In the shared responsibility model, it's your responsibility to fix configuration
issues that put your RDS Custom for Oracle DB instance into the unsupported-configuration
state.
If the issue is with the Amazon infrastructure, you can use the console or the Amazon CLI to fix
it. If the issue is with the operating system or the database configuration, you can log in
to the host to fix it.
Note
This section explains how to fix unsupported configurations in RDS Custom for Oracle. For information about RDS Custom for SQL Server, see Fixing unsupported configurations in RDS Custom for SQL Server.
In the following table, you can find descriptions of the notifications and events that the support perimeter sends and how to fix them. These notifications and the support perimeter are subject to change. For background on the support perimeter, see RDS Custom support perimeter. For event descriptions, see Amazon RDS event categories and event messages.
Configuration | RDS event message | Description | Action |
---|---|---|---|
Database | |||
Database health |
|
The support perimeter monitors the DB instance state. It also monitors how many restarts occurred during the previous hour and day. You're notified when the instance is in a state where it still exists, but you can't interact with it. |
Log in to your host and examine the database state.
Restart your RDS Custom for Oracle DB instance if necessary to get it running again. Sometimes it's necessary to reboot the host. After the restart, the RDS Custom agent detects that your DB instance is no longer in an unresponsive state. It then notifies the support perimeter to reevaluate your DB instance state. |
Oracle Data Guard role |
|
The support perimeter monitors the current database role every 15 seconds and sends a CloudWatch notification if the database role has changed. The Oracle Data Guard |
Restore your Oracle Data Guard database role to a supported value. RDS Custom only supports
If your RDS Custom for Oracle DB instance is standalone, you can use either of the
following statements to change it back to the
If your DB instance is a replica, you can use the following statement to
change it back to the
After the support perimeter determines that the database role is supported, your RDS Custom for Oracle DB instance becomes available within 15 seconds. |
Database archive lag target |
|
The support perimeter monitors the |
Log in to your host, connect to your RDS Custom for Oracle DB instance, and change the
For example, use the following SQL command.
Your DB instance becomes available within 30 minutes. |
Database log mode |
|
The DB instance log mode must be set to ARCHIVELOG . |
Log in to your host and shut down your RDS Custom for Oracle DB instance. Use the following SQL statement to initiate a consistent shutdown.
The RDS Custom agent restarts your DB instance and sets the log mode to
Your DB instance becomes available within 30 minutes. |
Operating system | |||
RDS Custom agent status |
|
The RDS Custom agent must always be running. The agent publishes the
The support perimeter also monitors the RDS Custom agent process state on the host every 30 minutes. On RDS Custom for Oracle, the DB instance goes outside the support perimeter if the RDS Custom agent stops. |
Log in to your host and make sure that the RDS Custom agent is running. You can use the following command to find the status of the agent.
You can use the following command to start the agent.
When the RDS Custom agent is running again, the |
Amazon Systems Manager agent (SSM agent) status |
|
The SSM agent must always be running. The RDS Custom agent is responsible for making sure that the Systems Manager agent is running. If the SSM agent was down and restarted, the RDS Custom agent publishes a metric to CloudWatch. The RDS Custom agent has an alarm on the metric set to trigger when there has been a restart in each of the previous three minutes. The support perimeter also monitors the SSM agent process state on the host every 30 minutes. |
For more information, see Troubleshooting SSM Agent. |
|
|
The support perimeter monitors that certain OS users are allowed to
run certain commands on the box. It monitors When the
|
If the overwrite is unsuccessful, you can log in to your host and
investigate why recent changes to the You can use the following command.
After the support perimeter determines that the |
Amazon resources | |||
Amazon EC2 instance state |
|
The support perimeter monitors EC2 instance state-change notifications. The EC2 instance must always be running. |
If your EC2 instance is stopped, start it and remount the binary and data volumes. If your EC2 instance is terminated, delete your RDS Custom for Oracle DB instance. |
Amazon EC2 instance attributes |
|
The support perimeter monitors the instance type of the EC2 instance where the RDS Custom DB instance is running. The EC2 instance type must stay the same as when you set it up during RDS Custom DB instance creation. |
Change the EC2 instance type back to the original type using the EC2 console or CLI. To change the instance type because of scaling requirements, begin point-in-time recovery and specify the new instance type and class. This action results in a new RDS Custom DB instance with a new host and Domain Name System (DNS) name. |
Amazon Elastic Block Store (Amazon EBS) volumes |
|
RDS Custom creates two types of EBS volume, besides the root volume created from the Amazon Machine Image (AMI), and associates them with the EC2 instance. The binary volume is where the database software binaries are located. The data volumes are where database files are located. The storage configurations that you set when creating the DB instance are used to configure the data volumes. The support perimeter monitors the following:
|
If you detached any initial EBS volumes, contact Amazon Support. If you modified the storage type, Provisioned IOPS, or storage throughput of an EBS volume, revert the modification to the original value. If you modified the storage size of an EBS volume, contact Amazon Support. (RDS Custom for Oracle only) If you attached any additional EBS volumes, do either of the following:
|
EBS-optimized state |
|
Amazon EC2 instances should be EBS optimized. If the |
To turn on the
|
Troubleshooting upgrades for RDS Custom for Oracle
Your upgrade of an RDS Custom for Oracle instance might fail. Following, you can find techniques that you can use during upgrades of RDS Custom DB for Oracle DB instances:
-
Examine the upgrade output log files in the
/tmp
directory on your DB instance. The names of the logs depend on your DB engine version. For example, you might see logs that contain the stringscatupgrd
orcatup
. -
Examine the
alert.log
file located in the/rdsdbdata/log/trace
directory. -
Run the following
grep
command in theroot
directory to track the upgrade OS process. This command shows where the log files are being written and determine the state of the upgrade process.ps -aux | grep upg
The following shows sample output.
root 18884 0.0 0.0 235428 8172 ? S< 17:03 0:00 /usr/bin/sudo -u rdsdb /rdsdbbin/scripts/oracle-control ORCL op_apply_upgrade_sh RDS-UPGRADE/2.upgrade.sh rdsdb 18886 0.0 0.0 153968 12164 ? S< 17:03 0:00 /usr/bin/perl -T -w /rdsdbbin/scripts/oracle-control ORCL op_apply_upgrade_sh RDS-UPGRADE/2.upgrade.sh rdsdb 18887 0.0 0.0 113196 3032 ? S< 17:03 0:00 /bin/sh /rdsdbbin/oracle/rdbms/admin/RDS-UPGRADE/2.upgrade.sh rdsdb 18900 0.0 0.0 113196 1812 ? S< 17:03 0:00 /bin/sh /rdsdbbin/oracle/rdbms/admin/RDS-UPGRADE/2.upgrade.sh rdsdb 18901 0.1 0.0 167652 20620 ? S< 17:03 0:07 /rdsdbbin/oracle/perl/bin/perl catctl.pl -n 4 -d /rdsdbbin/oracle/rdbms/admin -l /tmp catupgrd.sql root 29944 0.0 0.0 112724 2316 pts/0 S+ 18:43 0:00 grep --color=auto upg
-
Run the following SQL query to verify the current state of the components to find the database version and the options installed on the DB instance.
SET LINESIZE 180 COLUMN COMP_ID FORMAT A15 COLUMN COMP_NAME FORMAT A40 TRUNC COLUMN STATUS FORMAT A15 TRUNC SELECT COMP_ID, COMP_NAME, VERSION, STATUS FROM DBA_REGISTRY ORDER BY 1;
The output resembles the following.
COMP_NAME STATUS PROCEDURE ---------------------------------------- -------------------- -------------------------------------------------- Oracle Database Catalog Views VALID DBMS_REGISTRY_SYS.VALIDATE_CATALOG Oracle Database Packages and Types VALID DBMS_REGISTRY_SYS.VALIDATE_CATPROC Oracle Text VALID VALIDATE_CONTEXT Oracle XML Database VALID DBMS_REGXDB.VALIDATEXDB 4 rows selected.
-
Run the following SQL query to check for invalid objects that might interfere with the upgrade process.
SET PAGES 1000 LINES 2000 COL OBJECT FOR A40 SELECT SUBSTR(OWNER,1,12) OWNER, SUBSTR(OBJECT_NAME,1,30) OBJECT, SUBSTR(OBJECT_TYPE,1,30) TYPE, STATUS, CREATED FROM DBA_OBJECTS WHERE STATUS <>'VALID' AND OWNER IN ('SYS','SYSTEM','RDSADMIN','XDB');
Troubleshooting replica promotion for RDS Custom for Oracle
You can promote managed Oracle replicas in RDS Custom for Oracle using the console, promote-read-replica
Amazon CLI command, or
PromoteReadReplica
API. If you delete your primary DB instance, and all replicas are healthy, RDS Custom for Oracle promotes your managed
replicas to standalone instances automatically. If a replica has paused automation or is outside the support perimeter, you must fix the
replica before RDS Custom can promote it automatically. For more information, see Replica promotion limitations for
RDS Custom for Oracle.
The replica promotion workflow might become stuck in the following situation:
-
The primary DB instance is in the state
STORAGE_FULL
. -
The primary DB can't archive all of its online redo logs.
-
A gap exists between the archived redo log files on your Oracle replica and the primary database.
To respond to the stuck workflow, complete the following steps:
-
Synchronize the redo log gap on your Oracle replica DB instance.
-
Force the promotion of your read replica to the latest applied redo log. Run the following commands in SQL*Plus:
ALTER DATABASE ACTIVATE STANDBY DATABASE; SHUTDOWN IMMEDIATE STARTUP
-
Contact Amazon Web Services Support and request it to move your DB instance to available status.