Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions,
see Getting Started with Amazon Web Services in China
(PDF).
View and restart Amazon EMR and application
processes (daemons)
When you troubleshoot a cluster, you may want to list running processes. You may also want to stop or restart processesS. For example, you can restart a
process after you change a configuration or notice a problem with a particular process after
you analyze log files and error messages.
There are two types of processes that run on a cluster: Amazon EMR processes (for example,
instance-controller and Log Pusher), and processes associated with the applications
installed on the cluster (for example, hadoop-hdfs-namenode, and
hadoop-yarn-resourcemanager).
To work with processes directly on a cluster, you must first connect to the master node.
For more information, see Connect to an Amazon EMR cluster.
Viewing running processes
The method you use to view running processes on a cluster differs according to the
Amazon EMR version you use.
- EMR 5.30.0 and 6.0.0 and later
-
Example : List all running processes
The following example uses systemctl
and specifies
--type
to view all processes.
systemctl --type=service
Example : List specific processes
The following example lists all processes with names that contain
hadoop
.
systemctl --type=service | grep -i hadoop
Example output:
hadoop-hdfs-namenode.service loaded active running Hadoop namenode
hadoop-httpfs.service loaded active running Hadoop httpfs
hadoop-kms.service loaded active running Hadoop kms
hadoop-mapreduce-historyserver.service loaded active running Hadoop historyserver
hadoop-state-pusher.service loaded active running Daemon process that processes and serves EMR metrics data.
hadoop-yarn-proxyserver.service loaded active running Hadoop proxyserver
hadoop-yarn-resourcemanager.service loaded active running Hadoop resourcemanager
hadoop-yarn-timelineserver.service loaded active running Hadoop timelineserver
Example : See a detailed status report for a specific process
The following example displays a detailed status report for the
hadoop-hdfs-namenode
service.
sudo systemctl status hadoop-hdfs-namenode
Example output:
hadoop-hdfs-namenode.service - Hadoop namenode
Loaded: loaded (/etc/systemd/system/hadoop-hdfs-namenode.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2021-08-18 21:01:46 UTC; 26min ago
Main PID: 9733 (java)
Tasks: 0
Memory: 1.1M
CGroup: /system.slice/hadoop-hdfs-namenode.service
‣ 9733 /etc/alternatives/jre/bin/java -Dproc_namenode -Xmx1843m -server -XX:OnOutOfMemoryError=kill -9 %p ...
Aug 18 21:01:37 ip-172-31-20-123 systemd[1]: Starting Hadoop namenode...
Aug 18 21:01:37 ip-172-31-20-123 su[9715]: (to hdfs) root on none
Aug 18 21:01:37 ip-172-31-20-123 hadoop-hdfs-namenode[9683]: starting namenode, logging to /var/log/hadoop-hdfs/ha...out
Aug 18 21:01:46 ip-172-31-20-123 hadoop-hdfs-namenode[9683]: Started Hadoop namenode:[ OK ]
Aug 18 21:01:46 ip-172-31-20-123 systemd[1]: Started Hadoop namenode.
Hint: Some lines were ellipsized, use -l to show in full.
- EMR 4.x - 5.29.0
-
Example : List all running processes
The following example lists all running processes.
initctl list
- EMR 2.x - 3.x
-
Example : List all running processes
The following example lists all running processes.
ls /etc/init.d/
Stopping and restarting processes
After you determine which processes are running, you can stop and then restart them if
necessary.
- EMR 5.30.0 and 6.0.0 and later
-
Example : Stop a process
The following example stops the hadoop-hdfs-namenode
process.
sudo systemctl stop hadoop-hdfs-namenode
You can query the status
to verify that the process is
stopped.
sudo systemctl status hadoop-hdfs-namenode
Example output:
hadoop-hdfs-namenode.service - Hadoop namenode
Loaded: loaded (/etc/systemd/system/hadoop-hdfs-namenode.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Wed 2021-08-18 21:37:50 UTC; 8s ago
Main PID: 9733 (code=exited, status=143)
Example : Start a process
The following example starts the hadoop-hdfs-namenode
process.
sudo systemctl start hadoop-hdfs-namenode
You can query the status to verify that the process is running.
sudo systemctl status hadoop-hdfs-namenode
Example output:
hadoop-hdfs-namenode.service - Hadoop namenode
Loaded: loaded (/etc/systemd/system/hadoop-hdfs-namenode.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2021-08-18 21:38:24 UTC; 2s ago
Process: 13748 ExecStart=/etc/init.d/hadoop-hdfs-namenode start (code=exited, status=0/SUCCESS)
Main PID: 13800 (java)
Tasks: 0
Memory: 1.1M
CGroup: /system.slice/hadoop-hdfs-namenode.service
‣ 13800 /etc/alternatives/jre/bin/java -Dproc_namenode -Xmx1843m -server -XX:OnOutOfMemoryError=kill -9 %p...
- EMR 4.x - 5.29.0
-
Example : Stop a running process
The following example stops the hadoop-hdfs-namenode
service.
sudo stop hadoop-hdfs-namenode
Example : Restart a stopped process
The following example restarts the hadoop-hdfs-namenode
service. You must use the start
command and not
restart
.
sudo start hadoop-hdfs-namenode
Example : Check the process status
The following fetches the status for
hadoop-hdfs-namenode
. You can use the status
command to verify that the process has stopped or started.
sudo status hadoop-hdfs-namenode
- EMR 2.x - 3.x
-
Example : Stop an application process
The following example stops the hadoop-hdfs-namenode
service, which is associated with the version of Amazon EMR installed on the
cluster.
sudo /etc/init.d/hadoop-hdfs-namenode
stop
Example : Restart an application process
The following example command restarts the
hadoop-hdfs-namenode
process:
sudo /etc/init.d/hadoop-hdfs-namenode
start
Example : Stop an Amazon EMR process
The following example stops a process, such as instance-controller,
which is not associated with the version of Amazon EMR on the cluster.
sudo /sbin/stop instance-controller
Example : Restart an Amazon EMR process
The following example restarts a process, such as instance-controller,
which is not associated with the version of Amazon EMR on the cluster.
sudo /sbin/start instance-controller
The /sbin/start, stop
and restart
commands
are symlinks to /sbin/intictl
. For more information about
initctl
, see the initctl man page by typing man
initctl
at the command prompt.