GPU训练

为您的集群创建 pod 文件。pod 文件将提供有关集群应运行什么的说明。此 pod 文件将下载 PyTorch存储库并运行MNIST示例。打开vi或vim，然后复制并粘贴以下内容。将此文件另存为 pytorch.yaml。

apiVersion: v1
kind: Pod
metadata:
  name: pytorch-training
spec:
  restartPolicy: OnFailure
  containers:
  - name: pytorch-training
    image: 763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-training:1.5.1-gpu-py36-cu101-ubuntu16.04
    command:
      - "/bin/sh"
      - "-c"
    args:
    - "git clone https://github.com/pytorch/examples.git && python examples/mnist/main.py --no-cuda"
    env:
    - name: OMP_NUM_THREADS
      value: "36"
    - name: KMP_AFFINITY
      value: "granularity=fine,verbose,compact,1,0"
    - name: KMP_BLOCKTIME
      value: "1"

使用将 pod 文件分配给集群kubectl。

$ kubectl create -f pytorch.yaml

您应看到以下输出：

pod/pytorch-training created

检查状态。作业“pytorch-training”的名称位于 pytorch.yaml 文件中。它现在将显示在状态中。如果您正在运行任何其他测试或以前运行过某些测试，则它会出现在此列表中。多次运行此命令，直到看到状态更改为 “Running”。

$ kubectl get pods

您应看到以下输出：

NAME READY STATUS RESTARTS AGE
pytorch-training 0/1 Running 8 19m

检查日志以查看训练输出。

$ kubectl logs pytorch-training

您应该可以看到类似于如下输出的内容：

Cloning into 'examples'...
Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to ../data/MNIST/raw/train-images-idx3-ubyte.gz
9920512it [00:00, 40133996.38it/s]                           
Extracting ../data/MNIST/raw/train-images-idx3-ubyte.gz to ../data/MNIST/raw
Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz to ../data/MNIST/raw/train-labels-idx1-ubyte.gz
Extracting ../data/MNIST/raw/train-labels-idx1-ubyte.gz to ../data/MNIST/raw
32768it [00:00, 831315.84it/s]
Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz to ../data/MNIST/raw/t10k-images-idx3-ubyte.gz
1654784it [00:00, 13019129.43it/s]                           
Extracting ../data/MNIST/raw/t10k-images-idx3-ubyte.gz to ../data/MNIST/raw
Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz to ../data/MNIST/raw/t10k-labels-idx1-ubyte.gz
8192it [00:00, 337197.38it/s]
Extracting ../data/MNIST/raw/t10k-labels-idx1-ubyte.gz to ../data/MNIST/raw
Processing...
Done!
Train Epoch: 1 [0/60000 (0%)]    Loss: 2.300039
Train Epoch: 1 [640/60000 (1%)]    Loss: 2.213470
Train Epoch: 1 [1280/60000 (2%)]    Loss: 2.170460
Train Epoch: 1 [1920/60000 (3%)]    Loss: 2.076699
Train Epoch: 1 [2560/60000 (4%)]    Loss: 1.868078
Train Epoch: 1 [3200/60000 (5%)]    Loss: 1.414199
Train Epoch: 1 [3840/60000 (6%)]    Loss: 1.000870

查看日志以查看训练进度。您也可以继续选中 “get pods” 以刷新状态。当状态变为 Completed “” 时，训练工作就完成了。

为您的集群创建 pod 文件。pod 文件将提供有关集群应运行什么的说明。此 pod 文件将下载 Keras 并运行 Keras 示例。此示例使用 TensorFlow 框架。打开vi或vim并复制并粘贴以下内容。将此文件另存为 tf.yaml。你可以将其与 TensorFlow或 TensorFlow 2 一起使用。要将其与 TensorFlow 2 一起使用，请将 Docker 镜像更改为 TensorFlow 2 镜像。

apiVersion: v1
kind: Pod
metadata:
name: tensorflow-training
spec:
restartPolicy: OnFailure
containers:
- name: tensorflow-training
image: 763104351884.dkr.ecr.us-east-1.amazonaws.com/tensorflow-training:1.15.2-gpu-py37-cu100-ubuntu18.04
command: ["/bin/sh","-c"]
args: ["git clone https://github.com/fchollet/keras.git && python /keras/examples/mnist_cnn.py"]
resources:
  limits:
    nvidia.com/gpu: 1

使用将 pod 文件分配给集群kubectl。

$ kubectl create -f tf.yaml

您应看到以下输出：

pod/tensorflow-training created

检查状态。任务“tensorflow-training”的名称位于 tf.yaml 文件中。它现在将显示在状态中。如果您正在运行任何其他测试或以前运行过某些测试，则它会出现在此列表中。多次运行此命令，直到看到状态更改为 “Running”。

$ kubectl get pods

您应看到以下输出：

NAME READY STATUS RESTARTS AGE
tensorflow-training 0/1 Running 8 19m

检查日志以查看训练输出。

$ kubectl logs tensorflow-training

您应该可以看到类似于如下输出的内容：

Cloning into 'keras'...
Using TensorFlow backend.
Downloading data from https://s3.amazonaws.com/img-datasets/mnist.npz

8192/11490434 [..............................] - ETA: 0s
6479872/11490434 [===============>..............] - ETA: 0s
8740864/11490434 [=====================>........] - ETA: 0s
11493376/11490434 [==============================] - 0s 0us/step
x_train shape: (60000, 28, 28, 1)
60000 train samples
10000 test samples
Train on 60000 samples, validate on 10000 samples
Epoch 1/12
2019-03-19 01:52:33.863598: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX512F
2019-03-19 01:52:33.867616: I tensorflow/core/common_runtime/process_util.cc:69] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.

128/60000 [..............................] - ETA: 10:43 - loss: 2.3076 - acc: 0.0625
256/60000 [..............................] - ETA: 5:59 - loss: 2.2528 - acc: 0.1445
384/60000 [..............................] - ETA: 4:24 - loss: 2.2183 - acc: 0.1875
512/60000 [..............................] - ETA: 3:35 - loss: 2.1652 - acc: 0.1953
640/60000 [..............................] - ETA: 3:05 - loss: 2.1078 - acc: 0.2422
...

查看日志以查看训练进度。您也可以继续选中 “get pods” 以刷新状态。当状态变为 “Completed” 时，训练作业就完成了。