Intergrated energy use profiling

Florin Manaila ( )


This example shows how to gather detailed energy use information for a code. The example employs several commands that will interfere with other workloads and so should only be used on whole nodes under exclusive access. The example uses a Tensorflow Convolutional Neural Network benchmark that assumes that a conda virtual environment based on WMLCE powerai is active.

Commands to run this example

The example can be run interactively or in batch mode. Here we shown interactive execution on one node and using four GPUs.

Running on a single node

The following commands can be used

  1. request a whole node from the scheduler:

    bsub -n 4 -R "span[ptile=4]" -gpu "num=4" -Is bash
  2. once the node is available, set up the environment under your account. eg:

    conda activate wmlce-1.6.2
  3. create a sub directory for the run e.g.:

    mkdir powertest; cd powertest
  4. load the power profiling tools:

    module load perftools
    git clone
    cp bff35521a2fa0c499578c98751be1b3c/ .
    cp bff35521a2fa0c499578c98751be1b3c/main_script_commands .
    cp bff35521a2fa0c499578c98751be1b3c/ .
    cp bff35521a2fa0c499578c98751be1b3c/ .
    cp bff35521a2fa0c499578c98751be1b3c/ .
    rm -fr bff35521a2fa0c499578c98751be1b3c
    chmod +x *.sh
  5. Once the environment is all set, beginin running the power monitoring script

    mpirun --tag-output ./
  6. If things work as expected this should create a file that logs power use you should be able to see the power readings using the command cat like this:

    cat energy-consumption.out.<###>

    where <###> is your job number. You should see:

    1580143466:     Instantaneous power reading:                   497 Watts
    1580143470:     Instantaneous power reading:                   555 Watts
    1580143473:     Instantaneous power reading:                   557 Watts
    1580143476:     Instantaneous power reading:                   559 Watts
    1580143480:     Instantaneous power reading:                   469 Watts
  7. now we can start an application. First lets use the example from as follows:

        git clone
        cd pytorchstyletransfer_satori
        conda install nbconvert
        jupyter nbconvert --to script TorchTransfer.ipynb
        cd ..
    **NOTE** the ``conda install nbconvert`` above is only needed once.
  8. While the python step is executing you can open another terminal using and look at the output. To do that cd to the dirctory you started the job in (e.g. powertest/pytorchstyletransfer_satori) and type:

    tail -f energy-consumption.out.<###>
    where <###> is your job number.
  9. we can now try and do the same with a four gpu application:

        git clone
        cd benchmarks
        git checkout 1ef603fd7e568ff75127ec07f160808fcc59911c
        cd ..
        conda install gxx_linux-ppc64le=7.3.0 cffi cudatoolkit-dev
        HOROVOD_CUDA_HOME=$CONDA_PREFIX HOROVOD_GPU_ALLREDUCE=DDL pip install horovod --no-cache-dir
        ddlrun -v ./ python /nobackup/users/florin/hpms/tf_cnn_benchmarks/ --model resnet50 --batch_size 128 --variable_update=horovod --num_batches=1000 --use_fp16
    in this case the application is a Tensor Flow benchmark, but any application can be used. **NOTE** as before we
    only need to add the ``conda install`` steps once. There is also another wrinkle in this example. The Horovod modules
    need to be built for a specific configuraiton. We pass the environment variables to ``pip install`` to
    select the configuraiton. We also need to use the pip ( ) installation tool for Horovod, not conda. There
    is no clear reason why Horovod could not be in Conda, but sometimes computing is like that!
    Again, while the application is running we can check out the power log and we can make a plot from
    the log too. Unfortunately this example doesn't quite work yet!
  10. once the application is finished then finish power logging using the command:

    mpirun --tag-output ./
  • The power history (recorded as snapshots every 3 seconds) will be written to a file with name begining energy-consumption.out. followed by the LSF job id. This file can be listed to the screen e.g.:

    cat energy-consumption.out.7843_0
  • Example all in one batch script. To see an example all in one script of the above steps for measuring power and capturing energy use use the following commands:

    git clone
    cp bff35521a2fa0c499578c98751be1b3c/batch-job-example.lsf .
  • Congratulations if yuu got this far. You have earned a cookie and are now in a good state to try measuring and optimizing resource use in your favorite project! GOOD LUCK.

Code and input data repositories for this example

see text above

Useful references