[Tensorflow] 以 Tensorlfow 內建的 Timeline 函數來作為 profiling 工具

最近對手邊沒有一個比較好的 Tensorflow profiling工具感到發愁的時候

在網路上搜尋資訊的時候剛好看到下面這篇 stackoverflow 的文章

Can I measure the execution time of individual operations with TensorFlow?

裡面就提到了 Tensorflow 裡面內建的 Timeline 這個函數



另外，Github 上面的也有類似的疑問

Profiling tools for open source TensorFlow #1824

其中，prb12 也提到 Timeline 這個函數，並提供了簡單的文字教學 :

I'm unlikely to have much time to write a tutorial in the near future, but the current status of the open source tools are as follows: 

There is now a basic CUPTI GPU tracer integrated in the runtime. You can run a step with tracing enabled and it records both the ops which are executed and the GPU kernels which are launched. Here is an example:

run_metadata = tf.RunMetadata()
_, l, lr, predictions = sess.run(
            [optimizer, loss, learning_rate, train_prediction],
            feed_dict=feed_dict,
            options=tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE),
            run_metadata=run_metadata)

After the step completes, the run_metadata should contain a StepStats protobuf with lots of timing information, grouped by tensorflow device. The CUPTI GPU tracing appears as some additional devices with names like /gpu:0/stream:56 and /gpu:0/memcpy 

Note: to get GPU tracing you will need to ensure that libcupti.so is on you LD_LIBRARY_PATH. It is usually found in /usr/local/cuda/extras/lib64.

The simplest way to use this information is to load the stats into a 'Timeline' as follows:

from tensorflow.python.client import timeline
trace = timeline.Timeline(step_stats=run_metadata.step_stats)

The Timeline class can then be used to emit a JSON trace file in the Chrome Tracing Format, as follows:

trace_file = open('timeline.ctf.json', 'w')
trace_file.write(trace.generate_chrome_trace_format())

To view this trace, navigate to the URL 'chrome://tracing' in a Chrome web browser, click the 'Load' button and locate the timeline file.

It would be fairly simple to write a small python web server which served up these traces from a running TensorFlow program like this

打開 Chrome，並在網址尋列輸入：Chrome://tracing 就有簡單的 gui 能夠使用

按下Load，選取剛創造的json檔案，就能夠將前一個時刻創造的 Timeline 圖表給呈現出來