A decorator function to assist in performance testing of CUDA operations. This function will: 1. Automatically determine whether any parameters in the argument list, or the output of the `func`, are of type `torch.Tensor`. 2. If so, calculate the memory usage of the input an
(func: Callable, shape: List[int], tflops: float, steps: int, *args, **kwargs)
| 73 | |
| 74 | |
| 75 | def benchmark(func: Callable, shape: List[int], tflops: float, steps: int, *args, **kwargs): |
| 76 | """ |
| 77 | A decorator function to assist in performance testing of CUDA operations. |
| 78 | |
| 79 | This function will: |
| 80 | 1. Automatically determine whether any parameters in the argument list, |
| 81 | or the output of the `func`, are of type `torch.Tensor`. |
| 82 | 2. If so, calculate the memory usage of the input and output tensors |
| 83 | on the GPU (based on their data type and `torch.numel()`). |
| 84 | 3. Establish a CUDA graph and attempt to execute `func` repeatedly for `steps` iterations. |
| 85 | 4. Record the execution time during these iterations. |
| 86 | 5. Use the information above to compute the compute performance (TFLOPS) and memory throughput. |
| 87 | |
| 88 | Args: |
| 89 | func (function): The function to benchmark. |
| 90 | shape (list of int): The problem shape. |
| 91 | tflops (float): The computational workload (in TFLOPS) per call of `func`. |
| 92 | steps (int): The number of times the function is executed during benchmarking. |
| 93 | *args: Positional arguments to be passed to the `func`. |
| 94 | **kwargs: Keyword arguments to be passed to the `func`. |
| 95 | |
| 96 | Returns: |
| 97 | function result |
| 98 | """ |
| 99 | |
| 100 | # Ensure CUDA is available |
| 101 | if not torch.cuda.is_available(): |
| 102 | raise RuntimeError("CUDA is required for benchmarking.") |
| 103 | |
| 104 | # Check for torch.Tensor in inputs and outputs |
| 105 | input_tensors = [arg for arg in args if isinstance(arg, torch.Tensor)] |
| 106 | input_tensors += [value for value in kwargs.values() if isinstance(value, torch.Tensor)] |
| 107 | |
| 108 | def calculate_memory(tensor: torch.Tensor): |
| 109 | """Calculate memory usage in bytes for a tensor.""" |
| 110 | return tensor.numel() * tensor.element_size() |
| 111 | |
| 112 | input_memory = sum(calculate_memory(t) for t in input_tensors) |
| 113 | |
| 114 | # Execute the function to inspect outputs |
| 115 | with torch.no_grad(): |
| 116 | output = func(*args, **kwargs) |
| 117 | |
| 118 | output_memory = 0 |
| 119 | if isinstance(output, torch.Tensor): |
| 120 | output_memory = calculate_memory(output) |
| 121 | elif isinstance(output, (list, tuple)): |
| 122 | output_memory = sum(calculate_memory(o) for o in output if isinstance(o, torch.Tensor)) |
| 123 | |
| 124 | total_memory = input_memory + output_memory |
| 125 | |
| 126 | # Warm-up and CUDA graph creation |
| 127 | for _ in range(10): # Warm-up |
| 128 | func(*args, **kwargs) |
| 129 | |
| 130 | torch.cuda.synchronize() # Ensure no pending operations |
| 131 | |
| 132 | # Benchmark the function |