If video_meta is not empty, perform temporal selective decoding to sample a clip from the video with TorchVision decoder. If video_meta is empty, decode the entire video and update the video_meta. Args: video_handle (bytes): raw bytes of the video file. sampling_rate
(
video_handle,
sampling_rate,
num_frames,
clip_idx,
video_meta,
num_clips=10,
target_fps=30,
modalities=("visual",),
max_spatial_scale=0,
use_offset=False,
)
| 134 | |
| 135 | |
| 136 | def torchvision_decode( |
| 137 | video_handle, |
| 138 | sampling_rate, |
| 139 | num_frames, |
| 140 | clip_idx, |
| 141 | video_meta, |
| 142 | num_clips=10, |
| 143 | target_fps=30, |
| 144 | modalities=("visual",), |
| 145 | max_spatial_scale=0, |
| 146 | use_offset=False, |
| 147 | ): |
| 148 | """ |
| 149 | If video_meta is not empty, perform temporal selective decoding to sample a |
| 150 | clip from the video with TorchVision decoder. If video_meta is empty, decode |
| 151 | the entire video and update the video_meta. |
| 152 | Args: |
| 153 | video_handle (bytes): raw bytes of the video file. |
| 154 | sampling_rate (int): frame sampling rate (interval between two sampled |
| 155 | frames). |
| 156 | num_frames (int): number of frames to sample. |
| 157 | clip_idx (int): if clip_idx is -1, perform random temporal |
| 158 | sampling. If clip_idx is larger than -1, uniformly split the |
| 159 | video to num_clips clips, and select the clip_idx-th video clip. |
| 160 | video_meta (dict): a dict contains VideoMetaData. Details can be found |
| 161 | at `pytorch/vision/torchvision/io/_video_opt.py`. |
| 162 | num_clips (int): overall number of clips to uniformly sample from the |
| 163 | given video. |
| 164 | target_fps (int): the input video may has different fps, convert it to |
| 165 | the target video fps. |
| 166 | modalities (tuple): tuple of modalities to decode. Currently only |
| 167 | support `visual`, planning to support `acoustic` soon. |
| 168 | max_spatial_scale (int): the maximal resolution of the spatial shorter |
| 169 | edge size during decoding. |
| 170 | Returns: |
| 171 | frames (tensor): decoded frames from the video. |
| 172 | fps (float): the number of frames per second of the video. |
| 173 | decode_all_video (bool): if True, the entire video was decoded. |
| 174 | """ |
| 175 | # Convert the bytes to a tensor. |
| 176 | video_tensor = torch.from_numpy(np.frombuffer(video_handle, dtype=np.uint8)) |
| 177 | |
| 178 | decode_all_video = True |
| 179 | video_start_pts, video_end_pts = 0, -1 |
| 180 | # The video_meta is empty, fetch the meta data from the raw video. |
| 181 | if len(video_meta) == 0: |
| 182 | # Tracking the meta info for selective decoding in the future. |
| 183 | meta = io._probe_video_from_memory(video_tensor) |
| 184 | # Using the information from video_meta to perform selective decoding. |
| 185 | video_meta["video_timebase"] = meta.video_timebase |
| 186 | video_meta["video_numerator"] = meta.video_timebase.numerator |
| 187 | video_meta["video_denominator"] = meta.video_timebase.denominator |
| 188 | video_meta["has_video"] = meta.has_video |
| 189 | video_meta["video_duration"] = meta.video_duration |
| 190 | video_meta["video_fps"] = meta.video_fps |
| 191 | video_meta["audio_timebas"] = meta.audio_timebase |
| 192 | video_meta["audio_numerator"] = meta.audio_timebase.numerator |
| 193 | video_meta["audio_denominator"] = meta.audio_timebase.denominator |
no test coverage detected