Functions — github.com/antgroup/echomimic

↓ 1 callersFunctionmel_filters

load the mel filterbank matrix for projecting STFT into a Mel spectrogram. Allows decoupling librosa dependency; saved using: np.sav

src/models/whisper/whisper/audio.py:77

↓ 1 callersFunctionmulticlass_nms

Multiclass NMS implemented in Numpy. Class-aware version. Args: boxes (np.ndarray): shape=(N,4); N is number of boxes scores (np.

src/models/dwpose/onnxdet.py:43

↓ 1 callersFunctionnms

Single class NMS implemented in Numpy. Args: boxes (np.ndarray): shape=(N,4); N is number of boxes scores (np.ndarray): the score

src/models/dwpose/onnxdet.py:5

↓ 1 callersFunctionorigin_by_velocity_and_sample

(velocity, noisy_images, alphas_cumprod, timestep)

src/pipelines/step_func.py:105

↓ 1 callersMethodoutput

(result: Union[str, int])

src/models/whisper/whisper/normalizers/english.py:171

↓ 1 callersFunctionpad_or_trim

Pad or trim the audio array to N_SAMPLES, as expected by the encoder.

src/models/whisper/whisper/audio.py:52

↓ 1 callersFunctionparse_args

()

infer.py:34

↓ 1 callersFunctionparse_args

()

infer_acc.py:38

↓ 1 callersFunctionpil_to_cv2

(pil)

src/utils/img_utils.py:9

↓ 1 callersFunctionpostprocess

Postprocess for RTMPose model output. Args: outputs (np.ndarray): Output of RTMPose model. model_input_size (tuple): RTMPose mode

src/models/dwpose/onnxpose.py:80

↓ 1 callersMethodpostprocess

(self, s: str)

src/models/whisper/whisper/normalizers/english.py:410

↓ 1 callersMethodpred_prev

(self, noisy_images, model_output, timestep, timestep_prev )

src/pipelines/myschedulers.py:34

↓ 1 callersMethodprepare_extra_step_kwargs

(self, generator, eta)

src/pipelines/pipeline_echomimicv2.py:120

↓ 1 callersMethodprepare_extra_step_kwargs

(self, generator, eta)

src/pipelines/pipeline_echomimicv2_acc.py:119

↓ 1 callersMethodprepare_latents_smooth

( self, batch_size, num_channels_latents, width, height, video

src/pipelines/pipeline_echomimicv2.py:213

↓ 1 callersMethodprepare_latents_smooth

( self, batch_size, num_channels_latents, width, height, video

src/pipelines/pipeline_echomimicv2_acc.py:213

↓ 1 callersFunctionpreprocess

(img, input_size, swap=(2, 0, 1))

src/models/dwpose/onnxdet.py:98

↓ 1 callersFunctionpreprocess

Do preprocessing for RTMPose model inference. Args: img (np.ndarray): Input image in shape. input_size (tuple): Input image size

src/models/dwpose/onnxpose.py:7

↓ 1 callersMethodprocess

(self, x)

src/models/wav_clip.py:31

↓ 1 callersMethodprocess_words

(self, words: List[str])

src/models/whisper/whisper/normalizers/english.py:160

↓ 1 callersFunctionproject_points_with_trans

(points_3d, transformation_matrix, image_shape)

src/utils/motion_utils.py:66

↓ 1 callersFunctionpsuedo_velocity_wrt_noisy_and_timestep

(noisy_images, noisy_images_pre, alphas_cumprod, timestep, timestep_prev)

src/pipelines/step_func.py:86

↓ 1 callersMethodqkv_attention

(self, q: Tensor, k: Tensor, v: Tensor, mask: Optional[Tensor] = None)

src/models/whisper/whisper/model.py:88

↓ 1 callersMethodrank

Given a list of groups of samples and their cumulative log probabilities, return the indices of the samples in each group to select a

src/models/whisper/whisper/decoding.py:165

↓ 1 callersMethodrearrange_kv_cache

Update the key-value cache according to the updated beams

src/models/whisper/whisper/decoding.py:123

↓ 1 callersMethodregister_reference_hooks

( self, mode, do_classifier_free_guidance, attention_auto_machine_weight,

src/models/mutual_self_attention.py:52

↓ 1 callersFunctionremove_symbols_and_diacritics

Replace any other markers, symbols, and punctuations with a space, and drop any diacritics (category 'Mn' and some manual mappings)

src/models/whisper/whisper/normalizers/basic.py:27

↓ 1 callersMethodreset

Initialize any stateful variables for decoding a new sequence

src/models/whisper/whisper/decoding.py:200

↓ 1 callersFunctionresize_and_pad

(img, max_size)

EMTD_dataset/preprocess.py:63

↓ 1 callersFunctionresize_and_pad_param

(imh, imw, max_size)

EMTD_dataset/preprocess.py:86

↓ 1 callersFunctionsave_audio

(ori_video_path, sub_task)

EMTD_dataset/preprocess.py:255

↓ 1 callersFunctionsave_pose_params

(detected_poses, pose_params, draw_pose_params, ori_video_path)

EMTD_dataset/preprocess.py:227

↓ 1 callersFunctionsave_processed_video

(ori_frames, video_params, ori_video_path, max_size)

EMTD_dataset/preprocess.py:241

↓ 1 callersFunctionsave_videos_from_pil

(pil_images, path, fps=8, audio_path=None)

src/utils/util.py:51

↓ 1 callersFunctionscheduler_pred_onestep

( model, noisy_images, scheduler, timestep, timestep_prev, audio_cond_fea, face_musk_fea, guid

src/pipelines/step_func.py:34

↓ 1 callersMethodset_attention_slice

r""" Enable sliced attention computation. When this option is enabled, the attention module will split the input tensor in slices, to

src/models/unet_3d.py:283

↓ 1 callersMethodset_attention_slice

r""" Enable sliced attention computation. When this option is enabled, the attention module splits the input tensor in slices to c

src/models/unet_2d_condition.py:763

↓ 1 callersMethodset_attention_slice

r""" Enable sliced attention computation. When this option is enabled, the attention module will split the input tensor in slices, to

src/models/unet_3d_emo.py:283

↓ 1 callersMethodset_attn_processor

r""" Sets the attention processor to use to compute attention. Parameters: processor (`dict` of `AttentionProcessor`

src/models/unet_2d_condition.py:702

↓ 1 callersFunctionsinusoids

Returns sinusoids for positional embedding

src/models/whisper/whisper/model.py:48

↓ 1 callersFunctiontop_down_affine

Get the bbox image as the model input by affine transform. Args: input_size (dict): The input size of the model. bbox_scale (dict

src/models/dwpose/onnxpose.py:255

↓ 1 callersFunctiontranscribe

Transcribe an audio file using Whisper Parameters ---------- model: Whisper The Whisper model instance audio: Union[str

src/models/whisper/whisper/transcribe.py:19

↓ 1 callersFunctionwrite_srt

Write a transcript to a file in SRT format. Example usage: from pathlib import Path from whisper.utils import write_srt

src/models/whisper/whisper/utils.py:63

↓ 1 callersFunctionwrite_txt

(transcript: Iterator[dict], file: TextIO)

src/models/whisper/whisper/utils.py:47

↓ 1 callersFunctionwrite_vtt

(transcript: Iterator[dict], file: TextIO)

src/models/whisper/whisper/utils.py:52

Method__call__

( self, ref_image, audio_path, poses_tensor, width, height,

src/pipelines/pipeline_echomimicv2.py:409

Method__call__

( self, ref_image, audio_path, poses_tensor, width, height,

src/pipelines/pipeline_echomimicv2_acc.py:302

Method__call__

(self, s: str)

src/models/whisper/whisper/normalizers/english.py:435

Method__call__

(self, s: str)

src/models/whisper/whisper/normalizers/english.py:454

Method__call__

(self, s: str)

src/models/whisper/whisper/normalizers/english.py:519

Method__call__

(self, s: str)

src/models/whisper/whisper/normalizers/basic.py:60

Method__call__

call to process dwpose-detect Args: oriImg (np.ndarray): detected image

src/models/dwpose/wholebody.py:22

Method__call__

(self, oriImg)

src/models/dwpose/dwpose_detector.py:30

Method__getattr__

(self, key)

src/utils/util_emo.py:163

Method__init__

(self, model_path, device="cuda")

src/utils/util_emo.py:11

Method__init__

(self, auto_flow_dir, imh=512, imw=512)

src/utils/util_emo.py:25

Method__init__

(self, forehead_edge=False, upface_only=False, draw_eye=True,

src/utils/draw_utils.py:7

Method__init__

( self, vae, reference_unet, denoising_unet, audio_guider, pos

src/pipelines/pipeline_echomimicv2.py:36

Method__init__

( self, vae, reference_unet, denoising_unet, audio_guider, pos

src/pipelines/pipeline_echomimicv2_acc.py:36

Method__init__

(self)

src/pipelines/myschedulers.py:9

Method__init__

(self, eulera_scheduler, normnoise=False)

src/pipelines/myschedulers.py:135

Method__init__

( self, num_attention_heads: int = 16, attention_head_dim: int = 88, in_channe

src/models/transformer_3d.py:31

Method__init__

( self, unet, mode="write", do_classifier_free_guidance=False, attenti

src/models/mutual_self_attention.py:20

Method__init__

( self, num_attention_heads: int = 16, attention_head_dim: int = 88, in_ch

src/models/transformer_2d.py:64

Method__init__

( self, sample_size: Optional[int] = None, in_channels: int = 4, out_channels:

src/models/unet_3d.py:34

Method__init__

( self, sample_size: Optional[int] = None, in_channels: int = 4, out_chann

src/models/unet_2d_condition.py:162

Method__init__

(self, input_num=15, output_num=77, model_path=None)

src/models/wav_clip.py:35

Method__init__

( self, dim: int, num_attention_heads: int, attention_head_dim: int, d

src/models/attention.py:245

Method__init__

( self, conditioning_embedding_channels: int, conditioning_channels: int = 1,

src/models/pose_encoder.py:13

Method__init__

(self, in_channels: int, out_channels: int, act_fn: str)

src/models/unet_2d_blocks.py:203

Method__init__

( self, in_channels: int, temb_channels: int, dropout: float = 0.0,

src/models/unet_2d_blocks.py:255

Method__init__

( self, in_channels: int, temb_channels: int, dropout: float = 0.0,

src/models/unet_2d_blocks.py:358

Method__init__

( self, in_channels: int, out_channels: int, temb_channels: int,

src/models/unet_2d_blocks.py:512

Method__init__

( self, in_channels: int, out_channels: int, temb_channels: int,

src/models/unet_2d_blocks.py:683

Method__init__

( self, in_channels: int, out_channels: int, prev_output_channel: int,

src/models/unet_2d_blocks.py:781

Method__init__

( self, in_channels: int, temb_channels: int, dropout: float = 0.0, nu

src/models/unet_3d_blocks.py:172

Method__init__

( self, in_channels: int, out_channels: int, temb_channels: int, dropo

src/models/unet_3d_blocks.py:299

Method__init__

( self, in_channels: int, out_channels: int, temb_channels: int, dropo

src/models/unet_3d_blocks.py:473

Method__init__

( self, in_channels: int, out_channels: int, prev_output_channel: int,

src/models/unet_3d_blocks.py:592

Method__init__

( self, conditioning_embedding_channels: int, conditioning_channels: int = 3,

src/models/pose_guider.py:13

Method__init__

( self, sample_size: Optional[int] = None, in_channels: int = 4, out_channels:

src/models/unet_3d_emo.py:34

Method__init__

( self, channels, use_conv=False, out_channels=None, padding=1, name="conv" )

src/models/resnet.py:94

Method__init__

( self, *, in_channels, out_channels=None, conv_shortcut=False,

src/models/resnet.py:124

Method__init__

( self, in_channels, num_attention_heads=8, num_transformer_block=2, a

src/models/motion_module.py:45

Method__init__

( self, in_channels, num_attention_heads, attention_head_dim, num_laye

src/models/motion_module.py:95

Method__init__

( self, dim, num_attention_heads, attention_head_dim, attention_block_

src/models/motion_module.py:187

Method__init__

( self, attention_mode=None, cross_frame_attention_mode=None, temporal_positio

src/models/motion_module.py:282

Method__init__

( self, attention_mode=None, cross_frame_attention_mode=None, temporal_positio

src/models/motion_module.py:393

Method__init__

(self, whisper_model_type="tiny", model_path="./models/whisper/tiny.pt",

src/models/whisper/audio2feature.py:9

Method__init__

(self, model: "Whisper", initial_token_length: int)

src/models/whisper/whisper/decoding.py:133

Method__init__

(self, length_penalty: Optional[float])

src/models/whisper/whisper/decoding.py:179

Method__init__

(self, temperature: float, eot: int)

src/models/whisper/whisper/decoding.py:254

Method__init__

(self, beam_size: int, eot: int, inference: Inference, patience: Optional[float] = None)

src/models/whisper/whisper/decoding.py:282

Method__init__

(self, tokenizer: Tokenizer, sample_begin: int)

src/models/whisper/whisper/decoding.py:388

Method__init__

(self, suppress_tokens: Sequence[int])

src/models/whisper/whisper/decoding.py:398

Method__init__

( self, tokenizer: Tokenizer, sample_begin: int, max_initial_timestamp_index: Optional[int] )

src/models/whisper/whisper/decoding.py:406

Method__init__

(self, model: "Whisper", options: DecodingOptions)

src/models/whisper/whisper/decoding.py:450

Method__init__

(self, n_state: int, n_head: int)

src/models/whisper/whisper/model.py:58

Method__init__

(self, n_state: int, n_head: int, cross_attention: bool = False)

src/models/whisper/whisper/model.py:104

Method__init__

(self, n_mels: int, n_ctx: int, n_state: int, n_head: int, n_layer: int)

src/models/whisper/whisper/model.py:132

Functions395 in github.com/antgroup/echomimic_v2