Functions — github.com/IronSpiderMan/MuseTalkPlus

hub / github.com/IronSpiderMan/MuseTalkPlus / functions

Functions92 in github.com/IronSpiderMan/MuseTalkPlus

⨍ Functions 92 ◇ Types & classes 21 ↳ Endpoints 1

↓ 5 callersFunctionread_images

根据图像的文件列表，使用多线程读取图像，返回图像RGB模式的ndarray列表

common/utils.py:74

↓ 3 callersMethodanalysis

(self, image: str)

musetalk/faces/face_analysis.py:14

↓ 3 callersMethodextract_features

(self, audio_path, return_tensor=False)

musetalk/audio/feature_extractor.py:21

↓ 3 callersMethodface_location

(key_points, shift: Optional[int] = 15)

musetalk/faces/face_analysis.py:43

↓ 3 callersMethodinference

(self, audio_path: Optional[str], text: Optional[str], batch_size=4)

musetalk/avatar.py:183

↓ 3 callersMethodload_frame

(self, video_name, frame_idx, half_masked=False)

musetalk/datasets.py:125

↓ 2 callersMethod__init__

(self, cin, cout, kernel_size, stride, padding, residual=False, *args, **kwargs)

musetalk/models/sync_net.py:29

↓ 2 callersMethodincrease_idx

(self)

musetalk/avatar.py:225

↓ 2 callersMethodload_audio_feature_with_window

(self, video_name, frame_idx: int)

musetalk/datasets.py:101

↓ 2 callersFunctionsave_model

checkpoint_infos: { "minimal_loss": 0.0, "checkpoints": [ {'loss': train_loss, 'iters': iters, 'epoch': epoch + 1, 'f

musetalk/utils.py:60

↓ 2 callersMethodsort_files

(files)

musetalk/datasets.py:45

↓ 2 callersFunctiontts

(message, voice="zh-CN-XiaoxiaoNeural")

common/utils.py:92

↓ 2 callersFunctionvideo2images

(vid_path, save_path)

common/utils.py:25

↓ 1 callersMethod__init__

(self, model_path)

musetalk/models/musetalk.py:27

↓ 1 callersMethodcreate_smooth_curve

(points)

musetalk/faces/face_analysis.py:37

↓ 1 callersFunctiondatagen

( whisper_chunks, vae_encode_latents, batch_size=8, delay_frames=0, )

musetalk/utils.py:12

↓ 1 callersMethodde_process

(self, image: torch.Tensor)

musetalk/processors.py:29

↓ 1 callersFunctionevaluate

(model, val_loader, loss_fn)

scripts/train_syncnet.py:92

↓ 1 callersFunctionevaluate

(models, val_loader, gamma=2.0, output_dir="outputs")

scripts/train_musetalk.py:129

↓ 1 callersMethodextract_features

( self, audio: Union[str, np.ndarray, torch.Tensor], audio_window=2 )

musetalk/audio/audio_feature_extract.py:76

↓ 1 callersMethodface_landmark_mask

(image_size: [int, int], key_points)

musetalk/faces/face_analysis.py:22

↓ 1 callersMethodfrom_yaml

(cls, file_path: str)

common/setting.py:60

↓ 1 callersFunctiongenerate_from_txt

This function reads a txt file, splits the content into sentences, and then generates audio files for each sentence. Parameters --------

train/whisper/generate_dataset.py:72

↓ 1 callersFunctionget_compressed_image_data

(image, max_width=450, max_height=450)

server.py:39

↓ 1 callersMethodinit_avatar

(self)

musetalk/avatar.py:75

↓ 1 callersMethodinit_directories

(self)

musetalk/avatar.py:106

↓ 1 callersMethodload_filenames

(self)

musetalk/datasets.py:69

↓ 1 callersMethodload_filenames_from_json

(self)

musetalk/datasets.py:48

↓ 1 callersMethodload_frames

(self, video_name, frame_idx: int)

musetalk/datasets.py:114

↓ 1 callersFunctionmain

()

musetalk/avatar.py:254

↓ 1 callersFunctionmain

()

scripts/train_syncnet.py:153

↓ 1 callersFunctionmain

()

scripts/prepare_dataset.py:172

↓ 1 callersFunctionmain

()

scripts/preprocess.py:79

↓ 1 callersFunctionmain

()

scripts/train_musetalk.py:239

↓ 1 callersFunctionmain

()

train/whisper/generate_dataset.py:114

↓ 1 callersFunctionmake_multiple_dirs

(path_list, exist_ok=True)

common/utils.py:58

↓ 1 callersMethodnext_frame

(self)

musetalk/avatar.py:229

↓ 1 callersFunctionparse_args

()

scripts/train_syncnet.py:108

↓ 1 callersFunctionparse_args

()

scripts/prepare_dataset.py:147

↓ 1 callersFunctionparse_args

()

scripts/preprocess.py:59

↓ 1 callersFunctionparse_args

()

scripts/train_musetalk.py:179

↓ 1 callersFunctionparse_args

()

train/whisper/generate_dataset.py:99

↓ 1 callersMethodprepare_avatar

(self)

musetalk/avatar.py:129

↓ 1 callersFunctionprocess_video

video_path: 视频路径 face_shift: 人脸往上偏移的像素值，如果为None则偏移到额头 return { "image_files": ['path/to/image01', 'path/to/image02'], "a

scripts/prepare_dataset.py:32

↓ 1 callersFunctionprocess_video

(video_path)

scripts/preprocess.py:25

↓ 1 callersFunctionprocess_videos

(video_dir="./datasets/videos", face_shift=None, test_split=0.2, include_latents=False)

scripts/prepare_dataset.py:128

↓ 1 callersFunctionprocess_videos

(video_dir="./datasets/videos", face_shift=None, test_split=0.2)

scripts/preprocess.py:53

↓ 1 callersFunctionrecreate_multiple_dirs

(path_list)

common/utils.py:69

↓ 1 callersFunctionremove_multiple_dirs

(path_list)

common/utils.py:63

↓ 1 callersFunctionsplit_sentence

(text, min_length=5)

train/whisper/generate_dataset.py:33

↓ 1 callersFunctionsvc_tts

(text: str, speaker: str)

webui.py:30

↓ 1 callersFunctiontraining_loop

( model, train_loader, test_loader, optimizer, loss_fn, epochs, accelerator, max_grad_norm=1,

scripts/train_syncnet.py:30

↓ 1 callersFunctiontraining_loop

( models, train_loader, test_loader, optimizer, epochs, accelerator, gamma=2.0, max_grad_norm=

scripts/train_musetalk.py:33

↓ 1 callersFunctiontts

Pass a text and voice to generate an audio file. The filename will be generated using the text and voice, so when the text and voice are the

train/whisper/generate_dataset.py:46

↓ 1 callersMethodvalidate_avatar

validate if this avatar is valid a valid avatar should have directories named full_images, full_masks, vid_output and files n

musetalk/avatar.py:113

↓ 1 callersFunctionvideo2audio

(vid_path, save_path)

common/utils.py:40

Method__call__

(self, image, half_mask=False)

musetalk/processors.py:17

Method__call__

(self, features: List[Dict[str, Union[List[int], torch.Tensor]]])

scripts/train_whisper.py:53

Method__call__

(self, features: List[Dict[str, Union[List[int], torch.Tensor]]])

train/whisper/fine_tuning.py:54

Method__getitem__

(self, idx)

musetalk/datasets.py:133

Method__getitem__

(self, item)

musetalk/datasets.py:160

Method__init__

(self, image_size=256)

musetalk/processors.py:8

Method__init__

( self, audio_window=2, reference_window=5, sync_t=5,

musetalk/datasets.py:24

Method__init__

avatar_id: avatar的唯一标识 video_path: 视频路径

musetalk/avatar.py:27

Method__init__

(self, model_name_or_path, device='cuda', dtype=torch.float16)

musetalk/audio/feature_extractor.py:10

Method__init__

( self, n_mels: int, n_ctx: int, n_state: int, n_head: int, n_layer: int )

musetalk/audio/audio_feature_extract.py:12

Method__init__

(self, model_path, device, dtype)

musetalk/audio/audio_feature_extract.py:53

Method__init__

(self, config_path, model_path)

musetalk/faces/face_analysis.py:11

Method__init__

(self, margin=1.0)

musetalk/models/sync_net.py:9

Method__init__

(self)

musetalk/models/sync_net.py:47

Method__init__

(self, d_model=384, max_len=5000)

musetalk/models/musetalk.py:9

Method__len__

(self)

musetalk/datasets.py:130

Functioncompute_metrics

(pred)

scripts/train_whisper.py:33

Functioncompute_metrics

(pred)

train/whisper/fine_tuning.py:34

Methodforward

x : torch.Tensor, shape = (batch_size, n_mels, n_ctx) the mel spectrogram of the audio

musetalk/audio/audio_feature_extract.py:25

Methodforward

(self, output1, output2, label)

musetalk/models/sync_net.py:13

Methodforward

(self, x)

musetalk/models/sync_net.py:38

Methodforward

(self, inputs)

musetalk/models/sync_net.py:108

Methodforward

(self, x)

musetalk/models/musetalk.py:19

Methodforward

(self, inputs)

musetalk/models/musetalk.py:34

Functionimages2video

(images_dir, output, fps=25)

Functionmerge_audio_video

(video_path, audio_path, output_path)

musetalk/utils.py:43

Functionprepare_dataset

Function to preprocess the dataset with the .map method

scripts/train_whisper.py:14

Functionprepare_dataset

Function to preprocess the dataset with the .map method

train/whisper/fine_tuning.py:15

Methodsave_to_yaml

(self, file_path: str)

common/setting.py:70

Methodshift_bbox

(self, xyxy)

musetalk/avatar.py:98

Functiontalk

(text: str, background_tasks: BackgroundTasks)

Functionwebsocket_endpoint

(websocket: WebSocket)

server.py:55