MCPcopy Create free account
hub / github.com/apache/singa / load_dataset

Function load_dataset

examples/singa_peft/examples/data/mnist.py:36–51  ·  view source on GitHub ↗
(dir_path)

Source from the content-addressed store, hash-verified

34
35
36def load_dataset(dir_path):
37 dir_path = check_dataset_exist(dirpath=dir_path)
38 train_x_path = os.path.join(dir_path, 'train-images-idx3-ubyte.gz') # need to change to local disk
39 train_y_path = os.path.join(dir_path, 'train-labels-idx1-ubyte.gz') # need to change to local disk
40 valid_x_path = os.path.join(dir_path, 't10k-images-idx3-ubyte.gz') # need to change to local disk
41 valid_y_path = os.path.join(dir_path, 't10k-labels-idx1-ubyte.gz') # need to change to local disk
42
43 train_x = read_image_file(check_dataset_exist(train_x_path)).astype(
44 np.float32)
45 train_y = read_label_file(check_dataset_exist(train_y_path)).astype(
46 np.float32)
47 valid_x = read_image_file(check_dataset_exist(valid_x_path)).astype(
48 np.float32)
49 valid_y = read_label_file(check_dataset_exist(valid_y_path)).astype(
50 np.float32)
51 return train_x, train_y, valid_x, valid_y
52
53
54def read_label_file(path):

Callers 1

loadFunction · 0.70

Calls 3

check_dataset_existFunction · 0.70
read_image_fileFunction · 0.70
read_label_fileFunction · 0.70

Tested by

no test coverage detected