MCPcopy Create free account
hub / github.com/ray-project/ray / _drop_empty_block_split

Function _drop_empty_block_split

python/ray/data/_internal/split.py:138–152  ·  view source on GitHub ↗

drop split indices that creates empty block split. This could happen when there are duplicated indices, or index equal to 0 (start of the block) or num_block_rows (end of the block).

(block_split_indices: List[int], num_rows: int)

Source from the content-addressed store, hash-verified

136
137
138def _drop_empty_block_split(block_split_indices: List[int], num_rows: int) -> List[int]:
139 """drop split indices that creates empty block split. This could happen when there
140 are duplicated indices, or index equal to 0 (start of the block) or num_block_rows
141 (end of the block).
142 """
143 prev_index = -1
144 optimized_indices = []
145 for index in block_split_indices:
146 if index == 0 or index == num_rows:
147 continue
148 if index == prev_index:
149 continue
150 optimized_indices.append(index)
151 prev_index = index
152 return optimized_indices
153
154
155def _split_all_blocks(

Callers 2

_split_all_blocksFunction · 0.85

Calls 1

appendMethod · 0.45

Tested by 1

Used in the wild real call sites across dependent graphs

searching dependent graphs…