MCPcopy Create free account
hub / github.com/PaddlePaddle/Paddle / LocalShuffle

Method LocalShuffle

paddle/fluid/framework/data_set.cc:716–739  ·  view source on GitHub ↗

Source from the content-addressed store, hash-verified

714// do local shuffle
715template <typename T>
716void DatasetImpl<T>::LocalShuffle() {
717 VLOG(3) << "DatasetImpl<T>::LocalShuffle() begin";
718 platform::Timer timeline;
719 timeline.Start();
720
721 if (!input_channel_ || input_channel_->Size() == 0) {
722 VLOG(3) << "DatasetImpl<T>::LocalShuffle() end, no data to shuffle";
723 return;
724 }
725 auto fleet_ptr = framework::FleetWrapper::GetInstance();
726 input_channel_->Close();
727 std::vector<T> data;
728 input_channel_->ReadAll(data);
729 std::shuffle(data.begin(), data.end(), fleet_ptr->LocalRandomEngine());
730 input_channel_->Open();
731 input_channel_->Write(std::move(data));
732 data.clear();
733 data.shrink_to_fit();
734 input_channel_->Close();
735
736 timeline.Pause();
737 VLOG(3) << "DatasetImpl<T>::LocalShuffle() end, cost time="
738 << timeline.ElapsedSec() << " seconds";
739}
740
741template <typename T>
742void DatasetImpl<T>::DumpWalkPath(std::string dump_path, size_t dump_rate) {

Callers 2

PostprocessInstanceMethod · 0.80
PreprocessInstanceMethod · 0.80

Calls 11

GetInstanceFunction · 0.85
shuffleFunction · 0.85
shrink_to_fitMethod · 0.80
PauseMethod · 0.80
ElapsedSecMethod · 0.80
StartMethod · 0.45
SizeMethod · 0.45
CloseMethod · 0.45
beginMethod · 0.45
endMethod · 0.45
clearMethod · 0.45

Tested by

no test coverage detected