Functions — github.com/delta-io/kafka-delta-ingest

hub / github.com/delta-io/kafka-delta-ingest / functions

Functions298 in github.com/delta-io/kafka-delta-ingest

⨍ Functions 298◇ Types & classes 61

↓ 1 callersFunctionbuild_coercion_node

(data_type: &DataType)

src/coercions.rs:42

↓ 1 callersFunctioncommit_partition_offsets

( table: &mut DeltaTable, offsets: Vec<(String, DataTypeOffset)>, offsets_as_str: &str, app_id

src/offsets.rs:104

↓ 1 callersFunctioncompile_transforms

( definitions: &HashMap<String, String>, )

src/transforms.rs:95

↓ 1 callersMethodcomplete_file

Writes parquet buffers to a file in the destination delta table.

src/lib.rs:933

↓ 1 callersMethodcomplete_record_batch

Writes the transformed messages currently held in buffer to parquet byte buffers.

src/lib.rs:891

↓ 1 callersMethodconsume_timeout_duration

Returns the timeout duration to wait for the next message.

src/lib.rs:793

↓ 1 callersFunctioncreate_add

( partition_values: &HashMap<String, Option<String>>, null_counts: NullCounts, path: String, s

src/writer.rs:1040

↓ 1 callersFunctioncreate_data_topic

()

tests/dead_letter_tests.rs:189

↓ 1 callersFunctioncreate_dlq_table

()

tests/dead_letter_tests.rs:195

↓ 1 callersFunctioncreate_kdi_with

( topic: &str, table: &str, worker_name: Option<String>, options: IngestOptions, )

tests/helpers/mod.rs:265

↓ 1 callersFunctioncreate_local_table_in

(schema: Value, partitions: Vec<&str>, path: &str)

tests/helpers/mod.rs:249

↓ 1 callersFunctioncreate_metadata_action_json

(schema: Value, partitions: &[&str])

tests/helpers/mod.rs:177

↓ 1 callersFunctioncreate_options

()

tests/emails_azure_blob_tests.rs:104

↓ 1 callersFunctioncreate_options

(name: &str)

tests/emails_s3_tests.rs:79

↓ 1 callersFunctioncreate_partitions_app_ids

(num_p: i32)

tests/emails_azure_blob_tests.rs:96

↓ 1 callersFunctioncreate_partitions_app_ids

(num_p: i32)

tests/emails_s3_tests.rs:165

↓ 1 callersFunctioncreate_queue

Creates a statsd metric scope to send metrics to.

src/metrics.rs:312

↓ 1 callersFunctioncreate_table

()

src/offsets.rs:229

↓ 1 callersFunctioncreate_table

()

tests/dead_letter_tests.rs:175

↓ 1 callersFunctioncreate_temp_table

(table_path: &Path)

src/writer.rs:1266

↓ 1 callersFunctiondead_letter_queue_from_options

Creates a [`DeadLetterQueue`] to send broken messages to based on options.

src/lib.rs:1329

↓ 1 callersFunctiondelete_topic

(topic: &str)

tests/helpers/mod.rs:61

↓ 1 callersMethoddelta_file_size

records a guage for delta file size.

src/metrics.rs:98

↓ 1 callersMethoddelta_lag

records total, max, and min consumer lag for offsets written to delta. also records the number of partitions represented by the write lag vector.

src/metrics.rs:121

↓ 1 callersMethoddelta_write_completed

increments a counter for delta write started. records a timer stat for delta write duration.

src/metrics.rs:86

↓ 1 callersMethoddelta_write_failed

increments a counter for delta write failed.

src/metrics.rs:93

↓ 1 callersMethoddelta_write_started

increments a counter for delta write started

src/metrics.rs:80

↓ 1 callersMethoddeserialize

(&mut self, payload: &[u8])

src/serialization.rs:118

↓ 1 callersMethoddeserialize_message

Deserializes a message received from Kafka

src/lib.rs:872

↓ 1 callersMethoddivide_by_partition_values

( &self, records: Vec<Value>, )

src/writer.rs:545

↓ 1 callersFunctiondlq_from_opts

Factory method for creating a [DeadLetterQueue] based on the passed options. The default implementation is [NoopDeadLetterQueue]. To opt-in for the [D

src/dead_letters.rs:173

↓ 1 callersFunctionend_of_partition_reached

( message: &BorrowedMessage, offset_map: &HashMap<DataTypePartition, DataTypeOffset>, )

src/lib.rs:532

↓ 1 callersFunctionexpect_termination_within

(kdi: JoinHandle<()>, seconds: i64)

tests/helpers/mod.rs:370

↓ 1 callersFunctionextract_partition_values

( partition_cols: &[String], record_batch: &RecordBatch, )

src/writer.rs:1078

↓ 1 callersFunctionfetch_latest_offsets

( topic: &String, consumer: &Arc<StreamConsumer<KafkaContext>>, )

src/lib.rs:578

↓ 1 callersFunctionget_high_watermarks

Fetches high watermarks (latest offsets) from Kafka from the iterator of partitions.

src/lib.rs:1352

↓ 1 callersMethodhandle_counter

Records a counter metric for the given [`StatType`].

src/metrics.rs:211

↓ 1 callersMethodhandle_gauge

Records a gauge metric for the given [`StatType`].

src/metrics.rs:203

↓ 1 callersFunctionhandle_rebalance

Handles a [`RebalanceSignal`] if one exists. The [`RebalanceSignal`] is wrapped in a tokio [`RwLock`] so that it can be written to from the thread tha

src/lib.rs:605

↓ 1 callersMethodhandle_timer

Records a timer metric for the given [`StatType`].

src/metrics.rs:190

↓ 1 callersFunctioninit_logger

(app_id: String)

src/main.rs:242

↓ 1 callersMethodinsert_all

Inserts the given values immediately into the delta table. TODO: Re-using existing methods for now, but this method is batch oriented. We may be able

src/writer.rs:579

↓ 1 callersFunctioninspect_table

(path: &str)

tests/helpers/mod.rs:458

↓ 1 callersFunctionis_safe_to_commit_transactions

( table: &DeltaTable, offsets: &[(String, DataTypeOffset)], )

src/offsets.rs:150

↓ 1 callersFunctionis_utf8

(opt: Option<LogicalType>)

src/writer.rs:994

↓ 1 callersFunctionjson_encode

(value: &serde_json::Value, topic: String)

tests/deserialization_tests.rs:359

↓ 1 callersMethodjson_to_partition_values

(&self, value: &Value)

src/writer.rs:564

↓ 1 callersFunctionkafka_client_config_from_options

Creates an rdkafka [`ClientConfig`] from the provided [`IngestOptions`].

src/lib.rs:1283

↓ 1 callersMethodlen

Returns the total number of items stored across each partition specific [`ValueBuffer`].

src/value_buffers.rs:38

↓ 1 callersMethodlen

(&self)

src/transforms.rs:338

↓ 1 callersMethodmessage_deserialization_failed

increments a counter for message deserialization failed

src/metrics.rs:42

↓ 1 callersMethodmessage_deserialized

increments a counter for message deserialized

src/metrics.rs:37

↓ 1 callersMethodmessage_deserialized_size

records a guage stat for message size

src/metrics.rs:47

↓ 1 callersMethodmessage_transform_failed

increments a counter for message transform failed

src/metrics.rs:57

↓ 1 callersMethodmessage_transformed

increments a counter for message transformed

src/metrics.rs:52

↓ 1 callersFunctionmin_and_max_from_parquet_statistics

( statistics: &[&Statistics], column_descr: Arc<ColumnDescriptor>, )

src/writer.rs:853

↓ 1 callersFunctionmin_max_strings_from_stats

( stats_with_min_max: &[&Statistics], )

src/writer.rs:998

↓ 1 callersFunctionmin_max_values_from_file_metadata

( partition_values: &HashMap<String, Option<String>>, file_metadata: &FileMetaData, )

src/writer.rs:657

↓ 1 callersMethodnext_data_path

TODO: parquet files have a 5 digit zero-padded prefix and a "c\d{3}" suffix that I have not been able to find documentation for yet.

src/writer.rs:504

↓ 1 callersFunctionparse_kafka_property

(val: &str)

src/main.rs:298

↓ 1 callersMethodpart_at

(&self, index: usize)

src/transforms.rs:334

↓ 1 callersFunctionpartition_vec_from_topic_partition_list

Creates a vec of partition numbers from a topic partition list.

src/lib.rs:1341

↓ 1 callersFunctionprepare_avro_schema

(topic: String)

tests/deserialization_tests.rs:378

↓ 1 callersFunctionprepare_json_schema

(topic: String)

tests/deserialization_tests.rs:365

↓ 1 callersFunctionprepare_table

(topic: &str)

tests/emails_azure_blob_tests.rs:75

↓ 1 callersFunctionprepare_table

(topic: &str)

tests/emails_s3_tests.rs:117

↓ 1 callersMethodprocess_message

Processes a single message received from Kafka. This method deserializes, transforms and writes the message to buffers.

src/lib.rs:811

↓ 1 callersFunctionquarantine_failed_parquet_rows

( arrow_schema: Arc<ArrowSchema>, values: Vec<Value>, )

src/writer.rs:618

↓ 1 callersMethodread

(&mut self, buf: &mut [u8])

src/cursor.rs:72

↓ 1 callersFunctionread_files_from_store

Example parquet read is taken from https://docs.rs/parquet/4.1.0/parquet/arrow/index.html#example-of-reading-parquet-file-into-arrow-record-batch TODO

tests/helpers/mod.rs:111

↓ 1 callersFunctionread_table_content_as

(table_uri: &str)

tests/helpers/mod.rs:388

↓ 1 callersFunctionread_table_content_at_version_as_jsons

(table_uri: &str, version: i64)

tests/helpers/mod.rs:413

↓ 1 callersFunctionrecord_buffer_lag

Sends buffer lag to statsd.

src/lib.rs:678

↓ 1 callersFunctionrecord_write_lag

Sends delta write lag to statsd.

src/lib.rs:693

↓ 1 callersMethodremaining_slice

(&self)

src/cursor.rs:53

↓ 1 callersMethodreset_with

Resets the [`PartitionAssignment`] with a new list of partitions. Offsets are set as [`None`] for all partitions.

src/lib.rs:1195

↓ 1 callersMethodseek_consumer

Seeks the Kafka consumer to the appropriate offsets based on the [`PartitionAssignment`].

src/lib.rs:1050

↓ 1 callersMethodshould_complete_file

Returns a boolean indicating whether a delta file should be completed based on current state.

src/lib.rs:1128

↓ 1 callersMethodshould_complete_record_batch

Returns a boolean indicating whether a record batch should be written based on current state.

src/lib.rs:1103

↓ 1 callersMethodshould_process_offset

Returns a boolean indicating whether a message with `partition` and `offset` should be processed given current state.

src/lib.rs:1088

↓ 1 callersFunctionshould_record_buffer_lag

Returns a boolean indicating whether buffer lag should be reported based on the time of the last buffer lag report.

src/lib.rs:665

↓ 1 callersMethodslice

Create a slice cursor using the same data as a current one.

src/cursor.rs:40

↓ 1 callersFunctionstringified_partition_value

very naive implementation for plucking the partition value from the first element of a column array. ideally, we would do some validation to ensure th

src/writer.rs:1104

↓ 1 callersFunctiontimestamp_value_from_kafka

( kafka_timestamp: rdkafka::Timestamp, )

src/transforms.rs:467

↓ 1 callersFunctionto_schema_path

(input: Option<&String>)

src/main.rs:224

↓ 1 callersFunctionunassign_partition

( cancellation_token: Arc<CancellationToken>, consumer: Arc<StreamConsumer<KafkaContext>>, message

src/lib.rs:542

↓ 1 callersMethodupdate_offsets

Updates the offsets for each partition stored in the [`PartitionAssignment`].

src/lib.rs:1203

↓ 1 callersMethodupdate_schema

Retrieves the latest schema from table, compares to the current and updates if changed. When schema is updated then `true` is returned which signals t

src/writer.rs:370

↓ 1 callersMethodwrite_offsets_to_delta_if_any

If `opts.seek_offsets` is set then it calls the `offsets::write_offsets_to_delta` function.

src/lib.rs:802

↓ 1 callersMethodwrite_partial

( &mut self, partition_columns: &[String], arrow_schema: Arc<ArrowSchema>, jso

src/writer.rs:227

Methodbuffered_record_batch_count

Returns the number of records held in the current buffer.

src/writer.rs:485

Methodbuild_sr_settings

(registry_url: &url::Url)

src/serialization.rs:74

Functioncommit_file_path

(table: &str, version: i64)

tests/helpers/mod.rs:454

Functioncount_records

(table: DeltaTable)

tests/offset_tests.rs:91

Functioncreate_epoch_micros_to_iso8601_fn

()

src/transforms.rs:202

Functioncreate_epoch_millis_to_iso8601_fn

TODO: Consolidate these custom function factories

src/transforms.rs:195

Functioncreate_epoch_millis_to_micro_fn

()

src/transforms.rs:209

Functioncreate_epoch_seconds_to_iso8601_fn

Returns a Jmespath CustomFunction for converting an epoch timestamp number to an ISO 8601 formatted string. For example given the object: ``` { "ts"

src/transforms.rs:187

Functioncreate_substr_fn

Returns a Jmespath CustomFunction for selecting substrings from a string. This function can be registered and used within a Jmespath runtime. Logical

src/transforms.rs:149

Methodcursor_from_bytes

(bytes: &[u8])

src/writer.rs:323

← previous next →101–200 of 298, ranked by callers