Functions — github.com/delta-io/kafka-delta-ingest

hub / github.com/delta-io/kafka-delta-ingest / functions

Functions298 in github.com/delta-io/kafka-delta-ingest

⨍ Functions 298◇ Types & classes 61

↓ 21 callersMethodlen

Get the length of the current cursor slice

src/cursor.rs:60

↓ 21 callersFunctionwait_until_version_created

(table: &str, version: i64)

tests/helpers/mod.rs:365

↓ 20 callersFunctionsend_json

(producer: &FutureProducer, topic: &str, json: &Value)

tests/helpers/mod.rs:80

↓ 15 callersFunctionadd

(buffers: &mut ValueBuffers, offset: i64)

src/value_buffers.rs:237

↓ 14 callersMethodrecord_stat

Records a metric for the given [`StatType`] with the given value.

src/metrics.rs:160

↓ 13 callersMethodwrite

Writes the given values to internal parquet buffers for each represented partition.

src/writer.rs:390

↓ 12 callersMethodis_empty

return true if the cursor is empty (self.len() == 0)

src/cursor.rs:65

↓ 11 callersFunctioncreate_and_run_kdi

( app_id: &str, schema: Value, delta_partitions: Vec<&str>, kafka_num_partitions: i32, opt

tests/helpers/mod.rs:201

↓ 10 callersMethodseek

(&mut self, pos: SeekFrom)

src/cursor.rs:80

↓ 9 callersFunctioninit_logger

()

tests/helpers/mod.rs:320

↓ 9 callersFunctionread_table_content_at_version_as

( table_uri: &str, version: i64, )

tests/helpers/mod.rs:396

↓ 9 callersMethodrecord_one

Records a count of 1 for the metric.

src/metrics.rs:155

↓ 8 callersFunctioncreate_local_table

(schema: Value, partitions: Vec<&str>, table_name: &str)

tests/helpers/mod.rs:243

↓ 8 callersFunctioncreate_producer

()

tests/helpers/mod.rs:73

↓ 8 callersFunctioncreate_topic

(topic: &str, num_partitions: i32)

tests/helpers/mod.rs:47

↓ 8 callersFunctiontest_broker

* Return the KAFKA_BROKERS set in the environment or default ot the local machine's port 9092 */

tests/helpers/mod.rs:35

↓ 7 callersFunctionassert_defaults

(msg: &TestMsg)

tests/deserialization_tests.rs:389

↓ 7 callersFunctioncreate_kdi

( topic: &str, table: &str, options: IngestOptions, )

tests/helpers/mod.rs:301

↓ 7 callersFunctiondefault_schema

()

tests/deserialization_tests.rs:395

↓ 7 callersFunctionget_u8_range

Create a SliceableCursor of all u8 values in ascending order

src/cursor.rs:150

↓ 7 callersFunctionread_table_content_as_jsons

(table_uri: &str)

tests/helpers/mod.rs:407

↓ 6 callersMethodcreate_and_start

(&self, name: &str)

tests/helpers/mod.rs:598

↓ 6 callersFunctioncreate_generator

(starting_id: u64)

tests/offset_tests.rs:319

↓ 6 callersMethodflush

(&mut self)

src/cursor.rs:132

↓ 6 callersMethodtransform

Transforms a [`Value`] according to the list of transforms used to create the [`Transformer`]. The optional `kafka_message` must be provided to includ

src/transforms.rs:391

↓ 5 callersFunctioncheck_read_all

Reads all the bytes in the slice and checks that it matches the u8 range from start to end_included

src/cursor.rs:156

↓ 5 callersFunctioncreate_coercion_tree

Returns a [`CoercionTree`] so the schema can be walked efficiently level by level when performing conversions.

src/coercions.rs:30

↓ 5 callersFunctioniso8601_from_epoch

(epoch_unit: EpochUnit)

src/transforms.rs:243

↓ 5 callersFunctionsend_encoded

(producer: &FutureProducer, topic: &str, content: Vec<u8>)

tests/helpers/mod.rs:87

↓ 4 callersMethodadd

Adds a value to in-memory buffers and tracks the partition and offset.

src/value_buffers.rs:14

↓ 4 callersFunctioncleanup_kdi

(topic: &str, table: &str)

tests/helpers/mod.rs:196

↓ 4 callersMethodconsume

Returns values, partition offsets and partition counts currently held in buffer and resets buffers to empty.

src/value_buffers.rs:43

↓ 4 callersFunctioncreate_generator

(staring_id: u64)

tests/buffer_flush_tests.rs:156

↓ 4 callersFunctiondefault_settings

()

tests/deserialization_tests.rs:349

↓ 4 callersFunctioni64_from_args

( args: &[Rcvar], context: &mut Context, position: usize, )

src/transforms.rs:291

↓ 4 callersMethodsend_messages

(&self, amount: i32)

tests/helpers/mod.rs:613

↓ 4 callersFunctiontxn_app_id_for_partition

(app_id: &str, partition: DataTypePartition)

src/delta_helpers.rs:70

↓ 4 callersFunctionwrite_offsets_to_delta

Write provided seeking offsets as a new delta log version with a set of `txn` actions. The `txn` id for each partition is constructed as `<app_id>-<pa

src/offsets.rs:36

↓ 4 callersMethodwrite_parquet_files

Writes the existing parquet bytes to storage and resets internal state to handle another file.

src/writer.rs:447

↓ 3 callersFunctioncreate_runtime

(name: &str)

tests/helpers/mod.rs:309

↓ 3 callersFunctionget_subcommand_matches

(args: Vec<&str>)

src/main.rs:605

↓ 3 callersFunctionget_subcommand_matches_raw

(args: Vec<&str>)

src/main.rs:610

↓ 3 callersFunctionload_table

( table_uri: &str, options: HashMap<String, String>, )

src/delta_helpers.rs:6

↓ 3 callersFunctionrecord_batch_from_json

Creates an Arrow RecordBatch from the passed JSON buffer.

src/writer.rs:605

↓ 3 callersFunctionsend_bytes

(producer: &FutureProducer, topic: &str, bytes: &Vec<u8>)

tests/helpers/mod.rs:104

↓ 3 callersFunctionset_value

(object: &mut Map<String, Value>, path: &ValuePath, path_index: usize, value: Value)

src/transforms.rs:343

↓ 3 callersFunctionstart_ingest

Executes a run loop to consume from a Kafka topic and write to a Delta table.

src/lib.rs:332

↓ 3 callersFunctionstring_to_timestamp

Convert a given datetime looking string into microseconds using chrono's [DateTime] parsing Since this will convert to microseconds, if the value is

src/coercions.rs:128

↓ 3 callersFunctiontest_aws_endpoint

()

tests/helpers/mod.rs:39

↓ 3 callersFunctiontest_s3_bucket

()

tests/helpers/mod.rs:43

↓ 3 callersFunctionwait_until_file_created

(path: &FilePath)

tests/helpers/mod.rs:348

↓ 2 callersFunctionapply_null_counts

( partition_columns: &[String], array: &StructArray, null_counts: &mut HashMap<String, ColumnCount

src/writer.rs:711

↓ 2 callersFunctionarrow_array_from_bytes

( data_type: DataType, capacity: usize, byte_arrays: Vec<&[u8]>, )

src/writer.rs:1020

↓ 2 callersMethodarrow_schema

Returns the arrow schema representation of the delta table schema defined for the wrapped table.

src/writer.rs:499

↓ 2 callersMethodbuffer_len

Returns the current byte length of the in memory buffer. This may be used by the caller to decide when to finalize the file write.

src/writer.rs:431

↓ 2 callersFunctionbuild_app

()

src/main.rs:334

↓ 2 callersFunctioncalculate_lag

Calculates lag for all partitions in the given list of partition offsets.

src/lib.rs:705

↓ 2 callersMethodcalculate_lag_metrics

Calculates total, max, min and num_partitions from the vector of lags.

src/metrics.rs:140

↓ 2 callersFunctioncoerce

Applies all data coercions specified by the [`CoercionTree`] to the [`Value`]. Though it does not currently, this function should approximate or impro

src/coercions.rs:70

↓ 2 callersFunctioncollect_partial_write_failure

( partial_writes: &mut Vec<(Value, ParquetError)>, writer_result: Result<(), Box<DataWriterError>>, )

src/writer.rs:641

↓ 2 callersFunctionconvert_matches_to_message_format

( ingest_matches: &ArgMatches, )

src/main.rs:492

↓ 2 callersFunctioncreate_txn_action

(txn_app_id: String, offset: DataTypeOffset)

src/delta_helpers.rs:29

↓ 2 callersMethoddata

Returns a clone of the underlying buffer

src/cursor.rs:120

↓ 2 callersFunctionget_high_watermark_map

Fetches high watermarks (latest offsets) with partitions from Kafka from the iterator of partitions.

src/lib.rs:1365

↓ 2 callersFunctionjson_listify_table_content

(table: DeltaTable, store: ObjectStoreRef)

tests/helpers/mod.rs:422

↓ 2 callersFunctionlast_txn_version

Returns the last transaction version for the given transaction id recorded in the delta table.

src/delta_helpers.rs:75

↓ 2 callersFunctionmsgs_to_values

(values: Vec<TestMsg>)

tests/delta_partitions_tests.rs:130

↓ 2 callersMethodnonempty_partition_offsets

Returns a copy of the current partition offsets as a [`HashMap`] for all partitions that have an offset stored in memory. Partitions that do not have

src/lib.rs:1218

↓ 2 callersFunctionparse_seek_offsets

(val: &str)

src/main.rs:321

↓ 2 callersFunctionparse_transform

(val: &str)

src/main.rs:302

↓ 2 callersFunctionparse_tuple

parse argument as a duple and let clap format the error in case of invalid syntax. this function is used both as a validator in the clap config, and t

src/main.rs:309

↓ 2 callersMethodreset

Resets internal state.

src/writer.rs:493

↓ 2 callersMethodreset_state

Resets all current state to the correct starting points represented by the current partition assignment.

src/lib.rs:1028

↓ 2 callersFunctionrun_emails_azure_tests

(initiate_rebalance: bool)

tests/emails_azure_blob_tests.rs:29

↓ 2 callersFunctionrun_emails_s3_tests

(initiate_rebalance: bool)

tests/emails_s3_tests.rs:33

↓ 2 callersMethodshutdown

(self)

tests/helpers/mod.rs:592

↓ 2 callersFunctionsubstr

(args: &[Rcvar], context: &mut Context)

src/transforms.rs:216

↓ 2 callersFunctiontimestamp_to_delta_stats_string

Vendored from delta-rs since it's no longer a public API

src/writer.rs:1137

↓ 2 callersFunctionto_schema_source

( input: Option<&String>, disable_files: bool, )

src/main.rs:193

↓ 2 callersFunctiontry_create_checkpoint

( table: &mut DeltaTable, version: i64, )

src/delta_helpers.rs:42

↓ 2 callersMethodvalidate_data

(&self)

tests/helpers/mod.rs:655

↓ 2 callersMethodwait_on_total_offset

(&self, apps: Vec<String>, offset: i32)

tests/helpers/mod.rs:630

↓ 2 callersMethodwrite_dead_letter

Writes one [DeadLetter] to the [DeadLetterQueue].

src/dead_letters.rs:156

↓ 2 callersMethodwrite_dead_letters

( &mut self, _dead_letters: Vec<DeadLetter>, )

src/dead_letters.rs:192

↓ 2 callersMethodwrite_record_batch

Writes the record batch in-memory and updates internal state accordingly. This method buffers the write stream internally so it can be invoked for man

src/writer.rs:252

↓ 2 callersMethodwrite_values

Writes the given JSON buffer and updates internal state accordingly. This method buffers the write stream internally so it can be invoked for many jso

src/writer.rs:196

↓ 1 callersFunctionalter_schema

(table: &str, version: i64, schema: Value, partitions: Vec<&str>)

tests/schema_update_tests.rs:115

↓ 1 callersFunctionapply_coercion

(value: &mut Value, node: &CoercionNode)

src/coercions.rs:80

↓ 1 callersFunctionapply_expression_transform

( object_mut: &mut Map<String, Value>, message_variable: &Variable, value_path: &ValuePath, ex

src/transforms.rs:429

↓ 1 callersFunctionapply_kafka_meta_transform

( object_mut: &mut Map<String, Value>, value_path: &ValuePath, kafka_message: &M, meta_propert

src/transforms.rs:441

↓ 1 callersFunctionapply_min_max_for_column

( statistics: &[&Statistics], column_descr: Arc<ColumnDescriptor>, column_path_parts: &[String],

src/writer.rs:790

↓ 1 callersFunctionapply_null_counts_for_column

( partition_columns: &[String], null_counts: &mut HashMap<String, ColumnCountStat>, nest_level: i3

src/writer.rs:744

↓ 1 callersMethodare_partition_offsets_match

Returns a boolean indicating whether the partition offsets currently held in memory match those stored in the delta log.

src/lib.rs:1149

↓ 1 callersMethodassigned_partitions

Returns the full list of assigned partitions as a [`Vec`] whether offsets are recorded for them in-memory or not.

src/lib.rs:1212

↓ 1 callersFunctionavro_encode

(item: impl Serialize, topic: String)

tests/deserialization_tests.rs:353

↓ 1 callersMethodbatch_completed

increments a counter for record batch completed. records a guage stat for buffered record batches. records a timer stat for record batch write duratio

src/metrics.rs:69

↓ 1 callersMethodbatch_started

increments a counter for record batch started

src/metrics.rs:62

↓ 1 callersMethodbuffer_lag

records total, max, and min consumer lag for offsets held in buffer. also records the number of partitions represented by the buffer lag vector.

src/metrics.rs:104

↓ 1 callersMethodbuffered_record_batch_count

(&self)

src/lib.rs:1173

↓ 1 callersFunctionbuild_actions

( partition_offsets: &HashMap<DataTypePartition, DataTypeOffset>, app_id: &str, mut add: Vec<Add>,

src/delta_helpers.rs:15

next →1–100 of 298, ranked by callers