Start fetching data and pushing it into the channel. If this call blocks forever, the topology is permanent and acts like a long-running daemon; if th

Close releases resources allocated by the metrics client such as connections or files and flushes potentially buffered data that has not been processe

Set changes the value of a field (either standard or custom) to a new value.

Get the value of a field.

String returns a string representation of the exported fields of c.

Parse finds the next newline in data and parse log line fields from it into the current LogLine. This is the moral equivalent of bytes.Split(), but w

Set the value of a field.

Put puts buf in the cache. Either a new index, key, is created with buf, or buf is appended to the previous buffer that is indexed by key. This may

ToText converts back the LogLine to textual format and appends it to the specified buffer. If called on a default constructed LogLine (zero-value), To

Parse decodes a buffer representing a record in its data format into the current record instance. The given Metadata will be attached to that record.

Return the global (sticky) error state of the topology. Calling this function makes sense after Wait() is complete (before that, it is potentially su

Get the value of a field (either standard or custom).

Error implements the error interface.

ToText returns the reconstructed data format of a record. In case a big enough buf is passed, it will be used to serialize the record.

New creates a new BufferCache. A BufferCache is made of 2 components, a cold cache for keys that have been presented to the cache only once, and a ho

NewConfigFromToml creates a Config from a reader reading from a TOML configuration. comp describes all the existing components.

Wait until the topology shuts itself down. This can happen because the input component exits (in a batch topology), or in response to a SIGINT signal

Merge merges metrics with the same name and types from another MetricsBag into this bag. Counters are summed up, gauges are averaged, and histograms a

NewTopologyFromConfig gets a baker configuration and returns a Topology

closeFiles closes a list of files and returns the first error (if any).

Gauge sets the value of a metric of type gauge. A Gauge represents a single numerical data point that can arbitrarily go up and down.

Force the input to stop as clean as possible, at a good boundary. This is usually issued at the user's request of exiting the process. For instance, i

put copies buf into b and returns the index of the cell where it has been copied or false if the bucket is currently full.

Copy creates and returns a copy of the current record. The copied record could have been obtained by: var dst Record src.Parse(dst.ToText(), nil) bu

DisableLogging is a test helper that disable logging (in fact it sets its level to panic). It returns a function which when called, resets it to its p

Duration adds a duration to a metric of type histogram. A histogram samples observations and counts them in different 'buckets'. Duration is basically

Clear clears the record internal state, making it empty.

genBuffer generates a compressible buffer, useful for testing, since lz4 library fails on incompressible data.

AddGauge adds a metric of type gauge to the bag and updates its value. A gauge represents a single numerical data point that can arbitrarily go up and

Copy creates and returns a copy of the current log line.

AddRawCounter adds a metric of type counter to the bag and sets its current value. A counter is a cumulative metric that can only increase. To be mean

Start starts the Topology, that is start all components. This function also intercepts the interrupt signal (ctrl+c) starting the graceful shutdown (c

Set this error as fatal: it will make List stop doing any processing, and Run() will report this error as return value

AddDeltaCounter adds a metric of type counter to the bag and increments its value. A counter is a cumulative metric that can only increase. To be mean

AddHistogram adds a metrics of type histogram to the bag and tracks its value. A histogram samples observations and counts them in different 'buckets'

AddTimings adds a metric of type histogram to the bag and tracks its value. Timing is basically a histogram but allows to sample values of type time.D

Cache returns the cache that is local to the current log line.

Histogram adds a sample to a metric of type histogram. A histogram samples observations and counts them in different 'buckets' in order to track and s

Load loads from the cache the value that is mapped to key, or bool if the cache doesn't contain key.

RawCount sets the value of a metric of type counter. A counter is a cumulative metrics that can only increase. RawCount sets the current value of the

Return stats about the input

Store caches key and value in the cache.

getBit is mainly used for testing purposes

compress compresses b and returns the compressed buffer. The compressed buffer is only valid until the next call to compres. If b is not-compressible

mockS3Service returns a mocked s3.S3 service which records all operations related to Upload S3 API calls. Once all interactions with the returned ser

DeltaCountWithTags increments the value of a metric or type counter and associates that value with a set of tags.

Main runs the topology corresponding to the provided configuration. Depending on the input, it either blocks forever (daemon) or terminates when all t

Meta returns the value of the attached metadata for the given key, if any. Records implementers may implement that method by declaring: type MyRecord

Enqueue a file for processing by compressedInput. This function must be called by subclasses to schedule processing a (gzip|zstd) logfile. The functio

flush calls f with each buffer in this bucket, afterwards the bucket is considered free again.

get returns the buffer held at a specific cell. NOTE: the returned slice directly points to the bucket backing array, so the slice content may be ove

isHot/isCold reports whether l points to an entry in the hot/cold cache.

uncompressSimple decompresses b and returns the resulting buffer. The returned buffer is only valid until the next call to uncompress.

DiffWithGolden is a test helper that compares the src bytes with a file content whose path is provided in the 'golden' argument. If UpdateGolden flag

Flush flushes the whole cache.

HistogramWithTags adds a sample to an histogram and associates that sample with a set of tags.

If the path is a full s3/s3a/s3n url the extract the bucket, key and scheme from it and return them, else use the bucket from the instance, path as ke

cloneConfig clones a configuration object.

coldBucketIdx/coldCellIdx returns the bucket/cell index in cold cache.

metrics returns a snapshot of the cold cache metrics.

prepareUploadS3TestFolder creates a temp forlder and the selected number of files in it

sqliteQuote returns a manually escaped string replacing single quotes with double-single quotes

testFileWriterIntegrationCheckRecords builds and run a topology reading from testdata/filewriter/input.csv.log.zst and using the FileWriter output, co

LessLogging is a test helper that decreases logging (in fact it sets its level to Error). It returns a function which when called, resets it to its pr

Metrics returns a snapshot of the cache performance counters. This is not safe for concurrent use by multiple goroutines.

Process processes a single Record, and then optionally sends it to next filter in the chain. Process might mutate the Record, adding/modifying/removin

TempFile is a test helper that creates a temporary file, returns its name and a function which when called removes that file. This is useful to be cal