Simple test class that is similar to an EventImpl.

This just counts the number of entries appended. Unlike the CounterSink, this sink does not reset the count when opened/closed

This accumulates ack tags in rollAckMap so that they can be pushed to the master when the the hdfs file associated with the rolltag is closed.

This tracks ack on batches of events. This looks for and tracks begin and end message for ack tagged values. Will this be a reasonable size? 8 bytes

This first sends a tag/checksum start message with the initial time stamp as the checksum. Then it tags each message that passes through with a hash o

When this decorator encounters a correct checksum pair, it add registers the checksume to the specified ack listener.

This a call back that will record if a proper ack start and ack end have been encountered.

This is a interface decouple event-stream extracted notifications from the actions that are triggered by them. This explicitly encapsulates any necess

Wrapper class to provide XmlRootElement.

This interface represents the client side of master control RPC exchange. We implement this interface for each RPC package that Flume uses. We interpr

Avro implementation of the Flume admin control RPC. This class manages the connection to a master and provides type conversion.

Thrift implementation of the Flume admin control RPC. This class manages the connection to a master and provides type conversion.

This build "auto agents" (automatically generated fail over chains) at different reliability levels

This is the default agent sink in a agent/collector architecture. This is purposely setup as a layer of indirection between the actual implementation

Apache Access Logs are configurable to have custom formats. The default (which we parse here) is : Default configured to be: LogFormat "%h %l %u %t \

Some array helper methods not found in apache commons or guava.

This generates an HBase output sink which puts event attributes into HBase record based on their names. It is similar to {@link com.cloudera.flume.han

A synthetic source that just creates a specified number of random events with a specified number of attributes with specified attribute size, and spec

This is a singleton that keeps a mapping of attribute to a type. This is a like a schema, and is for formatting used in reports.

This uses the Avro's Reflection serializer and DataFileWriter to output Flume events to a AvroDataFile. Under the hood this uses the BinaryEncoder for

This utility class contains methods that convert Avro-generated AvroFlumeEvents into an FlumeEvents and vice versa.

This is a sink that sends events to a remote host/port using Avro.

This sets up the port that listens for incoming flumeAvroEvent rpc calls using Avro. This class pretty much mimics ThriftEventSource.

This uses the Avro's Reflection serializer to output Flume events as json. Instead of using DataFileWriter we just write to a file and break records u

Avro implementation of a SinlgeMasterRPC. This manages the connection to an Avro server and manages type translation.

This writes native Avro formatted files out as an output format. Note: There is a separate Avro container that does encoding currently from the AvroE

Serves reports over Avro.

This failover sink initially opens the primary and the backup attempts to to append to the primary. If the primary fails, it falls back and appends to

This is the interface for state containing backoff algorithms. Basically, when a new algorithm is instantiated, reset is called. Reset should be calle

A do-nothing Sink that has default name (class name) and default report.

This takes many events and then batches them up into one batched event. If a small number of events has been received and a specified latency has pass

Simple class to consolidate benchmarking and timing code. This will help me find the bottlenecks. Usage: create new benchmark. call mark with comment

Benchmark tests on basic decorators.

This sets up a batttery of synthetic datasets for testing against different decorators and sinks. Generally, each test requires ~2GB mem. ~1GB for kee

This injects events tags that are inserted in the data stream that mark the beginning and end of different data streams. The a corresponding Benchmark

This extracts benchmark events inserted by the BenchmarkInjectDecorator. There should be a start event followed by many normal events, and eventually

BloomCheckDeco records checksums of the body of all the events received in a set represented as a bloom filter. This is intended to be downstream from

This decorator takes hashes of messages and then inserts them into a bloom filter. On deco close, the bit map representation of the bloom filter is tr

This is an interface for a bloom set. Currently this shim wraps the o.a.h.util.bloom.BloomFilter with a more generic API but uses inefficient calls.

This is a cheat class to present a CharSequence interface backed by a byte[]/ByteBuffer. This allows us to use regexes against the byte[]. Note, we ar

This is similar to a ByteArrayInputStream but instead wraps a ByteBuffer

This is a simple LRU cache implementation. It uses SoftReferences that allows the GC can collect memory from here if it needs memory. This assumes wr

This issues simple incrementing integer toString as a tag for the next wal file.

This provides a simple reusable exponential backoff state object. Note that this only does calculations and tracks the backoff state but doesn't actua

This is singleton object used to construct chart generators. It defaults to GoogleCharts based charts, but can be overridden with other chart generati

This thread takes checkConfig commands form the q and processes them. We purposely want to decouple the heartbeat from this thread.

This class simply checks that the current Java version is no older than 1.6.

This sink will thrown an exception on open if a empty or invalid context is threaded through to it.

This decorator adds a the capability to Throttle the data going out of the sink. Each ChokeDecorator is associated with a chokeId, and all the choke-d

This class contains the basic data elements of a ChokeDecorator and simple operations to perform on that data.

Main throttling Logic is here. All the choke-decorators have to call a method (deleteItems) of this class before calling the append of their super cla

This is a singleton class that wraps time functions so that they can be mock'ed out later for testing. Users are expected to only use the static meth

This takes a input stream (stdin) and interposes on the read call. The jline console will exit a readline if a CTRL_D (EOF) is "read" from the source,

This is for a collector to send notify the master of acknowledgements from successfully received tagged groups.

This collector sink is the high level specification a user would use. The subsink spec allows for specifying batching, gunzip, multiple sinks or whate

This is the default collector source in a agent/collector architecture. This is purposely setup as a layer of indirection between the actual implemen

This is a simple command that is fed to the Command Manager. This is an abstraction layer that allows new commands to be plugged into the flume Master

Builds a command object from a string. Needed for sane command line parsing. Unquoted tokens can contain alphanumeric, '.',':','_', or '-'. Tokens en

Super simple class to aggregate some command metadata.

A new exception for flume's command line parser.

We want to serialize the order of configuration commands sent to the flume configuration server. To do this we break up actions into Commands, and sub

This keeps a command it runtime information about the command. Think of it like a process control block in a OS. The command can be in 4 states. QUEU

Augments Pair with the Comparable interface so that we may create sorted sets of pairs. L and R types must implement Comparable.

This sink takes a data flow spec string as a constructor argument. It parses the construction and instantiates the sink. During execution, all the sin

JSP forms populate java bean. We want elements to be commands, so we take the bean and have a toCommand method that creates a Command that can be subm

This maintains the global configuration state of the flume nodes.

This class abstracts away the persistence of global node config data.

This is an interface for the object that manages configurations for nodes in flume. TODO (jon) we should be consistent with our naming. Maybe Configu

This is an implementation of a consistent hash. T is the type of a bin. It is mostly copied from Tom White's implementation found here: http://www.le

This is uses a consistent hash mechanism to generate failover chains for a particular agent.

Consistent hash lets you place bins and map values to bins. However, how does one get a bin and find out what values are there? For that we keep track

Simple print to console. This defaults to the "debug" output format instead of raw to give interactive users more feedback and information.

This context is an abstraction for scopes. Prior to this there were one scopes the physical node (essentially conf file). This allows for new subscope

The jetty server cannot properly reload contexts if it attempts to bind to a port and fails. To support automatically going finding a new port, we thu

This counts elements that are appended in each epoch.

This just counts the number of entries appended.

This is a bean interface for JSP form interaction.

This provides a simple reusable exponential backoff state object. Note that this only does calculations and tracks the backoff state but doesn't actua

To support multiple tail readers, we have a Cursor for each file name It takes a File and optionally a starting offset in the file. From there it att