Apache NiFi - MiNiFi - C++

MiNiFi is a child project effort of Apache NiFi. This repository is for a native implementation in C++.

Table of Contents

• Features
• Caveats
• Getting Started
• System Requirements
• Bootstrapping
• Building For Other Distros
• Cleaning
• Configuring
• Running
• Deploying
• Extensions
• Security
• Operations
• Issue Tracking
• Documentation
• License

Features

Apache NiFi - MiNiFi C++ is a complementary data collection approach that supplements the core tenets of NiFi in dataflow management, focusing on the collection of data at the source of its creation. The C++ implementation is an additional implementation to the one in Java with the aim of an even smaller resource footprint.

Specific goals for MiNiFi comprise: - small and lightweight footprint - central management of agents - generation of data provenance - integration with NiFi for follow-on dataflow management and full chain of custody of information

Perspectives of the role of MiNiFi should be from the perspective of the agent acting immediately at, or directly adjacent to, source sensors, systems, or servers.

Expression Language

A subset of the Apache NiFi Expression Language is supported.

Processors

MiNiFi - C++ supports the following C++ processors:

The following table lists the base set of processors.

Extension Set	Processors
Base	AppendHostInfo

ExecuteProcess

ExtractText

GenerateFlowFile

GetFile

GetTCP

HashContent

ListenSyslog

LogAttribute

PutFile

RetryFlowFile

RouteOnAttribute

TailFile

UpdateAttribute

The next table outlines CMAKE flags that correspond with MiNiFi extensions. Extensions that are enabled by default ( such as CURL ), can be disabled with the respective CMAKE flag on the command line.

Through JNI extensions you can run NiFi processors using NARs. The JNI extension set allows you to run these Java processors. MiNiFi C++ will favor C++ implementations over Java implements. In the case where a processor is implemented in either language, the one in C++ will be selected; however, will remain transparent to the consumer.

Extension Set	Processors	CMAKE Flag
Archive Extensions	ApplyTemplate

CompressContent

ManipulateArchive

MergeContent

FocusArchiveEntry

UnfocusArchiveEntry | -DBUILD_LIBARCHIVE=ON | | AWS | AWSCredentialsService

PutS3Object

DeleteS3Object | -DENABLE_AWS=ON | | CivetWeb | ListenHTTP | -DDISABLE_CIVET=ON | | CURL | InvokeHTTP | -DDISABLE_CURL=ON | | GPS | GetGPS | -DENABLE_GPS=ON | | Kafka | PublishKafka | -DENABLE_LIBRDKAFKA=ON | | JNI | NiFi Processors | -DENABLE_JNI=ON | | MQTT | ConsumeMQTT

PublishMQTT | -DENABLE_MQTT=ON | | OpenCV | CaptureRTSPFrame | -DENABLE_OPENCV=ON | | OpenWSMAN | SourceInitiatedSubscriptionListener | -DENABLE_OPENWSMAN=ON | | PCAP | CapturePacket | -DENABLE_PCAP=ON | | Scripting | ExecuteScript

Custom Python Processors | -DDISABLE_SCRIPTING=ON | | Sensors | GetEnvironmentalSensors

GetMovementSensors | -DENABLE_SENSORS=ON | | SFTP | FetchSFTP

ListSFTP

PutSFTP | -DENABLE_SFTP=ON | | SQL | ExecuteSQL

PutSQL

QueryDatabaseTable

| -DENABLE_SQL=ON | | SQLite | ExecuteSQL

PutSQL | -DENABLE_SQLITE=ON | | Tensorflow | TFApplyGraph

TFConvertImageToTensor

TFExtractTopLabels

  |    -DENABLE_TENSORFLOW=ON  |

| USB Camera | GetUSBCamera | -DENABLE_USB_CAMERA=ON | | Windows Event Log (Windows only) | CollectorInitiatedSubscription

ConsumeWindowsEventLog

TailEventLog | -DENABLE_WEL=ON |

Please see our Python guide on how to write Python processors and use them within MiNiFi C++.

Caveats

• We follow semver with regards to API compatibility, but no ABI compatibility is provided. See semver's website for more information
• Build and usage currently only supports Windows, Linux and OS X environments. MiNiFi C++ can be built and run through the Windows Subsystem for Linux but we provide no support for this platform.
• Provenance events generation is supported and are persisted using RocksDB. Volatile repositories can be used on systems without persistent storage.
• If MiNiFi C++ is built with the OPC-UA extension enabled, it bundles open62541, which is available under the Mozilla Public License Version 2.0, a Category B license under ASF 3rd party license policy.

System Requirements

To build

Utilities

• CMake
• 3.10 or greater
• gcc
• 4.8.4 or greater
• g++
• 4.8.4 or greater
• bison
• 3.0.x (3.2 has been shown to fail builds)
• flex
• 2.5 or greater

External Projects

The following utilities are needed to build external projects, when bundled versions of LibreSSL, cURL, or zlib are used:

• patch
• autoconf
• automake
• libtool

NOTE if Lua support is enabled, then a C++ compiler with support for c++-14 must be used. If using GCC, version 6.x or greater is recommended.

NOTE if bustache (ApplyTemplate) support is enabled, a recent version of a compiler supporting c++-11 must be used. GCC versions >= 6.3.1 are known to work.

NOTE if Kafka support is enabled, a recent version of a compiler supporting C++-11 regexes must be used. GCC versions >= 4.9.x are recommended.

NOTE if Expression Language support is enabled, FlexLexer must be in the include path and the version must be compatible with the version of flex used when generating lexer sources. Lexer source generation is automatically performed during CMake builds. To re-generate the sources, remove:

• extensions/expression-language/Parser.cpp
• extensions/expression-language/Parser.hpp
• extensions/expression-language/Scanner.cpp
• extensions/expression-language/location.hh
• extensions/expression-language/position.hh
• extensions/expression-language/stack.hh

and rebuild.

Libraries / Development Headers

• libcurl-openssl (If not available or desired, NSS will be used)
• libuuid and uuid-dev
• openssl
• Python 3 and development headers -- Required, unless Python support is disabled
• Lua and development headers -- Optional, unless Lua support is enabled
• libgps-dev -- Required if building libGPS support

CentOS 6

Additional environmental preparations are required for CentOS 6 support. Before building, install and enable the devtoolset-6 SCL:

$ sudo yum install centos-release-scl
$ sudo yum install devtoolset-6
$ scl enable devtoolset-6 bash

Additionally, for expression language support, it is recommended to install GNU Bison 3.0.4:

$ wget https://ftp.gnu.org/gnu/bison/bison-3.0.4.tar.xz
$ tar xvf bison-3.0.4.tar.xz
$ cd bison-3.0.4
$ ./configure
$ make
$ sudo make install

Finally, it is required to add the -lrt compiler flag by using the -DCMAKE_CXX_FLAGS=-lrt flag when invoking cmake.

On all distributions please use -DUSE_SHARED_LIBS=OFF to statically link zlib, libcurl, and OpenSSL.

SLES

SLES 11 requires manual installation of the SDK using the following link: https://www.novell.com/support/kb/doc.php?id=7015337 Once these are installed you will need to download and build CMAKE3, OpenSSL1.0, and Python3. Once these are installed follow the cmake procedures. The bootstrap script will not work.

SLES 12 requires you to enable the SDK module in YAST. It is advised that you use the bootstrap script to help guide installation. Please see the Bootstrapping section below.

Windows

Build and Installation has been tested with Windows 10 using Visual Studio 2017. You can build and create an MSI via the CPACK command. This requires the installation of the WiX toolset (http://wixtoolset.org/). To do this, open up a prompt into your build directory and type 'cpack' . The CPACK command will automatically generate and provide you a path to the distributable msi file. See Windows.md for more details.

To run

Libraries

• libuuid
• librocksdb (built and statically linked)
• libcurl-openssl (If not available or desired, NSS will be used)
• libssl and libcrypto from openssl (built and statically linked)
• libarchive (built and statically linked)
• librdkafka (built and statically linked)
• Python 3 -- Required, unless Python support is disabled
• Lua -- Optional, unless Lua support is enabled
• libusb -- Optional, unless USB Camera support is enabled
• libpng -- Optional, unless USB Camera support is enabled
• libpcap -- Optional, unless ENABLE_PCAP specified

The needed dependencies can be installed with the following commands for:

Yum based Linux Distributions

NOTE if a newer compiler is required, such as when Lua support is enabled, it is recommended to use a newer compiler using a devtools-* package from the Software Collections (SCL).

# ~/Development/code/apache/nifi-minifi-cpp on git:master
$ yum install cmake \
  gcc gcc-c++ \
  bison \
  flex \
  patch \
  autoconf \
  automake \
  libtool \
  rocksdb-devel rocksdb \
  libuuid libuuid-devel \
  boost-devel \
  openssl-devel \
  bzip2-devel \
  xz-devel \
  doxygen
$ # (Optional) for building Python support
$ yum install python34-devel
$ # (Optional) for building Lua support
$ yum install lua-devel
$ # (Optional) for building USB Camera support
$ yum install libusb-devel libpng-devel
$ # (Optional) for building docker image
$ yum install docker
$ # (Optional) for system integration tests
$ yum install docker python-virtualenv
# If building with GPS support
$ yum install gpsd-devel
$ # (Optional) for PacketCapture Processor
$ yum install libpcap-devel
$ #depending on your yum repo you may need to manually build libcurl-openssl if you do not wish to use
  libcurl with NSS support. By default we will use NSS when libcurl-openssl is not available.

Aptitude based Linux Distributions

# ~/Development/code/apache/nifi-minifi-cpp on git:master
$ apt-get install cmake \
  gcc g++ \
  bison \
  flex \
  patch \
  autoconf \
  automake \
  libtool \
  libcurl4-openssl-dev \
  librocksdb-dev librocksdb4.1 \
  uuid-dev uuid \
  libboost-all-dev libssl-dev \
  libbz2-dev liblzma-dev \
  doxygen
$ # (Optional) for building Python support
$ apt-get install libpython3-dev
$ # (Optional) for building Lua support
$ apt-get install liblua5.1-0-dev
$ # (Optional) for building USB Camera support
$ apt-get install libusb-1.0.0-0-dev libpng12-dev
$ # (Optional) for building docker image
$ apt-get install docker.io
$ # (Optional) for system integration tests
$ apt-get install docker.io python-virtualenv
# If building with GPS support
$ apt-get install libgps-dev
$ # (Optional) for PacketCapture Processor
$ apt-get install libpcap-dev

OS X Using Homebrew (with XCode Command Line Tools installed)

```

~/Development/code/apache/nifi-minifi-cpp on git:master

$ brew install cmake \ flex \ patch \ autoconf \ automake \ libtool \ rocksdb \ ossp-uuid \ boost \ openssl \ python \ lua \ xz \ bzip2 \ doxygen $ brew install curl $ brew link curl --force $ # (Optional) for building USB Camera support $ brew install libusb lib