hub / github.com/data-integrations/wrangler
github.com/data-integrations/wrangler @v4.11.0
Chat with this repo
3,678 symbols
14,582 edges
568 files
1,478 documented · 40%
201 cross-repo links
What it actually does AI analysis from the code graph — generated when you open this
loading…
README
Data Prep
A collection of libraries, a pipeline plugin, and a CDAP service for performing data cleansing, transformation, and filtering using a set of data manipulation instructions (directives). These instructions are either generated using an interative visual tool or are manually created.
- Data Prep defines few concepts that might be useful if you are just getting started with it. Learn about them here
- The Data Prep Transform is separately documented.
- Data Prep Cheatsheet
New Features
More here on upcoming features.
-
User Defined Directives, also known as UDD, allow you to create custom functions to transform records within CDAP DataPrep or a.k.a Wrangler. CDAP comes with a comprehensive library of functions. There are however some omissions, and some specific cases for which UDDs are the solution. Additional information on how you can build your custom directives here.
-
A new capability that allows CDAP Administrators to restrict the directives that are accessible to their users. More information on configuring can be found here
Demo Videos and Recipes
Videos and Screencasts are best way to learn, so we have compiled simple, short screencasts that shows some of the features of Data Prep. Additional videos can be found here
Videos
- [SCREENCAST] Creating Lookup Dataset and Joining
- [SCREENCAST] Restricted Directives
- [SCREENCAST] Parse Excel files in CDAP
- [SCREENCAST] Parse File As AVRO File
- [SCREENCAST] Parsing Binary Coded AVRO Messages
- [SCREENCAST] Parsing Binary Coded AVRO Messages & Protobuf messages using schema registry
- [SCREENCAST] Quantize a column - Digitize
- [SCREENCAST] Data Cleansing capability with send-to-error directive
- [SCREENCAST] Building Data Prep from the GitHub source
- [VOICE-OVER] End-to-End Demo Video
- [SCREENCAST] Ingesting into Kudu
- [SCREENCAST] Realtime HL7 CCDA XML from Kafka into Time Parititioned Parquet
- [SCREENCAST] Parsing JSON file
- [SCREENCAST] Flattening arrays
- [SCREENCAST] Data cleansing with send-to-error directive
- [SCREENCAST] Publishing to Kafka
- [SCREENCAST] Fixed length to JSON
Recipes
Available Directives
These directives are currently available:
| Directive | Description |
|---|---|
| Parsers | |
| JSON Path | Uses a DSL (a JSON path expression) for parsing JSON records |
| Parse as AVRO | Parsing an AVRO encoded message - either as binary or json |
| Parse as AVRO File | Parsing an AVRO data file |
| Parse as CSV | Parsing an input record as comma-separated values |
| Parse as Date | Parsing dates using natural language processing |
| Parse as Excel | Parsing excel file. |
| Parse as Fixed Length | Parses as a fixed length record with specified widths |
| Parse as HL7 | Parsing Health Level 7 Version 2 (HL7 V2) messages |
| Parse as JSON | Parsing a JSON object |
| Parse as Log | Parses access log files as from Apache HTTPD and nginx servers |
| Parse as Protobuf | Parses an Protobuf encoded in-memory message using descriptor |
| Parse as Simple Date | Parses date strings |
| Parse XML To JSON | Parses an XML document into a JSON structure |
| Parse as Currency | Parses a string representation of currency into a number. |
| Parse as Datetime | Parses strings with datetime values to CDAP datetime type |
| Output Formatters | |
| Write as CSV | Converts a record into CSV format |
| Write as JSON | Converts the record into a JSON map |
| Write JSON Object | Composes a JSON object based on the fields specified. |
| Format as Currency | Formats a number as currency as specified by locale. |
| Transformations | |
| Changing Case | Changes the case of column values |
| Cut Character | Selects parts of a string value |
| Set Column | Sets the column value to the result of an expression execution |
| Find and Replace | Transforms string column values using a "sed"-like expression |
| Index Split | (Deprecated) |
| Invoke HTTP | Invokes an HTTP Service (Experimental, potentially slow) |
| Quantization | Quantizes a column based on specified ranges |
| Regex Group Extractor | Extracts the data from a regex group into its own column |
| Setting Character Set | Sets the encoding and then converts the data to a UTF-8 String |
| Setting Record Delimiter | Sets the record delimiter |
| Split by Separator | Splits a column based on a separator into two columns |
| Split Email Address | Splits an email ID into an account and its domain |
| Split URL | Splits a URL into its constituents |
| Text Distance (Fuzzy String Match) | Measures the difference between two sequences of characters |
| Text Metric (Fuzzy String Match) | Measures the difference between two sequences of characters |
| URL Decode | Decodes from the application/x-www-form-urlencoded MIME format |
| URL Encode | Encodes to the application/x-www-form-urlencoded MIME format |
| Trim | Functions for trimming white spaces around string data |
| Encoders and Decoders | |
| Decode | Decodes a column value as one of base32, base64, or hex |
| Encode | Encodes a column value as one of base32, base64, or hex |
| Unique ID | |
| UUID Generation | Generates a universally unique identifier (UUID) .Recommended to use with Wrangler version 4.4.0 and above due to an important bug fix CDAP-17732 |
| Date Transformations | |
| Diff Date | Calculates the difference between two dates |
| Format Date | Custom patterns for date-time formatting |
| Format Unix Timestamp | Formats a UNIX timestamp as a date |
| DateTime Transformations | |
| Current DateTime | Generates the current datetime using the given zone or UTC by default |
| Datetime To Timestamp | Converts a datetime value to timestamp with the given zone |
| Format Datetime | Formats a datetime value to custom date time pattern strings |
| Timestamp To Datetime | Converts a timestamp value to datetime |
| Lookups | |
| Catalog Lookup | Static catalog lookup of ICD-9, ICD-10-2016, ICD-10-2017 codes |
| Table Lookup | Performs lookups into Table datasets |
| Hashing & Masking |
Extension points exported contracts — how you extend this code
DirectiveRegistry (Interface)A directive registry maintains a collection of directives either system provided or user provided. The directive inf [8 …
wrangler-core/src/main/java/io/cdap/wrangler/registry/DirectiveRegistry.java
Executor (Interface)
A interface defining the wrangle Executor in the wrangling RecipePipeline. @param type of input object @par [8 implementers]
wrangler-api/src/main/java/io/cdap/wrangler/api/Executor.java
Executor (Interface)
Executes something using a SQL connection. [8 implementers]
wrangler-service/src/main/java/io/cdap/wrangler/service/database/DatabaseHandler.java
Request (Interface)
Dataprep request body and it's structure. [2 implementers]
wrangler-proto/src/main/java/io/cdap/wrangler/proto/Request.java
AvroSchemaLoader (Interface)
The AvroSchemaLoader interface abstracts loading an AVRO data model schema definitions. [4 implementers]
wrangler-core/src/main/java/io/cdap/wrangler/utils/AvroSchemaLoader.java
Directive (Interface)
Directive is a user defined directive. DIE - Define, Initialize & Execute - Pattern Following is a simple example of ho [93 …
wrangler-api/src/main/java/io/cdap/wrangler/api/Directive.java
DatasetProvider (Interface)
This class implements the provider pattern for providing dataset at runtime. @param type of dataset provided [2 implementers]
wrangler-service/src/main/java/io/cdap/wrangler/service/explorer/DatasetProvider.java
Decoder (Interface)
Decoder interface for decoding the data. @param type of object to decode [4 implementers]
wrangler-core/src/main/java/io/cdap/wrangler/codec/Decoder.java
Core symbols most depended-on inside this repo
getcalled by 686
wrangler-core/src/main/java/io/cdap/wrangler/registry/DirectiveRegistry.java
add
called by 681
wrangler-core/src/main/java/io/cdap/wrangler/expression/ELContext.java
of
called by 673
wrangler-api/src/main/java/io/cdap/wrangler/api/lineage/Many.java
getValue
called by 501
wrangler-core/src/main/java/io/cdap/directives/parser/JsParser.java
format
called by 470
wrangler-core/src/main/java/io/cdap/functions/Global.java
size
called by 420
wrangler-api/src/main/java/io/cdap/wrangler/api/Arguments.java
value
called by 417
wrangler-api/src/main/java/io/cdap/wrangler/api/parser/Token.java
build
called by 392
wrangler-core/src/main/java/io/cdap/directives/validation/conformers/Conformer.java
Shape
Languages
Java100%
Modules by API surface
wrangler-core/src/test/java/com/example/tutorial/AddressBookProtos.java229 symbols
wrangler-core/src/main/java/io/cdap/functions/DateAndTime.java50 symbols
wrangler-core/src/test/java/io/cdap/functions/DateAndTimeTest.java36 symbols
wrangler-service/src/main/java/io/cdap/wrangler/service/database/DatabaseHandler.java30 symbols
wrangler-service/src/main/java/io/cdap/wrangler/service/directive/DirectivesHandler.java29 symbols
wrangler-core/src/main/java/io/cdap/functions/DataQuality.java28 symbols
wrangler-transform/src/main/java/io/cdap/wrangler/Wrangler.java25 symbols
wrangler-core/src/main/java/io/cdap/functions/Dates.java24 symbols
wrangler-core/src/main/java/io/cdap/wrangler/expression/EL.java22 symbols
wrangler-core/src/main/java/io/cdap/directives/nlp/internal/PorterStemmer.java22 symbols
wrangler-proto/src/main/java/io/cdap/wrangler/proto/recipe/v2/Recipe.java21 symbols
wrangler-transform/src/e2e-test/java/io/cdap/plugin/common/stepsdesign/TestSetupHooks.java20 symbols
Used by 201 indexed graphs manifest dependencies, hub-wide
… +191 more
Datastores touched
(mysql)Database · 1 repos
For agents
$ claude mcp add wrangler \
-- python -m otcore.mcp_server <graph>