hub / github.com/data-integrations/wrangler
github.com/data-integrations/wrangler
Chat with this repo
2,396 symbols
8,872 edges
386 files
1,113 documented · 46%
● updated 11d agov3.2.2 · 2018-12-10★ 108
Browse by type
What it actually does AI analysis from the code graph — generated when you open this
loading…
README
Data Prep
A collection of libraries, a pipeline plugin, and a CDAP service for performing data cleansing, transformation, and filtering using a set of data manipulation instructions (directives). These instructions are either generated using an interative visual tool or are manually created.
- Data Prep defines few concepts that might be useful if you are just getting started with it. Learn about them here
- The Data Prep Transform is separately documented.
- Data Prep Cheatsheet
New Features
More here on upcoming features.
-
User Defined Directives, also known as UDD, allow you to create custom functions to transform records within CDAP DataPrep or a.k.a Wrangler. CDAP comes with a comprehensive library of functions. There are however some omissions, and some specific cases for which UDDs are the solution. Additional information on how you can build your custom directives here.
-
A new capability that allows CDAP Administrators to restrict the directives that are accessible to their users. More information on configuring can be found here
Demo Videos and Recipes
Videos and Screencasts are best way to learn, so we have compiled simple, short screencasts that shows some of the features of Data Prep. Additional videos can be found here
Videos
- [SCREENCAST] Creating Lookup Dataset and Joining
- [SCREENCAST] Restricted Directives
- [SCREENCAST] Parse Excel files in CDAP
- [SCREENCAST] Parse File As AVRO File
- [SCREENCAST] Parsing Binary Coded AVRO Messages
- [SCREENCAST] Parsing Binary Coded AVRO Messages & Protobuf messages using schema registry
- [SCREENCAST] Quantize a column - Digitize
- [SCREENCAST] Data Cleansing capability with send-to-error directive
- [SCREENCAST] Building Data Prep from the GitHub source
- [VOICE-OVER] End-to-End Demo Video
- [SCREENCAST] Ingesting into Kudu
- [SCREENCAST] Realtime HL7 CCDA XML from Kafka into Time Parititioned Parquet
- [SCREENCAST] Parsing JSON file
- [SCREENCAST] Flattening arrays
- [SCREENCAST] Data cleansing with send-to-error directive
- [SCREENCAST] Publishing to Kafka
- [SCREENCAST] Fixed length to JSON
Recipes
Available Directives
These directives are currently available:
| Directive | Description |
|---|---|
| Parsers | |
| JSON Path | Uses a DSL (a JSON path expression) for parsing JSON records |
| Parse as AVRO | Parsing an AVRO encoded message - either as binary or json |
| Parse as AVRO File | Parsing an AVRO data file |
| Parse as CSV | Parsing an input record as comma-separated values |
| Parse as Date | Parsing dates using natural language processing |
| Parse as Excel | Parsing excel file. |
| Parse as Fixed Length | Parses as a fixed length record with specified widths |
| Parse as HL7 | Parsing Health Level 7 Version 2 (HL7 V2) messages |
| Parse as JSON | Parsing a JSON object |
| Parse as Log | Parses access log files as from Apache HTTPD and nginx servers |
| Parse as Protobuf | Parses an Protobuf encoded in-memory message using descriptor |
| Parse as Simple Date | Parses date strings |
| Parse XML To JSON | Parses an XML document into a JSON structure |
| Parse as Currency | Parses a string representation of currency into a number. |
| Output Formatters | |
| Write as CSV | Converts a record into CSV format |
| Write as JSON | Converts the record into a JSON map |
| Write JSON Object | Composes a JSON object based on the fields specified. |
| Format as Currency | Formats a number as currency as specified by locale. |
| Transformations | |
| Changing Case | Changes the case of column values |
| Cut Character | Selects parts of a string value |
| Set Column | Sets the column value to the result of an expression execution |
| Find and Replace | Transforms string column values using a "sed"-like expression |
| Index Split | (Deprecated) |
| Invoke HTTP | Invokes an HTTP Service (Experimental, potentially slow) |
| Quantization | Quantizes a column based on specified ranges |
| Regex Group Extractor | Extracts the data from a regex group into its own column |
| Setting Character Set | Sets the encoding and then converts the data to a UTF-8 String |
| Setting Record Delimiter | Sets the record delimiter |
| Split by Separator | Splits a column based on a separator into two columns |
| Split Email Address | Splits an email ID into an account and its domain |
| Split URL | Splits a URL into its constituents |
| Text Distance (Fuzzy String Match) | Measures the difference between two sequences of characters |
| Text Metric (Fuzzy String Match) | Measures the difference between two sequences of characters |
| URL Decode | Decodes from the application/x-www-form-urlencoded MIME format |
| URL Encode | Encodes to the application/x-www-form-urlencoded MIME format |
| Trim | Functions for trimming white spaces around string data |
| Encoders and Decoders | |
| Decode | Decodes a column value as one of base32, base64, or hex |
| Encode | Encodes a column value as one of base32, base64, or hex |
| Unique ID | |
| UUID Generation | Generates a universally unique identifier (UUID) |
| Date Transformations | |
| Diff Date | Calculates the difference between two dates |
| Format Date | Custom patterns for date-time formatting |
| Format Unix Timestamp | Formats a UNIX timestamp as a date |
| Lookups | |
| Catalog Lookup | Static catalog lookup of ICD-9, ICD-10-2016, ICD-10-2017 codes |
| Table Lookup | Performs lookups into Table datasets |
| Hashing & Masking | |
| Message Digest or Hash | Generates a message digest |
| Mask Number | Applies substitution masking on the column values |
| Mask Shuffle | Applies shuffle masking on the column values |
| Row Operations | |
| Filter Row if Matched | Filters rows that match a pattern for a column |
| Filter Row if True | Filters rows if the condition is true. |
| Filter Row Empty of Null | Filters |
Extension points exported contracts — how you extend this code
Executor (Interface)
A interface defining the wrangle Executor in the wrangling RecipePipeline. [78 implementers]
wrangler-api/src/main/java/co/cask/wrangler/api/Executor.java
Decoder (Interface)
Decoder interface for decoding the data. [4 implementers]
wrangler-core/src/main/java/co/cask/wrangler/codec/Decoder.java
Executor (Interface)
(no doc) [78 implementers]
wrangler-service/src/main/java/co/cask/wrangler/service/database/DatabaseService.java
Directive (Interface)
Directive is a user defined directive. DIE - Define, Initialize & Execute - Pattern Following is a simple example of ho [153 …
wrangler-api/src/main/java/co/cask/wrangler/api/Directive.java
Statistics (Interface)
Interface for calculating metrics over the records being processed. [2 implementers]
wrangler-core/src/main/java/co/cask/wrangler/statistics/Statistics.java
Request (Interface)
Dataprep request body and it's structure. [2 implementers]
wrangler-service/src/main/java/co/cask/wrangler/proto/Request.java
DirectiveRegistry (Interface)
A directive registry maintains a collection of directives either system provided or user provided. The directive inf [8 …
wrangler-api/src/main/java/co/cask/wrangler/api/DirectiveRegistry.java
ELRegistration (Interface)
Class description here. [2 implementers]
wrangler-core/src/main/java/co/cask/wrangler/expression/ELRegistration.java
Core symbols most depended-on inside this repo
get
called by 267
wrangler-api/src/main/java/co/cask/wrangler/api/TransientStore.java
size
called by 238
wrangler-api/src/main/java/co/cask/wrangler/api/Arguments.java
getValue
called by 167
wrangler-core/src/main/java/co/cask/directives/parser/JsParser.java
format
called by 162
wrangler-core/src/main/java/co/cask/functions/Global.java
getMessage
called by 137
wrangler-core/src/main/java/co/cask/wrangler/clients/Response.java
add
called by 117
wrangler-core/src/main/java/co/cask/wrangler/expression/ELContext.java
value
called by 91
wrangler-api/src/main/java/co/cask/wrangler/api/parser/Token.java
define
called by 82
wrangler-api/src/main/java/co/cask/wrangler/api/Directive.java
Shape
Languages
Java100%
Modules by API surface
wrangler-core/src/test/java/com/example/tutorial/AddressBookProtos.java229 symbols
wrangler-core/src/main/java/co/cask/functions/DataQuality.java27 symbols
wrangler-service/src/main/java/co/cask/wrangler/service/directive/DirectivesService.java26 symbols
wrangler-service/src/main/java/co/cask/wrangler/service/database/DatabaseService.java26 symbols
wrangler-core/src/main/java/co/cask/functions/Dates.java24 symbols
wrangler-core/src/main/java/co/cask/wrangler/utils/StructuredRecordJsonConverter.java22 symbols
wrangler-core/src/main/java/co/cask/directives/nlp/internal/PorterStemmer.java22 symbols
wrangler-service/src/test/java/co/cask/wrangler/service/database/DatabaseServiceTest.java20 symbols
wrangler-service/src/main/java/co/cask/wrangler/dataset/workspace/WorkspaceDataset.java20 symbols
wrangler-core/src/main/java/co/cask/wrangler/parser/RecipeVisitor.java20 symbols
wrangler-core/src/main/java/co/cask/wrangler/expression/EL.java20 symbols
wrangler-service/src/main/java/co/cask/wrangler/dataset/AbstractTableStore.java19 symbols
Dependencies from manifests, versioned
ca.uhn.hapi:hapi-base1×
ca.uhn.hapi:hapi-structures-v211×
ca.uhn.hapi:hapi-structures-v221×
ca.uhn.hapi:hapi-structures-v231×
ca.uhn.hapi:hapi-structures-v2311×
ca.uhn.hapi:hapi-structures-v241×
ca.uhn.hapi:hapi-structures-v251×
ca.uhn.hapi:hapi-structures-v2511×
ca.uhn.hapi:hapi-structures-v261×
co.cask.cdap:cdap-api1×
co.cask.cdap:cdap-etl-api1×
co.cask.cdap:cdap-etl-core1×
Datastores touched
(mysql)Database · 1 repos
For agents
$ claude mcp add wrangler \
-- python -m otcore.mcp_server <graph>