Event Ruler

Event Ruler (called Ruler in rest of the doc for brevity) is a Java library that allows matching Rules to Events. An event is a list of fields, which may be given as name/value pairs or as a JSON object. A rule associates event field names with lists of possible values. There are two reasons to use Ruler:

1. It's fast; the time it takes to match Events doesn't depend on the number of Rules.
2. Customers like the JSON "query language" for expressing rules.

Contents:

1. Ruler by Example
2. And and Or With Ruler
3. How to Use Ruler
4. JSON Text Matching
5. JSON Array Matching
6. Compiling and Checking Rules
7. Performance

It's easiest to explain by example.

Ruler by Example

An Event is a JSON object. Here's an example:

{
  "version": "0",
  "id": "ddddd4-aaaa-7777-4444-345dd43cc333",
  "detail-type": "EC2 Instance State-change Notification",
  "source": "aws.ec2",
  "account": "012345679012",
  "time": "2017-10-02T16:24:49Z",
  "region": "us-east-1",
  "resources": [
    "arn:aws:ec2:us-east-1:123456789012:instance/i-000000aaaaaa00000"
  ],
  "detail": {
    "c-count": 5,
    "d-count": 3,
    "x-limit": 301.8,
    "source-ip": "10.0.0.33",
    "instance-id": "i-000000aaaaaa00000",
    "state": "running"
  }
}

You can also see this as a set of name/value pairs. For brevity, we present only a sampling. Ruler has APIs for providing events both in JSON form and as name/value pairs:

    +--------------+------------------------------------------+
    | name         | value                                    |
    |--------------|------------------------------------------|
    | source       | "aws.ec2"                                |
    | detail-type  | "EC2 Instance State-change Notification" |
    | detail.state | "running"                                |
    +--------------+------------------------------------------+

Events in the JSON form may be provided in the form of a raw JSON String, or a parsed Jackson JsonNode.

Simple matching

The rules in this section all match the sample event above:

{
  "detail-type": [ "EC2 Instance State-change Notification" ],
  "resources": [ "arn:aws:ec2:us-east-1:123456789012:instance/i-000000aaaaaa00000" ],
  "detail": {
    "state": [ "initializing", "running" ]
  }
}

This will match any event with the provided values for the resource, detail-type, and detail.state values, ignoring any other fields in the event. It would also match if the value of detail.state had been "initializing".

Values in rules are always provided as arrays, and match if the value in the event is one of the values provided in the array. The reference to resources shows that if the value in the event is also an array, the rule matches if the intersection between the event array and rule-array is non-empty.

Prefix matching

{
  "time": [ { "prefix": "2017-10-02" } ]
}

Prefix matches only work on string-valued fields.

Prefix equals-ignore-case matching

{
  "source": [ { "prefix": { "equals-ignore-case": "EC2" } } ]
}

Prefix equals-ignore-case matches only work on string-valued fields.

Suffix matching

{
  "source": [ { "suffix": "ec2" } ]
}

Suffix matches only work on string-valued fields.

Suffix equals-ignore-case matching

javascript { "source": [ { "suffix": { "equals-ignore-case": "EC2" } } ] } Suffix equals-ignore-case matches only work on string-valued fields.

Equals-ignore-case matching

{
  "source": [ { "equals-ignore-case": "EC2" } ]
}

Equals-ignore-case matches only work on string-valued fields.

Wildcard matching

{
  "source": [ { "wildcard": "Simple*Service" } ]
}

Wildcard matches only work on string-valued fields. A single value can contain zero to many wildcard characters, but consecutive wildcard characters are not allowed. To match the asterisk character specifically, a wildcard character can be escaped with a backslash. Two consecutive backslashes (i.e. a backslash escaped with a backslash) represents the actual backslash character. A backslash escaping any character other than asterisk or backslash is not allowed.

Anything-but matching

Anything-but matching does what the name says: matches anything except what's provided in the rule.

Anything-but works with single string and numeric values or lists, which have to contain entirely strings or entirely numerics. It also may be applied to a prefix, suffix, or equals-ignore-case match of a string or a list of strings.

Single anything-but (string, then numeric):

{
  "detail": {
    "state": [ { "anything-but": "initializing" } ]
  }
}

{
  "detail": {
    "x-limit": [ { "anything-but": 123 } ]
  }
}

Anything-but list (strings):

{
  "detail": {
    "state": [ { "anything-but": [ "stopped", "overloaded" ] } ]
  }
}

Anything-but list (numbers):

{
  "detail": {
    "x-limit": [ { "anything-but": [ 100, 200, 300 ] } ]
  }
}

Anything-but prefix:

{
  "detail": {
    "state": [ { "anything-but": { "prefix": "init" } } ]
  }
}

Anything-but prefix list (strings):

{
  "detail": {
    "state": [ { "anything-but": { "prefix": [ "init", "error" ] } } ]
  }
}

Anything-but suffix:

{
  "detail": {
    "instance-id": [ { "anything-but": { "suffix": "1234" } } ]
  }
}

Anything-but suffix list (strings):

{
  "detail": {
    "instance-id": [ { "anything-but": { "suffix": [ "1234", "6789" ] } } ]
  }
}

Anything-but-ignore-case:

{
  "detail": {
    "state": [ { "anything-but": {"equals-ignore-case": "Stopped" } } ]
  }
}

Anything-but-ignore-case list (strings):

{
  "detail": {
    "state": [ { "anything-but": {"equals-ignore-case": [ "Stopped", "OverLoaded" ] } } ]
  }
}

Anything-but wildcard:

{
  "detail": {
    "state": [ { "anything-but": { "wildcard": "*/bin/*.jar" } } ]
  }
}

Anything-but wildcard list (strings):

{
  "detail": {
    "state": [ { "anything-but": { "wildcard": [ "*/bin/*.jar", "*/bin/*.class" ] } } ]
  }
}

Numeric matching

{
  "detail": {
    "c-count": [ { "numeric": [ ">", 0, "<=", 5 ] } ],
    "d-count": [ { "numeric": [ "<", 10 ] } ],
    "x-limit": [ { "numeric": [ "=", 3.018e2 ] } ]
  }
}  

Above, the references to c-count, d-count, and x-limit illustrate numeric matching, and only work with values that are JSON numbers. Numeric matching supports the same precision and range as Java's double primitive which implements IEEE 754 binary64 standard.

IP Address Matching

{
  "detail": {
    "source-ip": [ { "cidr": "10.0.0.0/24" } ]
  }
}

This also works with IPv6 addresses.

Exists matching

Exists matching works on the presence or absence of a field in the JSON event.

The rule below will match any event which has a detail.c-count field present.

{
  "detail": {
    "c-count": [ { "exists": true  } ]
  }
}  

The rule below will match any event which has no detail.c-count field.

{
  "detail": {
    "c-count": [ { "exists": false  } ]
  }
}  

Note Exists match only works on the leaf nodes. It does not work on intermediate nodes.

As an example, the above example for exists : false would match the event below:

{
  "detail-type": [ "EC2 Instance State-change Notification" ],
  "resources": [ "arn:aws:ec2:us-east-1:123456789012:instance/i-000000aaaaaa00000" ],
  "detail": {
    "state": [ "initializing", "running" ]
  }
}

but would also match the event below because c-count is not a leaf node:

{
  "detail-type": [ "EC2 Instance State-change Notification" ],
  "resources": [ "arn:aws:ec2:us-east-1:123456789012:instance/i-000000aaaaaa00000" ],
  "detail": {
    "state": [ "initializing", "running" ]
    "c-count" : {
       "c1" : 100
    }
  }
}

Complex example

{
  "time": [ { "prefix": "2017-10-02" } ],
  "detail": {
    "state": [ { "anything-but": "initializing" } ],
    "c-count": [ { "numeric": [ ">", 0, "<=", 5 ] } ],
    "d-count": [ { "numeric": [ "<", 10 ] } ],
    "x-limit": [ { "anything-but": [ 100, 200, 300 ] } ],
    "source-ip": [ { "cidr": "10.0.0.0/8" } ]
  }
}

And and Or Relationship among fields with Ruler

Default "And" relationship

As the examples above show, Ruler considers a rule to match if all of the fields named in the rule match, and it considers a field to match if any of the provided field values match, that is to say Ruler has applied "And" logic to all fields by default without "And" primitive is required.

"Or" relationship

There are two ways to reach the "Or" effects: * Add multiple rules with the same rule name and each individual rule will be treated as one of "Or" condition by Ruler. Refer to below under addRule() section on how to achieve an "Or" effect in that way. * Use the "$or" primitive to express the "Or" relationship within the rule.

The "$or" Primitive

The "$or" primitive to allow the customer directly describe the "Or" relationship among fields in the rule.

Ruler recognizes "Or" relationship only when the rule has met all below conditions: * There is "$or" on field attribute in the rule followed with an array – e.g. "$or": [] * There are 2+ objects in the "$or" array at least : "$or": [{}, {}] * There is no filed name using Ruler keywords in Object of "$or" array, refer to RESERVED_FIELD_NAMES_IN_OR_RELATIONSHIP in /src/main/software/amazon/event/ruler/Constants.java#L38 for example, below rule will be not parsed as "Or" relationship because "numeric" and "prefix" are Ruler reserved keywords. { "$or": [ {"numeric" : 123}, {"prefix": "abc"} ] } Otherwise, Ruler just treats the "$or" as normal filed name the same as other string in the rule.

Rule examples with "$or" Primitive

Normal "Or":

// Effect of "source" && ("metricName" || "namespace")
{
  "source": [ "aws.cloudwatch" ], 
  "$or": [
    { "metricName": [ "CPUUtilization", "ReadLatency" ] },
    { "namespace": [ "AWS/EC2", "AWS/ES" ] }
  ] 
}

Parallel "Or":

// Effect of ("metricName" || "namespace") && ("detail.source" || "detail.detail-type")
{
  "$or": [
    { "metricName": [ "CPUUtilization", "ReadLatency" ] },
    { "namespace": [ "AWS/EC2", "AWS/ES" ] }
  ], 
  "detail" : {
    "$or": [
      { "source": [ "aws.cloudwatch" ] },
      { "detail-type": [ "CloudWatch Alarm State Change"] }
    ]
  }
}

"Or" has an "And" inside

// Effect of ("source" && ("metricName" || ("metricType && "namespace") || "scope"))
{
  "source": [ "aws.cloudwatch" ],
  "$or": [
    { "metricName": [ "CPUUtilization", "ReadLatency" ] },
    {
      "metricType": [ "MetricType" ] ,
      "namespace": [ "AWS/EC2", "AWS/ES" ]
    },
    { "scope": [ "Service" ] }
  ]
}

Nested "Or" and "And"

// Effect of ("source" && ("metricName" || ("metricType && "namespace" && ("metricId" || "spaceId")) || "scope"))
{
  "source": [ "aws.cloudwatch" ],
  "$or": [
    { "metricName": [ "CPUUtilization", "ReadLatency" ] },
    {
      "metricType": [ "MetricType" ] ,
      "namespace": [ "AWS/EC2", "AWS/ES" ],
      "$or" : [
        { "metricId": [ 1234 ] },
        { "spaceId": [ 1000 ] }
      ]
    },
    { "scope": [ "Service" ] }
  ]
}

The backward compatibility of using "$or" as field name in the rule

"$or" is possibly already used as a normal key in some applications (though its likely rare). For these cases, Ruler tries its best to maintain the backward compatibility. Only when the 3 conditions mentioned above, will ruler change behaviour because it assumes your rule really wanted an OR and was mis-configured until today. For example, the rule below will keep working as normal rule with treating "$or" as normal field name in the rule and event:

{
    "source": [ "aws.cloudwatch" ],
    "$or": {
        "metricType": [ "MetricType" ] , 
        "namespace": [ "AWS/EC2", "AWS/ES" ]
    }
}

Refer to /src/test/data/normalRulesWithOrWording.json for more examples that "$or" is parsed as normal field name by Ruler.

Caveat

The keyword "$or" as "Or" relationship primitive should not be designed as normal field in both Events and Rules. Ruler supports the legacy rules where "$or" is parsed as normal field name to keep backward compatibility and give time for team to migrate their legacy "$or" usage away from their events and rules as normal filed name. Mix usage of "$or" as "Or" primitive, and "$or" as normal field name is not supported intentionally by Ruler to avoid the super awkward ambiguities on "$or" from occurring.

How to use Ruler

There are two ways to use Ruler. You can compile multiple rules into a "Machine", and then use either of its `rule