modsec_parser

Welcome to the modsec_parser documentation.

The parser runs under Python 3.6+ on Linux, Windows and Mac.

Installation

The parser relies on Ply as its underlying parsing library.

Therefore, to run it you will need:

• a Python 3 interpreter
• Ply - the Python Ley Yacc library
• YAML and/or JSON it you want your output to be either of those

Debian install

You can install these packages on Debian with this command:

sudo apt install python3-ply python3-yaml python3-ubjson

Installing using pip3

The module is published as a pip3 module.

Method 1 You can install it using:

pip3 install msc-pyparser>=1.2.0

That will install it system-wide.

Method 2 If you want to use a virtualenv:

pip3 install virtualenv

Then create the virtual environment and install dependencies:

cd msc_pyparser
mkdir ~/virtualenvs
virtualenv ~/virtualenvs/msc_pyparser
source ~/virtualenvs/msc_pyparser/bin/activate
python3 setup.py install

Method 3 Another option is to use pipenv that will give you isolation and dependency management:

pip3 install pipenv
pipenv install msc-pyparser

That will create the proper virtual environment and you can now switch to it using pipenv shell.

:point_right: Important notes after 1.0 :point_left:

After v1.0 the inside API has changed. The parser has extended with new capabilities, and the inside structure was aligned.

:point_right: Important notes after 1.2.0 :point_left:

After v1.2.0 the versioning structure has changed: from two digits (eg. 1.1) we have switched to three digits (1.2.0). The other important change is the lexer and parser exceptions contain extra information about the exception cause and position.

🎉 That's it!

Try to keep the module updated, because it is under heavy development now.

New features and changes in 1.0

• msc_pyparser can parse the whole ModSecurity config, not just the CRS rules

  • this means that you can pass the root configuration file to the parser, which contains the include /path/to/coreruleset/*.conf directive, and you will get the include directives and its arguments (path to files). So you can walk those files and you will get your whole config. This is the most important new feature.

  • please note, that the parser doesn't parses the files as recursively, you need to handle this directive

  • You can extend the parser (basically the scanner). If there is a token which missing (eg. you have any Apache directive in the config (see Comodo rules below), then you can add them. These tokens will handled as directives. Also you can extend the variables (see Comodo rules), operators, and actions.
  • the API has changed. the new structure stores the necessary data in much more detail.
  • there are four different structure, the previous versions had two:
    • Comment - it's simple, the comments
    • Directive - everythng which isn't comment, SecRule or SecAction
    • SecRule - the SecRule entity from config
    • SecAction - the SecAction entity from config
  • the structures stores more detailed info:
    • Directive stores the keyword, and the variable length argument list. Every item in list contains info about the quoted status of value
    • SecRule:
      • the variables expanded into more detailed structure: it contains the info about variable name, negated flag (!), counter flag (&), variable subpart and its quoted status
      • operator structure also has a new flag about its negated status
      • action structure got a new member: in the previous versions, for eg. the ctl actions last part (consider ctl:ruleRemoveTargetById=1234;ARGS:passwd - the ARGS:passed) now splitted into two member. Now the new keys are:
      • act_name (eg. ctl)
      • lineno (number of the line)
      • act_quote (arg. val is quoted - it isn't in the given example)
      • act_arg (eg. ruleRemoveTargetById)
      • act_arg_val (1234)
      • act_arg_val_param (ARGS)
      • act_arg_val_param_val (passwd)
    • SecAction - the action list is same as at SecRule
  • MSCWriter has two option argument: indentstr and indentchained - see below the details
  • in examples directory you don't need to create a loop to run for list of files anymore; just create the output directory, and run the script: mkdir crsmod; ./example3_addtag.py crsyaml/*.yaml crsmod but of course, it works for a single file too: ./example3_addtag.py crsyaml/specific_ruleset.yml mod_specific_ruleset.yml

New features and changes in 1.2.0

• msc_pyparser has a new version format with three digits 1.2.0:
• the exceptions of MSCLexer and MSCParser modules have an extra argument in their exceptions. It's a dictionary:

{
  'cause': <class 'str'> one of the "lexer" or "parser",
  'line': <class 'int'>,
  'column': <class 'int'>,
  'position': <class 'int'>
}

Module Contents

modsec_parser contains these classes:

• MSCLexer
• MSCParser
• MSCWriter
• MSCUtils

Module version

Before you start to work with msc_pyparser, please check the version to make sure you have the current one (1.1):

$ python3
...
>>> import msc_pyparser
>>> print(msc_pyparser.__version__)
1.2.0
>>>

MSCLexer

The MSCLexer class is a wrapper for Ply's lexer object. You can use it independently, to check and see what tokens are in your ModSecurity ruleset.

Here is a simple example:

$ python3
Python 3.8.5 (default, Aug  2 2020, 15:09:07)
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import msc_pyparser
>>> rule = """SecRule TX:EXECUTING_PARANOIA_LEVEL "@lt 1" "id:920011,phase:1,pass,nolog,skipAfter:END-REQUEST-920-PROTOCOL-ENFORCEMENT" """
>>> mlexer = msc_pyparser.MSCLexer()
>>> mlexer.lexer.input(rule)
>>> while True:
...     tok = mlexer.lexer.token()
...     if not tok:
...         break
...     print(tok)
...
LexToken(T_CONFIG_DIRECTIVE_SECRULE,'SecRule',1,0)
LexToken(T_SECRULE_VARIABLE,'TX',1,8)
LexToken(T_SECRULE_VARIABLE_PART,'EXECUTING_PARANOIA_LEVEL',1,10)
LexToken(T_SECRULE_OPERATOR_QUOTE_MARK,'"',1,36)
LexToken(T_SECRULE_OPERATOR,'@lt',1,37)
LexToken(T_SECRULE_OPERATOR_ARGUMENT,'1',1,41)
LexToken(T_SECRULE_OPERATOR_QUOTE_MARK,'"',1,42)
LexToken(T_SECRULE_ACTION_QUOTE_MARK,'"',1,44)
LexToken(T_SECRULE_ACTION,'id',1,45)
LexToken(T_SECRULE_ACTION_COLON,':',1,47)
LexToken(T_SECRULE_ACTION_ARGUMENT,'920011',1,48)
LexToken(T_SECRULE_ACTION_SEPARATOR,',',1,54)
LexToken(T_SECRULE_ACTION,'phase',1,55)
LexToken(T_SECRULE_ACTION_COLON,':',1,60)
LexToken(T_SECRULE_ACTION_ARGUMENT,'1',1,61)
LexToken(T_SECRULE_ACTION_SEPARATOR,',',1,62)
LexToken(T_SECRULE_ACTION,'pass',1,63)
LexToken(T_SECRULE_ACTION_SEPARATOR,',',1,67)
LexToken(T_SECRULE_ACTION,'nolog',1,68)
LexToken(T_SECRULE_ACTION_SEPARATOR,',',1,73)
LexToken(T_SECRULE_ACTION,'skipAfter',1,74)
LexToken(T_SECRULE_ACTION_COLON,':',1,83)
LexToken(T_SECRULE_ACTION_ARGUMENT,'END-REQUEST-920-PROTOCOL-ENFORCEMENT',1,84)
LexToken(T_SECRULE_ACTION_QUOTE_MARK,'"',1,120)

Note: the token list has changed in version 1.0.

Now see the exception:

>>> import msc_pyparser
>>>
>>> rule = """\nSecRule ARGS1 "@rx foo" "phase:1,id:1,block" """
>>> mlexer = msc_pyparser.MSCLexer()
>>> mlexer.lexer.input(rule)
>>> while True:
...     try:
...         tok = mlexer.lexer.token()
...         if not tok:
...             break
...         print(tok)
...     except Exception as e:
...         print(e.args[0])
...         print(e.args[1])
...         break
...
LexToken(T_CONFIG_DIRECTIVE_SECRULE,'SecRule',2,1)
Lexer error: illegal token found in line 2 at pos 14, column 13
SecRule ARGS1 "@rx foo" "phase:1,id:1,block"
~~~~~~~~~~~~~^
{'cause': 'lexer', 'line': 2, 'position': 14, 'column': 13}

Please note, that the given target does not exist.

For a more detailed example, see test_lexer.py in the examples directory.

MSCParser

The MSCParser class is a wrapper for Ply's parser object. The parser object needs a lexer, but MSCParser invokes MSCLexer and sets it up.

Here is a simple example:

$ python3
Python 3.8.5 (default, Aug  2 2020, 15:09:07) 
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import msc_pyparser
>>> rule = """SecRule TX:EXECUTING_PARANOIA_LEVEL "@lt 1" "id:920011,phase:1,pass,nolog,skipAfter:END-REQUEST-920-PROTOCOL-ENFORCEMENT" """
>>> mparser = msc_pyparser.MSCParser()
Generating LALR tables
>>> mparser.parser.parse(rule, debug = True)
PLY: PARSE DEBUG START

State  : 0
Stack  : . LexToken(T_CONFIG_DIRECTIVE_SECRULE,'SecRule',1,0)
Action : Shift and goto state 18


python
$ python3
Python 3.7.4 (default, Jul 11 2019, 10:43:21)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import msc_pyparser
>>> rule = """SecRule TX:EXECUTING_PARANOIA_LEVEL "@lt 1" "id:920011,phase:1,pass,nolog,skipAfter:END-REQUEST-920-PROTOCOL-ENFORCEMENT" """
>>> mparser = msc_pyparser.MSCParser()
>>> mparser.parser.parse(rule, debug = True)
PLY: PARSE DEBUG START

State  : 0
Stack  : . LexToken(CONFDIR_SECRULE,'SecRule',1,0)
Action : Shift and goto state 13
...
PLY: PARSE DEBUG END
>>> print(mparser.configlines)
[{'type': 'SecRule', 'lineno': 1, 'variables': [{'variable': 'TX', 'variable_part': 'EXECUTING_PARANOIA_LEVEL', 'quote_type': 'no_quote', 'negated': False, 'counter': False}], 'operator': '@lt', 'operator_argument': '1', 'actions': [{'act_name': 'id', 'lineno': 1, 'act_quote': 'no_quote', 'act_arg': '920011', 'act_arg_val': '', 'act_arg_val_param': '', 'act_arg_val_param_val': ''}, {'act_name': 'phase', 'lineno': 1, 'act_quote': 'no_quote', 'act_arg': '1', 'act_arg_val': '', 'act_arg_val_param': '', 'act_arg_val_param_val': ''}, {'act_name': 'pass', 'lineno': 1, 'act_quote': 'no_quote', 'act_arg': '', 'act_arg_val': '', 'act_arg_val_param': '', 'act_arg_val_param_val': ''}, {'act_name': 'nolog', 'lineno': 1, 'act_quote': 'no_quote', 'act_arg': '', 'act_arg_val': '', 'act_arg_val_param': '', 'act_arg_val_param_val': ''}, {'act_name': 'skipAfter', 'lineno': 1, 'act_quote': 'no_quote', 'act_arg': 'END-REQUEST-920-PROTOCOL-ENFORCEMENT', 'act_arg_val': '', 'act_arg_val_param': '', 'act_arg_val_param_val': ''}], 'chained': False, 'operator_negated': False, 'oplineno': 1}]

Note: the list of grammar rules has changed in version 1.0.

Now let's see an example for the exception:

>>> import msc_pyparser
>>> rule = """\nSecRule ARGS "@rx foo "phase:1,id:1,block" """
>>> mparser = msc_pyparser.MSCParser()
>>> try:
...     mparser.parser.parse(rule, debug = False)
... except Exception as e:
...     print(e.args[0])
...     print(e.args[1])
...
Parser error: syntax error in line 2 at pos 43, column 42
SecRule ARGS "@rx foo "phase:1,id:1,block"
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
{'cause': 'parser', 'line': 2, 'position': 43, 'column': 42}

For a detailed example, see test_parser.py program in the examples directory.

MSCWriter

This class transforms the inside structure to the string. You can save the result to a file. This class converts YAML, JSON, etc, to a config file. See the example file crs_writer.py for how it works.

Here is a simple example:

$ python3
Python 3.8.5 (default, Aug  2 2020, 15:09:07) 
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import msc_pyparser
>>> data = [{'type': 'SecRule', 'lineno': 1, 'variables': [{'variable': 'TX', 'variable_part': 'EXECUTING_PARANOIA_LEVEL', 'quote_type': 'no_quote', 'negated': False, 'counter': False}], 'operator': '@lt', 'operator_argument': '1', 'actions': [{'act_name': 'id', 'lineno': 1, 'act_quote': 'no_quote', 'act_arg': '920011', 'act_arg_val': '', 'act_arg_val_param': '', 'act_arg_val_param_val': ''}, {'act_name': 'phase', 'lineno': 1, 'act_quote': 'no_quote', 'act_arg': '1', 'act_arg_val': '', 'act_arg_val_param': '', 'act_arg_val_param_val': ''}, {'act_name': 'pass', 'lineno': 1, 'act_quote': 'no_quote', 'act_arg': '', 'act_arg_val': '', 'act_arg_val_param': '', 'act_arg_val_param_val': ''}, {'act_name': 'nolog', 'lineno': 1, 'act_quote': 'no_quote', 'act_arg': '', 'act_arg_val': '', 'act_arg_val_param': '', 'act_arg_val_param_val': ''}, {'act_name': 'skipAfter', 'lineno': 1, 'act_quote': 'no_quote', 'act_arg': 'END-REQUEST-920-PROTOCOL-ENFORCEMENT', 'act_arg_val': '', 'act_arg_val_param': '', 'act_arg_val_param_val': ''}], 'chained': False, 'operator_negated': False, 'oplineno': 1}]
>>> mwriter = msc_pyparser.MSCWriter(data)
>>> mwriter.generate()
>>> print(mwriter.output)
['SecRule TX:EXECUTING_PARANOIA_LEVEL "@lt 1" "id:920011,phase:1,pass,nolog,skipAfter:END-REQUEST-920-PROTOCOL-ENFORCEMENT"']

Note: the writer class has aligned to the modified structure in version 1.0.

MSCUtils

This class contains IO helper functions (currently only one function).

Inside of structure

The MSCParser class reads the ModSecurity rulesets, and transforms them into a Python list. Every item in this list is a dictionary. Every dictionary item has the keys type and lineno. Depending on the type there might be additional keys.

These are the supported types:

• Comment
• Directive
• SecRule
• SecAction

Note: the types has changed in version 1.0.

There are four types of dictionary objects for types above:

```python { 'type': "Comment", 'argument': , 'quoted':