中文版

This is a golang implemented Redis RDB parser for secondary development and memory analysis.

It provides abilities to:

• Generate memory report for rdb file
• Convert RDB files to JSON
• Convert RDB files to Redis Serialization Protocol (or AOF file)
• Find the biggest N keys in RDB files
• Find the hottest N keys by LFU frequency in RDB files
• Draw FlameGraph to analysis which kind of keys occupied most memory
• Customize data usage
• Generate RDB file

Support RDB version: 1 <= version <= 12(Redis 7.2)

If you read Chinese, you could find a thorough introduction to the RDB file format here: Golang 实现 Redis(11): RDB 文件格式

Thanks sripathikrishnan for his redis-rdb-tools

Install

If you have installed go on your compute, just simply use:

go install github.com/hdt3213/rdb@latest

Package Managers

If you're a Homebrew user, you can install rdb via:

$ brew install rdb

Or, you can download executable binary file from releases and put its path to PATH environment.

use rdb command in terminal, you can see it's manual

$ rdb
This is a tool to parse Redis' RDB files
Options:
  -c command, including: json/memory/aof/bigkey/hotkey/prefix/flamegraph
  -o output file path
  -n number of result, using in command: bigkey/hotkey/prefix
  -port listen port for flame graph web service
  -sep separator for flamegraph, rdb will separate key by it, default value is ":". 
    supporting multi separators: -sep sep1 -sep sep2 
  -prefix-sep separator for prefix analysis (flat-map mode, constant memory).
    when specified, uses separator-based analysis instead of radix tree.
    supporting multi separators: -prefix-sep sep1 -prefix-sep sep2
  -regex using regex expression filter keys
  -expire filter keys by its expiration time
    1. '1751731200~1751817600' get keys with expiration time in range [1751731200, 1751817600]
    2. '1751731200~now' 'now~1751731200' magic variable 'now' represents the current timestamp
    3. '1751731200~inf' 'now~inf' magic variable 'inf' represents the Infinity
    4. 'noexpire' get keys without expiration time
    5. 'anyexpire' get all keys with expiration time
  -size filter keys by size, supports B/KB/MB/GB/TB/PB/EB
    1. '1KB~1MB' get keys with size in range [1KB, 1MB]
    2. '10MB~inf' magic variable 'inf' represents the Infinity
    3. '1024~10KB' get keys with size in range [0Bytes, 10KB]
  -concurrent The number of concurrent json converters. 4 by default.
  -show-global-meta Show global meta likes redis-verion/ctime/functions
  -no-expired filter expired keys(deprecated, please use 'expire' option)

Examples:
parameters between '[' and ']' is optional
1. convert rdb to json
  rdb -c json -o dump.json dump.rdb
2. generate memory report
  rdb -c memory -o memory.csv dump.rdb
3. convert to aof file
  rdb -c aof -o dump.aof dump.rdb
4. get largest keys
  rdb -c bigkey [-o dump.aof] [-n 10] dump.rdb
5. get number and memory size by prefix
  rdb -c prefix [-n 10] [-max-depth 3] [-o prefix-report.csv] dump.rdb
6. get number and memory size by prefix with separator (constant memory)
  rdb -c prefix [-n 10] [-max-depth 3] -prefix-sep : [-o prefix-report.csv] dump.rdb
7. draw flamegraph
  rdb -c flamegraph [-port 16379] [-sep :] dump.rdb
7. get hottest keys by LFU frequency (requires maxmemory-policy allkeys-lfu/volatile-lfu)
  rdb -c hotkey [-o hotkey.csv] [-n 50] dump.rdb

Convert to Json

Usage:

rdb -c json -o <output_path> <source_path>

example:

rdb -c json -o intset_16.json cases/intset_16.rdb

You can get some rdb examples in cases

The examples for json result:

[
    {"db":0,"key":"hash","size":64,"type":"hash","hash":{"ca32mbn2k3tp41iu":"ca32mbn2k3tp41iu","mddbhxnzsbklyp8c":"mddbhxnzsbklyp8c"}},
    {"db":0,"key":"string","size":10,"type":"string","value":"aaaaaaa"},
    {"db":0,"key":"expiration","expiration":"2022-02-18T06:15:29.18+08:00","size":8,"type":"string","value":"zxcvb"},
    {"db":0,"key":"list","expiration":"2022-02-18T06:15:29.18+08:00","size":66,"type":"list","values":["7fbn7xhcnu","lmproj6c2e","e5lom29act","yy3ux925do"]},
    {"db":0,"key":"zset","expiration":"2022-02-18T06:15:29.18+08:00","size":57,"type":"zset","entries":[{"member":"zn4ejjo4ths63irg","score":1},{"member":"1ik4jifkg6olxf5n","score":2}]},
    {"db":0,"key":"set","expiration":"2022-02-18T06:15:29.18+08:00","size":39,"type":"set","members":["2hzm5rnmkmwb3zqd","tdje6bk22c6ddlrw"]}
]

You can use -concurrent to change the number of concurrent convertes. The default value is 4.

rdb -c json -o intset_16.json -concurrent 8 cases/intset_16.rdb

You can use -show-global-meta to get metadata (redis-ver,ctime,used-mem, etc.) and functions in rdb file.

rdb -c json -o function.json -show-global-meta cases/function.rdb

Examples:

[
{"db":0,"key":"redis-ver","size":0,"type":"aux","encoding":"","value":"7.2.5"},
{"db":0,"key":"redis-bits","size":0,"type":"aux","encoding":"","value":"64"},
{"db":0,"key":"ctime","size":0,"type":"aux","encoding":"","value":"1767107423"},
{"db":0,"key":"used-mem","size":0,"type":"aux","encoding":"","value":"1269264"},
{"db":0,"key":"aof-base","size":0,"type":"aux","encoding":"","value":"0"},
{"db":0,"key":"functions","size":0,"type":"functions","encoding":"functions","functionsLua":"#!lua name=mylib\nredis.register_function('myfunc', function(keys, args) return 'hello' end)"}
]

Json Fromat Detail

Common Fields

All objects contain these base fields:

• db: database index
• key: key name
• size: estimated memory size in bytes
• type: redis type (string/list/set/hash/zset/stream/aux/functions)
• expiration (optional): expiration time in RFC3339 format, omitted if key has no expiration
• encoding (optional): the encoding used by redis internally
• lru (optional): LRU idle time in seconds. Present when Redis uses the maxmemory-policy allkeys-lru or volatile-lru eviction policy
• lfu (optional): LFU frequency counter (0-255). Present when Redis uses the maxmemory-policy allkeys-lfu or volatile-lfu eviction policy

Example with LRU:

{
    "db": 0,
    "key": "mykey",
    "size": 56,
    "type": "string",
    "lru": 3600,
    "value": "hello"
}

Example with LFU:

{
    "db": 0,
    "key": "mykey",
    "size": 56,
    "type": "string",
    "lfu": 128,
    "value": "hello"
}

string

{
    "db": 0,
    "key": "string",
    "size": 10, // estimated memory size
    "type": "string",
    "expiration":"2022-02-18T06:15:29.18+08:00",
    "value": "aaaaaaa"
}

list

{
    "db": 0,
    "key": "list",
    "expiration": "2022-02-18T06:15:29.18+08:00",
    "size": 66,
    "type": "list",
    "values": [
        "7fbn7xhcnu",
        "lmproj6c2e",
        "e5lom29act",
        "yy3ux925do"
    ]
}

set

{
    "db": 0,
    "key": "set",
    "expiration": "2022-02-18T06:15:29.18+08:00",
    "size": 39,
    "type": "set",
    "members": [
        "2hzm5rnmkmwb3zqd",
        "tdje6bk22c6ddlrw"
    ]
}

hash

{
    "db": 0,
    "key": "hash",
    "size": 64,
    "type": "hash",
    "expiration": "2022-02-18T06:15:29.18+08:00",
    "hash": {
        "ca32mbn2k3tp41iu": "ca32mbn2k3tp41iu",
        "mddbhxnzsbklyp8c": "mddbhxnzsbklyp8c"
    }
}

zset

{
    "db": 0,
    "key": "zset",
    "expiration": "2022-02-18T06:15:29.18+08:00",
    "size": 57,
    "type": "zset",
    "entries": [
        {
            "member": "zn4ejjo4ths63irg",
            "score": 1
        },
        {
            "member": "1ik4jifkg6olxf5n",
            "score": 2
        }
    ]
}

stream

{
    "db": 0,
    "key": "mystream",
    "size": 1776,
    "type": "stream",
    "encoding": "",
    "version": 3, // Version 2 means is RDB_TYPE_STREAM_LISTPACKS_2, 3 means is RDB_TYPE_STREAM_LISTPACKS_3
    // StreamEntry is a node in the underlying radix tree of redis stream, of type listpacks, which contains several messages. There is no need to care about which entry the message belongs to when using it.
    "entries": [ 
        {
            "firstMsgId": "1704557973866-0", // ID of the master entry at listpack head 
            "fields": [ // master fields, used for compressing size
                "name",
                "surname"
            ],
            "msgs": [ // messages in entry
                {
                    "id": "1704557973866-0",
                    "fields": {
                        "name": "Sara",
                        "surname": "OConnor"
                    },
                    "deleted": false
                }
            ]
        }
    ],
    "groups": [ // consumer groups
        {
            "name": "consumer-group-name",
            "lastId": "1704557973866-0",
            "pending": [ // pending messages
                {
                    "id": "1704557973866-0",
                    "deliveryTime": 1704557998397,
                    "deliveryCount": 1
                }
            ],
            "consumers": [ // consumers in the group
                {
                    "name": "consumer-name",
                    "seenTime": 1704557998397,
                    "pending": [
                        "1704557973866-0"
                    ],
                    "activeTime": 1704557998397
                }
            ],
            "entriesRead": 1
        }
    ],
    "len": 1, // current number of messages inside this stream
    "lastId": "1704557973866-0",
    "firstId": "1704557973866-0",
    "maxDeletedId": "0-0",
    "addedEntriesCount": 1
}

aux

[
    {
        "db": 0,
        "key": "redis-ver",
        "size": 0,
        "type": "aux",
        "encoding": "",
        "value": "7.2.5"
    },
    {
        "db": 0,
        "key": "redis-bits",
        "size": 0,
        "type": "aux",
        "encoding": "",
        "value": "64"
    },
    {
        "db": 0,
        "key": "ctime",
        "size": 0,
        "type": "aux",
        "encoding": "",
        "value": "1767107423"
    },
    {
        "db": 0,
        "key": "used-mem",
        "size": 0,
        "type": "aux",
        "encoding": "",
        "value": "1269264"
    },
    {
        "db": 0,
        "key": "aof-base",
        "size": 0,
        "type": "aux",
        "encoding": "",
        "value": "0"
    }
]

functions

{
    "db": 0,
    "key": "functions",
    "size": 0,
    "type": "functions",
    "encoding": "functions",
    "functionsLua": "#!lua name=mylib\nredis.register_function('myfunc', function(keys, args) return 'hello' end)"
}

Generate Memory Report

RDB uses rdb encoded size to estimate redis memory usage.

rdb -c memory -o <output_path> <source_path>

Example:

rdb -c memory -o mem.csv cases/memory.rdb

The examples for csv result:

database,key,type,size,size_readable,element_count
0,hash,hash,64,64B,2
0,s,string,10,10B,0
0,e,string,8,8B,0
0,list,list,66,66B,4
0,zset,zset,57,57B,2
0,large,string,2056,2K,0
0,set,set,39,39B,2

Analyze By Prefix

If you can distinguish modules based on the prefix of the key, for example, the key of user data is User:<uid>, the key of Post is Post:<postid>, the user statistics is Stat:User:???, and the statistics of Post is Stat:Post:???.Then we can get the status of each module through prefix analysis:

database,prefix,size,size_readable,key_count
0,Post:,1170456184,1.1G,701821
0,Stat:,405483812,386.7M,3759832
0,Stat:Post:,291081520,277.6M,2775043
0,User:,241572272,230.4M,265810
0,Topic:,171146778,163.2M,694498
0,Topic:Post:,163635096,156.1M,693758
0,Stat:Post:View,133201208,127M,1387516
0,Stat:User:,114395916,109.1M,984724
0,Stat:Post:Comment:,80178504,76.5M,693758
0,Stat:Post:Like:,77701688,74.1M,693768

Format:

rdb -c prefix [-n <top-n>] [-max-depth <max-depth>] -o <output_path> <source_path>

• The prefix analysis results are arranged in descending order of memory space. The -n option can specify the number of outputs. All are output by default.

• -max-depth can limit the maximum depth of the prefix tree. In the above example, the depth of Stat: is 1, and the depth of Stat:User: and Stat:Post: is 2.

Example:

rdb -c prefix -n 10 -max-depth 2 -o prefix.csv cases/memory.rdb

Separator-based Prefix Analysis (Constant Memory)

When you specify -prefix-sep, RDB uses a flat map instead of a radix tree for prefix analysis. This mode has constant memory usage regardless of the number of keys, making it suitable for very large RDB files.

You need to specify the separator(s) used in your key naming convention. Multiple separators are supported and will be normalized to the first one.

rdb -c prefix -prefix-sep : -n 10 -max-depth 2 -o prefix.csv dump.rdb

With multiple separators (e.g