This example shows how to use DataFusion's `PruningPredicate` to prove filter expressions can never be true based on statistics such as min/max values of columns. The process is called "pruning" and is commonly used in query engines to quickly eliminate entire files / partitions / row groups of data from consideration using statistical information from a catalog or other metadata. This example
()
| 44 | /// `parquet_index.rs` for an example that uses pruning in the context of an |
| 45 | /// individual query. |
| 46 | pub async fn pruning() -> Result<()> { |
| 47 | // In this example, we'll use the PruningPredicate to determine if |
| 48 | // the expression `x = 5 AND y = 10` can never be true based on statistics |
| 49 | |
| 50 | // Start with the expression `x = 5 AND y = 10` |
| 51 | let expr = col("x").eq(lit(5)).and(col("y").eq(lit(10))); |
| 52 | |
| 53 | // We can analyze this predicate using information provided by the |
| 54 | // `PruningStatistics` trait, in this case we'll use a simple catalog that |
| 55 | // models three files. For all rows in each file: |
| 56 | // |
| 57 | // File 1: x has values between `4` and `6` |
| 58 | // y has the value 10 |
| 59 | // |
| 60 | // File 2: x has values between `4` and `6` |
| 61 | // y has the value of `7` |
| 62 | // |
| 63 | // File 3: x has the value 1 |
| 64 | // nothing is known about the value of y |
| 65 | let my_catalog = MyCatalog::new(); |
| 66 | |
| 67 | // Create a `PruningPredicate`. |
| 68 | // |
| 69 | // Note the predicate does not automatically coerce types or simplify |
| 70 | // expressions. See expr_api.rs examples for how to do this if required |
| 71 | let predicate = create_pruning_predicate(expr, &my_catalog.schema); |
| 72 | |
| 73 | // Evaluate the predicate for the three files in the catalog |
| 74 | let prune_results = predicate.prune(&my_catalog)?; |
| 75 | println!("Pruning results: {prune_results:?}"); |
| 76 | |
| 77 | // The result is a `Vec` of bool values, one for each file in the catalog |
| 78 | assert_eq!( |
| 79 | prune_results, |
| 80 | vec![ |
| 81 | // File 1: `x = 5 AND y = 10` can evaluate to true if x has values |
| 82 | // between `4` and `6`, y has the value `10`, so the file can not be |
| 83 | // skipped |
| 84 | // |
| 85 | // NOTE this doesn't mean there actually are rows that evaluate to |
| 86 | // true, but the pruning predicate can't prove there aren't any. |
| 87 | true, |
| 88 | // File 2: `x = 5 AND y = 10` can never evaluate to true because y |
| 89 | // has only the value of 7. Thus this file can be skipped. |
| 90 | false, |
| 91 | // File 3: `x = 5 AND y = 10` can never evaluate to true because x |
| 92 | // has the value `1`, and for any value of `y` the expression will |
| 93 | // evaluate to false (`x = 5 AND y = 10 -->` false AND null` --> |
| 94 | // `false`). Thus this file can also be skipped. |
| 95 | false |
| 96 | ] |
| 97 | ); |
| 98 | |
| 99 | Ok(()) |
| 100 | } |
| 101 | |
| 102 | /// A simple model catalog that has information about the three files that store |
| 103 | /// data for a table with two columns (x and y). |