LTR4L

Learning-to-Rank for Apache Lucene (compatibility with Apache Lucene is still a work-in-progress).

The purpose of this document is to provide instructions on how to execute the program, some experimental results. A general overview is also provided at the end of the document for those who would like to review or learn more about the algorithms. The general overview is meant to provide a "big picture," interpretation, or explanation of each algorithm, as well as pseudo code. For specific mathematical details and formulation, please see implementation or original articles.

Table of Contents

1. Introduction to LTR

2. Execution Instructions

   • Requirements
   • Data
   • How to Execute Training Program
   • How to Execute Prediction Program
   • Changing Parameters

3. Experiments

4. Overview of Algorithms
   • PRank
   • OAP-BPM
   • NNRank
   • RankNet
   • FRankNet
   • LambdaRank
   • SortNet
   • ListNet
5. Overview of Optimizers
6. Overview of Evaluation Metrics

Introduction

Ranking models are an essential component of information retrieval systems.

Consider the pool of search results for a given search. It is largely impossible to have a pool of search results which will exactly match the set of results the user is looking for. Thus, by vastly increasing the number of search results, one can hope to include all of the desired results of the user. The problem with this, however, is that it becomes much harder for the user to find the desired results among the large pool of search results.
Ranking serves as means of mitigating this problem. If all of the user's desired results are pushed to the top of the list, then it does not take much effort for the user to find their desired results.
The ability for a "ranker" to rank the search results accurately is dependent on the ranking model. Learning to Rank is the application of machine learning to construct the ranking model.

There are three different types of machine learning algorithms: supervised learning (labels, or the "correct answer," is provided for all data), unsupervised learning (no labels provided), and reinforcement learning (training through trial and error in a specific environment).

In Learning to Rank, there are three different approaches: pointwise, pairwise, and listwise. In pointwise, training involves looking at the data or documents independently. Each document will have a relevance score, which is unrelated or unaffected by other documents. Because of this, the final rank of the document is not visible to the loss function. In addition, this method does not take into account the fact that documents may actually be related, or that documents are related to a certain query.

The pairwise approach involves considering documents in pairs. Learning occurs by looking at which document is more relevant. For example, the model may make a prediction about which document in a pair is more relevant, and compare that to the labels of each document. Then, the model is adjusted. Depending on the algorithm, the model may output a score for a single document, however the loss function (and thus training) will usually require a pair of documents in this case.

The listwise approach involves looking at lists of documents associated with a query. While the model may output a score for one document, the loss function requires a full list of documents in a query. Thus, during learning, information about all of the documents is necessary.

This project currently only implements algorithms of the supervised learning variety, and thus we will focus on supervised learning in this document. The implemented algorithms include pointwise, pairwise, and listwise approaches.

Execution Instructions

Requirements

You will need to install the following:

1. Java 8+

2. Apache Ant 1.8+ (Not required to run, however the following instructions will assume Apache Ant has been installed.)

Data

You can download OHSUMED, LETOR MQ2007, LETOR MQ2008 from this web page.

Dataset Descriptions

Each row is a query-document pair. The first column is relevance label of this pair, the second column is query id, the following columns are features, and the end of the row is comment about the pair, including id of the document. The larger the relevance label, the more relevant the query-document pair. A query-document pair is represented by a 46-dimensional feature vector. Here are several example rows from MQ2007 dataset:

====================================================

2 qid:10032 1:0.056537 2:0.000000 3:0.666667 ... 46:0.076923 #docid=GX029-35-5894638 inc=0.02 prob=0.13984

0 qid:10032 1:0.279152 2:0.000000 3:0.000000 ... 46:1.000000 #docid=GX030-77-6315042 inc=0.71 prob=0.34136

0 qid:10032 1:0.130742 2:0.000000 3:0.333333 ... 46:1.000000 #docid=GX140-98-1356607 inc=0.12 prob=0.07013

1 qid:10032 1:0.593640 2:1.000000 3:0.000000 ... 46:0.000000 #docid=GX256-43-0740276 inc=0.01 prob=0.40073

====================================================

How to Execute Training Program

1- Download the project from Github

git clone https://github.com/LTR4L/ltr4l.git

2- go to the project folder

$ cd ltr4l

3- run ivy bootstrap (only needs to be run once after cloned)

ant ivy-bootstrap

4- package jar file

$ ant clean package

5-run the following command :

train ranknet

To see help of train command, use -help option:

train -help

usage: train <LTR-algorithm-name> [-config <file>] [-debug] [-help]
       [-iterations <num>] [-model <file>] [-report <file>] [-training
       <file>] [-validation <file>] [-verbose] [-version]

Execute Learning-to-Rank training algorithm. The algorithm is specified by
the required argument <LTR-algorithm-name>. The program will look for the
configuration file "config/<LTR-algorithm-name>.json" unless config option
is specified. The following options can be specified in order to override
the existing settings in the config file.

 -config <file>       use given file for configuration
 -debug               print debugging information
 -help                print this message
 -iterations <num>    use given number of iterations
 -model <file>        specify model file name
 -report <file>       specify report file name
 -training <file>     use given file for training
 -validation <file>   use given file for validation
 -verbose             be extra verbose
 -version             print the version information and exit

6- open the report file:

open report/ranknet-report.csv

How to Execute Prediction Program

If you have a model JSON file (for example, the model saved from running the training program), you can run the prediction program using that model file.

Assuming the default parameters (using the default location models are saved to by the training program), the execution instructions are as follows:

1- go to the project folder

cd ltr4l

2- run the following command (if jar file is not packaged, do step 3 in the previous section):

./predict ranknet

To see help for the predict command, use the -help option:

usage: predict <LTR-algorithm-name> [-debug] [-eval <evalType>] [-help]
       [-k <k>] [-model <file>] [-report <file>] [-test <file>] [-verbose]
       [-version]

Execute Learning-to-Rank predicting algorithm. The algorithm is specified
by the required argument <LTR-algorithm-name>. The program will look for
the model file "model/<LTR-algorithm-name>-model.json" unless model option
is specified. The following options can be specified in order to override
the existing settings in the config file.

 -debug             print debugging information
 -eval <evalType>   specify type of evaluator
 -help              print this message
 -k <k>             specify k-value for evaluators which use @k
 -model <file>      use given file for configuration and model
 -report <file>     specify report file name
 -test <file>       use given file for testing the model
 -verbose           be extra verbose
 -version           print the version information and exit

Changing Parameters

Below is an example of a config file:

{
  "algorithm" : "RankNet",
  "numIterations" : 100,
  "params" : {
    "learningRate" : 0.00001,
    "optimizer" : "adam",
    "weightInit" : "xavier",
    "regularization" : {
      "regularizer" : "L2",
      "rate" : 0.01
    },
    "layers" : [
      {
        "activator" : "Sigmoid",
        "num" : 10
      }
    ]
  },

  "dataSet" : {
    "training" : "data/MQ2008/Fold1/train.txt",
    "validation" : "data/MQ2008/Fold1/vali.txt",
    "test" : "data/MQ2008/Fold1/test.txt"
  },

  "model" : {
    "format" : "json",
    "file" : "model/ranknet-model.json"
  },

  "evaluation" : {
    "evaluator" : "NDCG",
    "params" : {
      "k" : 10
    }
  },

  "report" : {
    "format" : "csv",
    "file" : "report/ranknet-report.csv"
  }
}

You must specify the number of nodes and activation for each hidden layer and the output layer. However, for NNRank, you do not need to specify the final layer. For example, to add another layer of 3 ReLu nodes and the output layer to Sigmoid, change layers to:

    "layers" : [
      {
        "activator" : "Sigmoid",
        "num" : 10
      },
      {
        "activator" : "ReLU",
        "num" : 3
      },
      {
        "activator" : "Sigmoid",
        "num" : 1
      }
    ]

You can also change the training data and validation data by changing the path of the first and second argument while executing the program:

java -jar LTR4L-X.X.X.jar ranknet data/MQ2007/Fold1/train.txt data/MQ2007/Fold1/vali.txt confs/ranknet.json

Experiments

In this section, first we provide barcharts which show NDCG@10 for each algorithm, across different datasets. Then we provide some graphs of NDCG and Loss, as well as the parameters used and elapsed time. If batch size is not included in the table, default batch size was used (i.e. weights were updated according to original papers). Note that elapsed time can vary quite a bit depending on the parameters (especially on the number of hidden layers, activation, etc...), and is only provided to give a general idea of the speed of the algorithm. Even with no change in parameters, the elapsed time can vary. Note that this is the time required for all epochs to finish: i.e. for all training and validation to complete.

NDCG Comparisons

Note that SortNet is missing from MQ2007 and OHSUMED, due to a bug. When it is fixed, the graphs will be updated.

PRank

OAP-BPM

Note: N is the number of PRanks used.

NNRank

RankNet

Note: The document pairs used are only the pairs which have different labels (i.e. two documents with label of "0") are ignored during training. In addition, only a maximum 1/6 of the queries are actually used during training (randomly selected each time). Note that elapsed time is quite large compared to other algorithms, despite this reduction in the document pairs. Also note the steady rise in NDCG and fall in loss despite the reduction of document pairs used.

As an example of the importance of tuning and the difference datasets can make, here are the results of RankNet performed on MQ2008:

FRankNet

Note: FRankNet is still work in progress.

![Alt Text](