Getting started
This library implements a knoweledge probing approach which uses LM's inherent ability to estimate the log-likelihood of any given textual statement. For more information visit the LM Pub Quiz website.
In this guide, we will first cover how to set up the packaged (as there are multiple options). For information on how to use the pacakge, see an example workflow using the API or the CLI-guide.
Recommend Setup
In this short guide, we explain how to setup your environment and run the first command.
If you just want to use the CLI utilities, we recommend using pipx
.
If you want to make changes to the package (in order to contribute or customize the experiments),
we recommend cloing it and use either an environment automatically managed by hatch
or a manually managed environment.
Install in a manually managed environment
(Optional) Set up your desired environment
In this example we are using conda
(and assume you want to install the CPU version of PyTorch; modify accordingly):
# create an new environment
conda create --name knowledge-probe
# activate the new environment
conda activate knowledge-probe
# install pytorch in conda
conda install pytorch cpuonly -c pytorch
Install the Package
You can install the package locally or directly from PyPI.
From PyPI
From the Source Code
Alternatively, you can clone the repository, then (in your desired environment) run:
This allows you to make changes source code which will be reflected directly.
Install in a pipx-managed environment
If you only want to use the command line interface, we recommend to install the package using pipx
.
With pipx
, you can install the package directly using pipx install lm-pub-quiz
.
This will make the commands available on your system, but isolate all dependencies.
Install in a hatch-managed environment
If you want to contribute to lm-pub-quiz, we recommend to use hatch. In this case you need to:
- Clone the repository,
- in the directoy run your respective commands in a hatch shell (either
hatch run <command>
or runhatch shell
and continue your work there).
This allows you to run the test cases by executing
To run the test cases, run hatch run test
or hatch run all:test
(to test on multiple python versions) and to check the formatting and correct typing using hatch run lint:all
.
Verify the Installation
To verify the package has been installed correctly, execute:
You should than see an output similar to this:
$ evaluate_model --help
usage: evaluate_model [-h] [config_file] <configuration options to overwrite>
Evaluate a given model on a dataset.
positional arguments:
config_file Top-level config file to load (optional).
optional arguments:
-h, --help show this help message and exit
--series-skip N Skip first N trials in the execution of a series.
configuration:
--model PATH
--model.name_or_path MODEL.NAME_OR_PATH
--model.tokenizer MODEL.TOKENIZER
--model.reduction MODEL.REDUCTION
--model.pll_metric MODEL.PLL_METRIC
--dataset PATH
--dataset.path DATASET.PATH
--output_base_path OUTPUT_BASE_PATH