From 5fd4376802153bd15de7dad53a6a74a540441794 Mon Sep 17 00:00:00 2001 From: ishaan-jaff Date: Thu, 16 Nov 2023 14:30:33 -0800 Subject: [PATCH] (docs) lm harness --- docs/my-website/docs/simple_proxy.md | 29 +++++++++++++++++++++++++--- 1 file changed, 26 insertions(+), 3 deletions(-) diff --git a/docs/my-website/docs/simple_proxy.md b/docs/my-website/docs/simple_proxy.md index 0e7d5c3550..4b3ac7e30d 100644 --- a/docs/my-website/docs/simple_proxy.md +++ b/docs/my-website/docs/simple_proxy.md @@ -221,7 +221,10 @@ LiteLLM allows you to set `openai.api_base` to the proxy server and use all Lite This tutorial assumes you're using the `big-refactor` branch of LM Harness https://github.com/EleutherAI/lm-evaluation-harness/tree/big-refactor +NOTE: LM Harness has not updated to using `openai 1.0.0+`, in order to deal with this we will run lm harness in a venv + **Step 1: Start the local proxy** +see supported models [here](https://docs.litellm.ai/docs/simple_proxy) ```shell $ litellm --model huggingface/bigcode/starcoder ``` @@ -232,10 +235,22 @@ Using a custom api base $ export HUGGINGFACE_API_KEY=my-api-key #[OPTIONAL] $ litellm --model huggingface/tinyllama --api_base https://k58ory32yinf1ly0.us-east-1.aws.endpoints.huggingface.cloud ``` - OpenAI Compatible Endpoint at http://0.0.0.0:8000 -**Step 2: Set OpenAI API Base & Key** +**Step 2: Create a Virtual Env for LM Harness + Use OpenAI 0.28.1** +We will now run lm harness with a new virtual env with openai==0.28.1 + +```shell +python3 -m venv lmharness +source lmharness/bin/activate +``` + +Pip install openai==0.28.01 in the venv +```shell +pip install openai==0.28.01 +``` + +**Step 3: Set OpenAI API Base & Key** ```shell $ export OPENAI_API_BASE=http://0.0.0.0:8000 ``` @@ -245,7 +260,15 @@ LM Harness requires you to set an OpenAI API key `OPENAI_API_SECRET_KEY` for run export OPENAI_API_SECRET_KEY=anything ``` -**Step 3: Run LM-Eval-Harness** +**Step 4: Run LM-Eval-Harness** +```shell +cd lm-evaluation-harness +``` + +pip install lm harness dependencies in venv +``` +python3 -m pip install -e . +``` ```shell python3 -m lm_eval \