Paillat-dev/FABLE
1
0
Fork 0
mirror of https://github.com/Paillat-dev/FABLE.git synced 2026-01-02 01:06:20 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(tokens.py): add docstring explaining the purpose of the file and its reference

Browse Source
This commit is contained in:
Paillat
2023-06-17 11:51:35 +02:00
parent 2372c18997
commit 09c98a460b
1 changed files with 7 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
utils/tokens.py
View File

@@ -1,3 +1,10 @@
'''
This file's purpose is to count the number of tokens used by a list of messages.
It is used to check if the token limit of the model is reached.
Reference: https://github.com/openai/openai-cookbook/blob/main/examples/How_to_format_inputs_to_ChatGPT_models.ipynb
'''
import tiktoken import tiktoken
async def num_tokens_from_messages(messages, model="gpt-3.5-turbo"): async def num_tokens_from_messages(messages, model="gpt-3.5-turbo"):