ti.sil/pr-agent
1
0
Fork 0
mirror of https://github.com/qodo-ai/pr-agent.git synced 2025-12-11 18:35:18 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

disabling encoding error on special_tokens

Browse source
This commit is contained in:
Hussam.lawen 2023-07-06 13:22:12 +03:00
parent 74dfae8dbe
commit 259fa84eeb
No known key found for this signature in database
GPG key ID: 07E08C11EA208BC2
1 changed files with 1 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
pr_agent/algo/token_handler.py
View file

@ -21,4 +21,4 @@ class TokenHandler:
return system_prompt_tokens + user_prompt_tokens
def count_tokens(self, patch: str) -> int:
return len(self.encoder.encode(patch))
return len(self.encoder.encode(patch, disallowed_special=()))