ti.sil/pr-agent
1
0
Fork 0
mirror of https://github.com/qodo-ai/pr-agent.git synced 2025-12-12 10:55:17 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Merge pull request #7 from Codium-ai/algo/fix_speacial_tokens

Browse source 
Fix encoding error on special_tokens
This commit is contained in:
Ori Kotek 2023-07-06 14:14:52 +03:00 committed by GitHub
commit 1eea60c6a5
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
1 changed files with 1 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
pr_agent/algo/token_handler.py
View file

@ -21,4 +21,4 @@ class TokenHandler:
return system_prompt_tokens + user_prompt_tokens return system_prompt_tokens + user_prompt_tokens
def count_tokens(self, patch: str) -> int: def count_tokens(self, patch: str) -> int:
return len(self.encoder.encode(patch)) return len(self.encoder.encode(patch, disallowed_special=()))