When search results exceed maxTokens limit, now returns partial truncated
content instead of discarding the file completely.
Changes:
- Calculate remaining token budget before breaking
- Truncate file content to fit within remaining tokens (if > 100 tokens left)
- Append truncation marker to indicate content was cut off
- Still add truncation message at end of all results
Benefits:
- Users get partial data instead of nothing
- Better debugging and analysis experience
- More useful for AI-powered code analysis tasks
- Consistent with expected behavior when limits are reached
Example: If file would use 10K tokens but only 2K remain, return
first ~8K chars of content + truncation marker instead of dropping it.
Signed-off-by: Wayne Sun <gsun@redhat.com>
This PR alters the behaviour of the search api (and all apis that depend on it) to return raw source code instead of a base64 encoding. Reasoning: we are decoding it on the client in multiple different places, so it would be beneficial to decode it in a single spot.
**Note**: This is a **breaking change** to the API surface. However, since the API surface is still unofficial/unsupported, I will roll this as a patch version change. See #101
