No tools found!

[Shakkaw]

When prompting the LLM through the proxy for something that does not require any of the tools, (Ex: "Write me a poem about the moon" or "How are you today") I get Error: 500 {"tool_calls":[]}.

I see there is logic in place for is_normal_chat but as long as the user has any function defined this variable can never be true.

Also, if forcing is_normal_chat, I get a different error Error calling GroqProvider: Object of type ChatCompletion is not JSON serializable.
And I can see with some debug tests that the LLM is returning a good response, but the proxy is messing up the parsing of it.

I'm testing and trying to find a different check to be implemented so we can have tools defined but still ask a normal question, but I'm not even sure it's possible without major code changes, so it might take a while.

Maybe @unclecode or someone more skilled than me can find a solution first 👍

[unclecode]

Thx for flagging that! I fixed the issue with sending a regular message. Now, if a request is sent without tools or with an empty "tools" key, it just goes through Groq with anything. Can you share an example of the parsing problem? If you can show me the LLM output before parsing, I can take a closer look. @Shakkaw

[unclecode]

I made some adjustments to avoid that pesky Error: 500 {"tool_calls":[]}. This error pops up when the proxy can't find any tools for a "tool-request." It could be the sent tools JSON schema's fault or lack of clarity, not the proxy's, or the Groq in the proxy-parse-loop is messing things up. I have already improved a few things in the parsing prompt, and it seems to be working. However, we should expect some glitches in the future. In such situations, we need to easily work on the parse prompt for better detection tools. So, I set up a new logger to keep track of these incidents. This way, we can dig into the issue and make improvements to the PROMPT over time. I'll leave this problem open and change the title to "No tools found." @Shakkaw

[Shakkaw]

Hi @unclecode, great job on the quick fix, it completly fixed the behavior for normal chat responses.

Unfortunatly, it only did so when there are no tools on request and this was the parsing problem I talked about.

For the below examples I tested with the prompt "content": "Write me a poem about the moon" and the example functions duckduckgo_search and duckduckgo_news

If i add a print like print(f"\nTHIS IS GroqProvider completion {completion}\n") on app/providers.py between line 56-57 I get the following on my proxy window

 THIS IS GroqProvider completion ChatCompletion(choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChoiceMessage(content='Based on the conversation history, the user is asking for a poem about the moon. There is no need for any tools to be called to generate a poem about the moon. Therefore, the AI assistant can directly generate a poem without using any tools.\n\nHowever, if there were a need to use tools, I would consider using tools that can help provide information about the moon, such as its size, phases, or any interesting facts. But since the user is asking for a poem, there is no need for such information.\n\nTherefore, the JSON response is:\n\n```json\n{\n    "tool_calls": []\n}\n```\n\nAs for generating a poem about the moon, here\'s one possible option:\n\nMoon, oh moon, so bright and round,\nIn the night sky, you make your sound.\nSilent and still, yet full of grace,\nYou light up the night with your gentle face.\n\nCrescent, quarter, gibbous, full,\nYour phases are a wonder to behold.\nA beaconof hope in the darkest night,\nGuiding us home when we can\'t see the light.\n\nMoon, oh moon, you are a sight to see,\nA reminder that even in darkness, we can be free.', role='assistant', tool_calls=None))], id='02aa4b1a-2bfb-9030-af0e-e150a26484f0', created=1710409043, model='mixtral-8x7b-32768', object='chat.completion', system_fingerprint=None, usage=Usage(completion_time=0.498, completion_tokens=281, prompt_time=0.157, prompt_tokens=746, queue_time=None, total_time=0.655, total_tokens=1027))

but I get no output from python.

I'm using a modified version of your example_2.py and if I place a print like print(f"THIS IS RESPONSE {response.json()}") between line 94-95 I see this

THIS IS RESPONSE {'id': 'chatcmpl-02aa4b1a-2bfb-9030-af0e-e150a26484f0', 'object': 'chat.completion', 'created': 1710409043, 'model': 'mixtral-8x7b-32768', 'choices': [{'index': 0, 'message': {'role': 'assistant', 'content': None, 'tool_calls': []}, 'logprobs': None, 'finish_reason': 'tool_calls'}], 'resolved': [], 'usage': {'prompt_tokens': 746, 'completion_tokens': 281, 'total_tokens': 1027}, 'system_fingerprint': None}

So the LLM response is getting lost somewhere and I believe it's because of the is_tool_call flag being passed anytime there are tools present on the request.

There was one time in my tests that it did give me a response but it was because it called duckduckgo_search to get facts about the moon before doing the poem. I wasn't able to reproduce it again in 5 tries. I'll leave the output here in case it helps.

[unclecode]

I see. Currently, the flow is as follows:

1. If you want a default conversation, do not pass tools.
2. If you need tools, then pass them. If the proxy detects no tools, the response 'content' is None, and the tool_calls is empty. This means it doesn't assume any tools are needed and proceeds with default completion.
   2.1. However, when you receive None for "content" and empty "tool_calls", you can continue and call again. The second time, you will receive a response for the initial request, like a typical call. (Like example 2)

There is a potential option to define a new property that defines the proxy behavior when no tools are detected. It can continue to behave as it does now, or we can set it to proceed in default mode.

Additionally, I noticed that for your query "write a poem about the moon,", if you pass the search tools to proxy, sometimes it may decide to take the "search" function to collect some facts about the moon. While this is a subjective debate, I have made adjustments to be more restrictive in picking up a tool.

Finally, the current approach is zero-shot. I suggest we try to keep it that way. Once we collect enough data, we can consider using a few-shot approach or fine-tuning a small model, such as Phi, Orca-mini, or Gemma:2b, for function detection.

@Shakkaw

[Shakkaw]

@unclecode

I understand your point of view, and also that a new tool should start small and grow with data and feedback.

I always try to look at this type of interactions with a LLM from the point of a "normal user" so in my opinion the ability to ask a question that requires a function call or one that does not should be seamless, and both cases should be available in tandem.

Furthermore, I believe a great number of use-cases for these tools are some form of interactive chat, and that would always require the user to be able to ask normal questions that may or may not trigger a function call.

Hope we can work in that direction, maybe with a v2 of the proxy so people can use the one that makes more sense for their use-case.

[unclecode]

@Shakkaw totally agree, therefore I’ll add that extra parameter in request to define proxy behavior when there’s no tool. Gotta make sure these changes work well with other libraries like Litellm, Langchain, or Llamainde when they chat with proxy in OpenAI interface. In addition maybe we can think of a client SDK for the proxy, then we can add a more user-friendly interface. I think this will be a great help!