Multimodal multi-LLM chatbot

Applications

Multimodal, multi-LLM chatbot

Our Web site is best known for hosting the WISPL chatbot: a science-centric chatbot that works well with a multitude of language models including both frontier-class models like the latest GPT from OpenAI as well as smaller models hosted on premises.

The WISPL chatbot was created in early 2023, in response to a real need: a chatbot that renders formatted output, especially math, consistently, and produces a clean transcript that can be printed to PDF with ease.

Around that time, however, Microsoft Research published a paper that described, among other things, robust "tool use" by models such as GPT-4. They demonstrated that advanced language models can, even without pretraining, efficiently "use tools" -- that is, follow specific instructions to invoke an external tool, and then efficiently process the result in the next conversational turn -- if they receive suitable instructions.

It was this paper that inspired the creation of the WISPL chatbot in its current form. Predating "function calling" and "agentic behavior", models on WISPL were instructed on the efficient use of external tools, including calls to Maxima for computer algebra, and calls to Google for current information.

The WISPL chatbot

The WISPL chatbot has thereafter been in active development ever since. Originally designed to invoke GPT-3.5 and GPT-4, its capabilities have been expanded to include the Claude language model of Anthropic and, more recently, Gemma (Google) and Grok (xAI). The chatbot can also communicate with local, "on-premises" installations of language models that are run in the context of the llama.cpp runtime engine, an efficient C++ inference engine that can run smaller models, like the 7-billion parameter Llama model, efficiently on CPU, but also has the capability to run larger models on available GPU hardware.

Featuers of the WISPL chatbot include abilities such as:

  • Invoking Maxima for symbolic algebra, calculus or arithmetic that may exceed the model's capabilities
  • Using Google for "most recent" searches for time-sensitive information (e.g., current politics)
  • Using Google for "most relevant" searches of the model requires background information beyond its training
  • Including Maxima-generated function plots in its response
  • Producing rendered math using LaTeX, via the MathJax library
  • Poducing formatted output with markup using the markdown library
  • Automatically rendering unique formats, such as scalable vector graphics (SVG) diagrams, Lilypond music score, even FEN-encoded chessboard positions.

The WISPL engine also works with recent multimodal, text+vision models. User prompts can be supplemented with uploaded images that are transmitted to the model as part of the prompt.

The engine can also process PDF documents for inclusion. Unlike most chatbot implementations, the WISPL chatbot does not attempt to convert PDF files to text by OCR or other means. Rather, PDF content is converted to page images, which are then transmitted to the multimodal model. Modern text+vision models can "view" such document pages efficiently. This capability is no longer the exclusive domain of leading edge models: the on premises text+vision Gemma model also demonstrates robust vision capability, including the ability to process and assess multipage scientific manuscripts.

The WISPL user interface also features some unique capabilities:

  • At the beginning of the conversation, the model is informed of the current date and time and, with the user's permission, the user's geographic (lat/lon) location.
  • Conversations are not retained server-side, but may be saved by users on their local device, and reloaded/resumed at a later time.
  • Conversations are formatted for easy, high-quality printed output, especially suitable for saving as a PDF.
  • The WISPL chatbot user interface is multilingual, with four user-selectable languages currently implemented.
  • Though not presently monetized, WISPL has a fully functional user management module, including user account creation, account activation and secure password reset.

All users as welcome to try the WISPL chatbot. One caveat: Until a new account is approved by the system administrator, it is restricted to use only one chatbot: the original Eliza, the 1960s-era algorithmic chatbot that was an early experiment in machine intelligence. While this chatbot can still be useful to explore the WISPL user interface, needless to say its capabilities do not resemble those of modern large language models.

GPT chess... »

© 2023-2025, Viktor T. Toth., All Rights Reserved.