Best practices for developing a generative AI co-pilot for businesses

Since launch from ChatGPT, I can’t remember a meeting with a prospect or client where they didn’t ask me how they could leverage generative AI for their business. From internal efficiency and productivity to external products and services, businesses are racing to implement generative AI technologies across all sectors of the economy.

Although GenAI is still in its infancy, its capabilities are developing rapidly: from vertical search to photo editing to writing assistants, the common thread is to exploit conversational interfaces to make software more accessible and more powerful. Chatbots, now rebranded as “co-pilots” and “assistants”, are in vogue again, and while a set of best practices is starting to emerge, the first step in developing a chatbot is to identify the problem and start small .

A copilot is an orchestrator that helps a user perform many different tasks through a free-text interface. There are an infinite number of possible input prompts, and all of them must be handled elegantly and securely. Rather than attempting to solve every task and running the risk of not meeting user expectations, developers should start by solving a single task well and learning along the way.

At AlphaSense, for example, we focused on summarizing results calls as our first single task, a well-defined but high-value task for our customer base that also fits well with existing workflows in the product. Along the way, we gleaned insights into LLM development, model choice, training data generation, augmented retrieval generation, and user experience design that enabled the expansion of open chat.

LLM development: choose open or closed

By early 2023, the LLM performance rankings were clear: OpenAI was ahead with GPT-4, but well-capitalized competitors like Anthropic and Google were determined to catch up. Open source held sparks of promise, but performance on text generation tasks was not competitive with closed models.

My experience with AI over the last decade led me to believe that open source would make a furious comeback and that is exactly what happened. The open source community has improved performance while reducing cost and latency. LLaMA, Mistral and other models provide strong foundations for innovation, and major cloud providers like Amazon, Google and Microsoft are largely adopting a multi-vendor approach, including support and amplification of open source.

Although open source has not caught up in published performance benchmarks, it is clearly a state-of-the-art closed model when it comes to the set of trade-offs any developer must make when introducing a product in the real world. The 5 Ss of template selection can help developers decide which type of template is right for them: