AI-Powered Conversations with Speckle

Recently, we highlighted a community contribution from Jon Ryan that took the AEC social media by storm. It's a part demo, part inspiration, showcasing the potential of Speckle as an open data developer platform. In this post, we'll introduce you to Jon, discuss his achievements, and give you a glimpse of how he integrated ChatGPT with Speckle's commenting feature using Langchain. Stay tuned for Part 2, where we'll dive into the code and show you how to replicate it yourself.

Featured Developer: Jon Ryan

Jon's story is about curiosity, self-learning, and the transformative potential of emerging technologies.

With a keen interest in Architectural, Engineering, and Construction (AEC) data analysis, Jon embarked on self-improvement, emphasising data science. He enrolled in courses, honed his Python skills, and discovered Speckle - the open-source data platform that transforms complex AEC data into accessible, manageable information.

Jon noticed the rising popularity of Generative AI, Natural Language Processing (NLP), and ChatGPT as he began his studies. Intrigued, he set out on a side quest to understand these technologies and their potential applications in AEC data analysis. Inspired and ready for a challenge, Jon rolled up his sleeves and began experimenting. He explored Speckle's commenting functionality and integrated it with ChatGPT using Langchain. His mission? To automate responses to comments and replies in a Speckle environment, supercharging the efficiency and effectiveness of data analysis.

Speckle's Real-Time Commenting

One of the standout features of Speckle is its real-time commenting functionality. This feature lets users directly engage in dynamic, collaborative discussions within the platform, making it a powerful team communication tool.

Effective communication is crucial in the complex world of AEC, where projects often involve multiple stakeholders working across different disciplines. Speckle Comments facilitate this by providing a platform for instant, context-rich discussions. Users can comment on specific data objects, making addressing issues easier, sharing insights, and collaborating on solutions.

‍Try Speckle Here

But the real magic of Speckle's commenting feature is its accessibility. Not only is it available to collaborators without needing special AEC tools through Speckle's user-friendly web interface on any device, but it's also accessible to developers through the GraphQL API. This means that developers can tap into the power of real-time commenting, integrating it into their own applications or using it as a foundation for innovative new features. This is precisely what Jon did when he integrated ChatGPT with Speckle's commenting feature.

By leveraging the accessibility of the commenting feature, Jon was able to create a system that automates responses to comments and replies. This enhances the efficiency of data analysis and opens up exciting new possibilities for AI-enhanced collaboration in the AEC industry.

Speckle's real-time commenting feature is more than just a communication tool. It's a gateway to a more collaborative, efficient, and innovative AEC industry. And with its open accessibility, the possibilities for what can be built upon it are virtually limitless.

Accessing PDF Attachments

What truly set Jon's demo apart and caught everyone's attention was not just the AI's interaction with BIM properties of objects but its ability to access and understand the contents of PDF attachments to a comment. This feature significantly expands the context and depth of information available to the AI, enabling it to provide more accurate and relevant responses.

When we first built the ability to add attachments to Speckle, we envisioned it as a way to include snapshots or other contextual information that could enhance the richness of the data. We thought it would be a useful feature, but we didn't anticipate how innovative our users would be in leveraging it.

In a surprisingly short time, we've seen many Speckle users take this feature and run with it, building sophisticated workflows that we hadn't even imagined. They've been programmatically using attachments and even emojis to automate processes, streamline workflows, and enhance the depth and context of their data.

But it's not just about adding pictures or screenshots. Users have attached files - from PDFs and Excel spreadsheets to CAD drawings, Grasshopper scripts and BIM models. These attachments provide a wealth of information that can be accessed and utilised in various ways.

So, who knew files were still so important? Well, it turns out they are - and our users are finding new and exciting ways to make the most of them every day.

In the AEC sector, PDF documents often contain a plethora of information - from detailed design specifications and technical drawings to project reports and regulatory compliance documents. By enabling the AI to access and understand this information, Jon's system can provide insights and responses considering a much broader context than possible with just the BIM data alone.

This ability to access PDF attachments is insightful. It opens up new possibilities for automating complex tasks, enhancing decision-making, and improving collaboration and communication. It's a testament to the power of AI, Speckle's versatility, and developers like Jon's ingenuity.

OpenAI isn’t free

While Jon's project is built on open-source technologies, there's one important caveat: the OpenAI API, which powers the GPT component of the system, requires a developer key. This key is not available for free; you'll need a paid developer account with OpenAI to access it. This is important if you consider replicating or building upon Jon's work. The rest of the tools - Speckle, Langchain, and Weaviate - are open-source and free.

Valuing open-source and the Speckle Community

Jon's journey and his willingness to share his work are emblematic of the Speckle community's spirit.

The open-source nature of Speckle fosters a culture of collaboration and generosity, where community members are encouraged to share their projects, learn from each other, and contribute to the collective knowledge base. This ethos makes the Speckle community a vibrant and innovative space for AEC professionals.

There are open-source alternatives to GPT; they often come with their own set of challenges. Some are designed for specific tasks and may be less versatile than GPT. Others are still in the early stages of development and may lack the robustness or features of more mature models. And while promising projects are in the works, they may not be widely available.

For instance, EleutherAI's GPT-Neo and GPT-J models are open-source alternatives to GPT-3. However, they may not perform at the same level as GPT-3 in certain tasks. Similarly, Hugging Face's Transformers library provides access to a wide range of pre-trained models, but using these effectively often requires a fair amount of machine-learning expertise.

In summary, while there are open-source alternatives to GPT, they may require more effort to use effectively and may not offer the same level of performance or versatility. This is why, for many developers, using GPT-3 through the OpenAI API is a worthwhile investment.

Decoding the Tech: GPT, Langchain, and Weaviate

Everyone has heard a lot about ChatGPT, but how accessible is it, and what tooling does a hacker/developer need to get started? The tools Jon put together in this hack go a long way toward demonstrating some fundamentals that are not specific to Speckle Comments but are very much an insight into recent AI hype.

We discovered an excellent primer that goes further than we can here: What Product People Need To Know About LangChain by Paul Thompson at Commandbar. It makes LangChain easy to understand, even for less technical readers.

GPT (Generative Pretrained Transformer)

Imagine GPT as a highly-skilled linguist who can understand and generate human-like text. It's like a digital Shakespeare, capable of creating engaging and coherent narratives. But instead of writing plays, GPT generates responses to user inputs, making it feel like you're conversing with an AI.

However, while GPT is a master wordsmith, it doesn't inherently understand the context of AEC data or how to interact with other software tools. It's like having a well-crafted script, but without a director to interpret it and bring it to life on stage.

Vectors

In the world of AI, vectors are like the DNA of data. They are arrays of numbers that represent data in a multi-dimensional space. GPT uses vectors to generate coherent and contextually relevant responses, converting words or phrases into vectors in a process called vectorisation. This allows GPT to perform mathematical operations on words and sentences, enabling it to understand and generate human-like text.

Langchain

This is where Langchain comes in. Think of Langchain as the director of our digital play. It provides a framework for GPT to interact with various tools and APIs, guiding the AI's responses to ensure they're linguistically impressive, contextually relevant, and actionable.

Langchain is like a translator and coordinator, helping our digital Shakespeare understand and interact with the world of AEC data. It's the bridge between GPT's linguistic prowess and the practical needs of data analysis.

Weaviate

Finally, we have Weaviate. If GPT is our playwright and Langchain is our director, then Weaviate is the library where our play is performed. It's a knowledge graph that stores and manages all GPT and Langchain's data.

Weaviate allows users to perform semantic search queries, making finding and retrieving relevant data easy. It's like a librarian who knows exactly where every book is and can instantly provide the necessary information.

In this AI-powered system, GPT, vectors, Langchain, and Weaviate each play a distinct role. GPT generates the responses, vectors enable GPT to understand and manipulate data, Langchain guides these responses to be contextually relevant and actionable, and Weaviate manages the data that the system works with. Together, they form a powerful tool for interacting with and understanding AEC data.

In this AI-powered system, GPT acts as our wordsmith, creating engaging narratives; vectors are the DNA of data, enabling context understanding; Langchain is the director, guiding responses to be relevant and actionable; and Weaviate is the vast library, managing all the knowledge.

Tokens

In GPT-3, tokens can range from one character to a whole word in length. For example, "ChatGPT is great!" is encoded into six tokens: ["Chat", "G", "PT", " is", " great", "!"]. This means very long texts may need to be truncated, summarised, or reduced to fit within this limit.

In the context of this project, this means that the amount of data that can be processed in a single interaction with the API is limited. If the comments and attached documents in Speckle are too long, they may need to be shortened or split across multiple API calls. This is an important consideration when designing and implementing a system that integrates GPT-3 with other tools and platforms. The code tutorial in part 2 will highlight how this was handled and managed.

Sharing and on to the code

Jon provides this code as-is as a general introduction and MVP demo. It may require adjustments to fit your specific needs. He's generously offered his code and insights to the Speckle open-source community and is open to suggestions and comments for expansion or corrections. We’ll cover a version of this in Part 2 and break down step by step how you could implement it yourself or adapt it to a different use case.

If you want to play along, you may want to signup and get hold of an OpenAI developer key, and If you don’t have one already, have someplace to host a webhook responsive script. If you are brave, you can run a webserver locally and open a channel with ngrok or similar. Join us at speckle.community and tell us where this could go.