SlackBolt is Underrated: Building an AI Knowledge Bot for Your Team

William VanSickle III
Jul 17, 2025 • 4 minutes

I usually start my blog posts with something that dropped recently, but today I want to highlight a library that’s been around for a while and still provides massive value while being surprisingly underrated. I’m talking about SlackBolt.(opens in a new tab)

Picture this: You’re at a company with a giant, difficult-to-manage knowledge base spread across multiple platforms — Confluence,(opens in a new tab) Notion,(opens in a new tab) internal wikis, and scattered documentation. You’ve been there before — frantically searching for answers while a potential client waits (im)patiently on the line. We’ve all made it through situations like that as a team, but there has to be a better way.

SlackBolt has been sitting in my GitHub stars for MONTHS because I kept seeing it in random documentation and thinking “oh cool, another Slack thing” — completely missing the point. Then I actually read the docs during a late-night rabbit hole session and realized this thing is basically a Swiss Army knife disguised as a butter knife.

The SDK is genuinely beautiful — like, whoever designed the Python interface actually cares about developer experience. Compare this to wrestling with webhooks and parsing Slack’s Event API manually, or trying to make Zapier do something it wasn’t designed for. SlackBolt feels like someone finally said “what if developers could just… write code?”

So I threw together a prototype using SlackBolt, Atlassian’s MCP Server, and an MCP Agent to show how easy (and fun!) it can be to home roll useful Q&A bots. MCP (Model Context Protocol)(opens in a new tab) basically standardizes how AI agents talk to external systems — the Atlassian server means I can search Confluence without writing API integration code. SlackBolt + MCP + some routing logic = internal knowledge assistant that actually works.

I’m actually going to walk you through this ~

Our Slackbot will:

  • Accept questions via a /ai command

  • Route queries intelligently based on content

  • Search through Confluence (or other knowledge bases)

  • Provide contextual answers with conversation memory

  • Scale to handle multiple knowledge sources

Architecture Overview

  • I’m a novice developer and wanted to make sure that my architecture was scalable and easily extendable due to the varying number of requests I get for these bots.

  • There’s definitely more advanced hybrid strategies to get better results than the Confluence search and get page tools provide, this was a POC so please extend and modify then share what you do!

  • Slackbolts framework made this super easy to build out once I got my slackbot connected locally — deployment was a little more tricky but will easily run in a Docker ;-)

Main Components:

  1. Slack Front-End — Accepts /ai queries and forwards them to the processing pipeline

  2. Router LLM — Classifies messages using a lightweight model

  3. Context Builder — Dynamically assembles the correct prompt and knowledge base

  4. MCP Agent — Runs with various tools (Confluence, etc.) to fetch information

  5. Memory System — Maintains conversation context across threads

Tutorial: Building Your Own AI Slack Bot

Step 1: Setting Up SlackBolt

First, let’s set up the basic Slack application: Here you make sure the Slack app can listen and respond to Slack messages. You can also control interactivity here ~ I made my bot start a new thread and continue the conversation with a new UUID to keep track over multiple threads of conversation history.

# slack-bolt.py
import os
import logging
from slack_bolt import App
from slack_bolt.adapter.socket_mode import SocketModeHandler

# Initialize your app with your bot token and socket mode handler
app = App(token=os.environ.get("SLACK_BOT_TOKEN"))

@app.command("/ai")
def handle_ai_command(ack, respond, command):
    ack()
    user_query = command['text']
    thread_ts = command.get('thread_ts', command['ts'])
    
    # Process the query (we'll implement this next)
    response = generate_response(user_query, thread_ts)
    respond(response, thread_ts=thread_ts)

if __name__ == "__main__":
    handler = SocketModeHandler(app, os.environ["SLACK_APP_TOKEN"])
    handler.start()

Step 2: Implementing the Router System

Create a routing system that classifies queries (use a lighter-weight model for this, I picked 4.1-mini):

# bot.py
import openai
from typing import Dict, Any

# Define your categories
CATEGORIES = {
    "general": {
        "prompt_file": "prompts/general.txt",
        "kb_file": None,
        "description": "For general company questions and policies",
        "aliases": ["general", "company", "policy", "process"],
        "context_prefix": "**Answering a general company question.**"
    },
    "technical": {
        "prompt_file": "prompts/technical.txt",
        "kb_file": "knowledge_bases/tech_kb.txt",
        "description": "For technical documentation and development questions",
        "aliases": ["tech", "development", "api", "code", "deployment"],
        "context_prefix": "**Answering a technical question. Here is relevant context:**"
    }
}

async def get_query_category(user_message: str) -> str:
    """Route the user query to the appropriate category"""
    
    # Build router prompt dynamically
    router_prompt = """
You are a query router. Classify the user's question into one of the following categories and respond with ONLY the category name:
"""
    
    for category, config in CATEGORIES.items():
        router_prompt += f"\n- '{category}': {config['description']}"
    
    response = await openai.AsyncOpenAI().chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": router_prompt},
            {"role": "user", "content": user_message}
        ],
        temperature=0,
    )
    
    category = response.choices[0].message.content.strip().lower()
    return category if category in CATEGORIES else "general"

Step 3: Setting Up MCP Integration

Install the MCP agent and configure it for your knowledge sources:

pip install mcp-agent
# mcp_integration.py
from mcp_agent import MCPApp, Agent
from mcp_agent.models import Settings, MCPSettings, MCPServerSettings, OpenAISettings

def setup_mcp_agent():
    """Configure MCP agent with Confluence tools"""
    
    # Configure Atlassian/Confluence server
    atlassian_args = [
        "mcp-atlassian",
        f"--confluence-url={os.environ.get('CONFLUENCE_URL')}",
        f"--confluence-username={os.environ.get('CONFLUENCE_USER')}",
        f"--confluence-api-token={os.environ.get('CONFLUENCE_TOKEN')}"
    ]
    
    settings = Settings(
        execution_engine="asyncio",
        mcp=MCPSettings(
            servers={
                "atlassian": MCPServerSettings(
                    command="uvx",
                    args=atlassian_args
                )
            }
        ),
        openai=OpenAISettings(
            api_key=os.environ.get("OPENAI_API_KEY"),
            default_model="gpt-4o"
        ),
    )
    
    return MCPApp(name="knowledge_bot", settings=settings)

Step 4: Adding Conversation Memory

Implement a memory system to maintain context across conversations:

# memory.py
from typing import Dict, List, NamedTuple
from datetime import datetime, timedelta
import threading

class ConversationMessage(NamedTuple):
    role: str
    content: str
    timestamp: datetime

class ConversationMemory:
    def __init__(self, max_messages: int = 10, ttl_hours: int = 24):
        self.conversations: Dict[str, List[ConversationMessage]] = {}
        self.max_messages = max_messages
        self.ttl_hours = ttl_hours
        self.lock = threading.Lock()
    
    def add_message(self, thread_id: str, role: str, content: str):
        """Add a message to the conversation history"""
        with self.lock:
            if thread_id not in self.conversations:
                self.conversations[thread_id] = []
            
            message = ConversationMessage(
                role=role,
                content=content,
                timestamp=datetime.now()
            )
            
            self.conversations[thread_id].append(message)
            
            # Keep only the most recent messages
            if len(self.conversations[thread_id]) > self.max_messages:
                self.conversations[thread_id] = self.conversations[thread_id][-self.max_messages:]
    
    def get_conversation_history(self, thread_id: str) -> str:
        """Get formatted conversation history"""
        if thread_id not in self.conversations:
            return ""
        
        history = []
        for msg in self.conversations[thread_id]:
            history.append(f"{msg.role}: {msg.content}")
        
        return "\n".join(history)

# Global memory instance
memory = ConversationMemory()

Step 5: Putting It All Together

Now let's create the main response generation function:

# response_generator.py
async def generate_response(message: str, thread_id: str) -> str:
    """Generate AI response to user query"""
    
    # Get conversation history
    conversation_history = memory.get_conversation_history(thread_id)
    
    # Route the query
    category = await get_query_category(message)
    config = CATEGORIES[category]
    
    # Load prompt and knowledge base
    prompt = load_prompt(config["prompt_file"])
    kb_content = load_knowledge_base(config["kb_file"]) if config["kb_file"] else ""
    
    # Build final message with context
    final_message = f"""
{config['context_prefix']}

{kb_content}

Conversation History:
{conversation_history}

Current Question: {message}
"""
    
    # Execute with MCP agent
    mcp_app = setup_mcp_agent()
    async with mcp_app:
        finder_agent = Agent(
            name="knowledge_finder",
            instruction=prompt,
            server_names=["atlassian"]
        )
        
        async with finder_agent:
            from mcp_agent.models import OpenAIAugmentedLLM
            llm = await finder_agent.attach_llm(OpenAIAugmentedLLM)
            result = await llm.generate_str(message=final_message)
    
    # Store the interaction in memory
    memory.add_message(thread_id, "User", message)
    memory.add_message(thread_id, "Assistant", result)
    
    return result

def load_prompt(filename: str) -> str:
    """Load prompt from file"""
    try:
        with open(filename, 'r') as f:
            return f.read()
    except FileNotFoundError:
        return "You are a helpful assistant that answers questions based on available knowledge."

def load_knowledge_base(filename: str) -> str:
    """Load knowledge base content"""
    if not filename:
        return ""
    
    try:
        with open(filename, 'r') as f:
            return f.read()
    except FileNotFoundError:
        return ""

Extending Your Bot

This is where the bot truly shines. Implement custom prompts and knowledge bases and pass the responses to those inferences to your main MCP Agent (or not) to extend the knowledge base to artifacts that are not in your company's main knowledge platform.

To add new capabilities to your bot:

  1. Create new prompt files in a prompts/ directory

  2. Add knowledge base files in a knowledge_bases/ directory

  3. Register new categories in the CATEGORIES dictionary

  4. Define routing keywords in the aliases list

Example of adding a new category:

"sales": {
    "prompt_file": "prompts/sales.txt",
    "kb_file": "knowledge_bases/sales_kb.txt",
    "description": "For sales process and customer questions",
    "aliases": ["sales", "customer", "pricing", "demo"],
    "context_prefix": "**Answering a sales question. Here is relevant context:**"
}

Production Considerations

Resources to Get Started

People should start building now to take advantage of the vacuum in the space. There are so many opportunities to level up your career and get un-stuck leveraging AI-Coding assistants to help you through this stuff.

Quick start: Copy and paste this blog post into Cursor. 😉

Happy Coding!

The combination of SlackBolt and MCP creates a powerful foundation for building intelligent workplace assistants. With this architecture, you can easily extend your bot to work with any system that has an MCP server implementation, making it a scalable solution for growing teams and evolving knowledge bases.

Do you want to build code that helps entrepreneurs and small businesses grow with confidence? We’re hiring across our Technology teams, come build with us! Check out our Careers page.