> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zyeta.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Chat System

> Understanding the chat system, message management, file uploads, and real-time streaming

The Chat System is the core conversational interface in Definable.ai that enables users to interact with AI models and agents through natural language conversations. It provides a complete messaging infrastructure with support for multi-turn dialogues, file attachments, knowledge base integration, and real-time streaming responses.

## What is a Chat?

A Chat (also called Chat Session) is a conversational thread that:

* **Maintains Context**: Keeps track of the entire conversation history
* **Supports Multi-turn Dialogue**: Allows back-and-forth exchanges between users and AI
* **Integrates Knowledge**: Can leverage knowledge bases for enhanced responses
* **Handles Media**: Supports file uploads including documents, images, audio, and video
* **Streams Responses**: Provides real-time token-by-token streaming for immediate feedback
* **Tracks Usage**: Monitors token consumption and billing information

## Chat Architecture

```mermaid theme={null}
graph TB
    subgraph "Chat System Components"
        Session[📝 Chat Session]
        Messages[💬 Messages]
        Uploads[📎 File Uploads]
        Settings[⚙️ Settings]
    end

    subgraph "AI Integration"
        Models[🤖 LLM Models]
        Agents[🎯 AI Agents]
        KB[📚 Knowledge Bases]
        Prompts[📋 Prompts/Instructions]
    end

    subgraph "Processing Pipeline"
        Input[📥 User Input]
        Context[🧠 Context Building]
        Generation[✨ AI Generation]
        Stream[📡 Response Streaming]
        Storage[💾 Message Storage]
    end

    subgraph "External Services"
        LLMProviders[🌐 LLM Providers]
        FileStorage[☁️ GCP Storage]
        Transcription[🎤 Speech-to-Text]
        Billing[💳 Credits System]
    end

    Session --> Messages
    Session --> Uploads
    Session --> Settings

    Messages --> Models
    Messages --> Agents
    Messages --> Prompts

    Input --> Context
    Context --> KB
    Context --> Uploads
    Context --> Generation
    Generation --> Models
    Generation --> Agents
    Generation --> Stream
    Stream --> Storage

    Generation --> LLMProviders
    Uploads --> FileStorage
    Input --> Transcription
    Generation --> Billing

    style Session fill:#e3f2fd
    style Messages fill:#f3e5f5
    style Models fill:#e8f5e8
    style Agents fill:#fff3e0
    style Stream fill:#fce4ec
```

## Core Components

### 1. Chat Sessions

A chat session is the container for an entire conversation.

**Key Features:**

* **Unique Identifier**: Each session has a UUID for tracking
* **Title Management**: Auto-generates meaningful titles from conversation content
* **Status Tracking**: Active, Archived, or Deleted states
* **Metadata Storage**: Flexible JSON storage for custom data
* **Settings Persistence**: Saves user preferences for temperature, max tokens, etc.

**Session Lifecycle:**

```mermaid theme={null}
stateDiagram-v2
    [*] --> Created: Create Session
    Created --> Active: First Message
    Active --> Active: Continue Conversation
    Active --> Archived: Archive Session
    Active --> Deleted: Delete Session
    Archived --> Active: Restore Session
    Archived --> Deleted: Delete Session
    Deleted --> [*]

    state Active {
        [*] --> Chatting
        Chatting --> WaitingResponse: Send Message
        WaitingResponse --> Chatting: Receive Response
    }
```

### 2. Messages

Messages are the individual exchanges within a chat session.

**Message Types:**

* **USER**: Messages sent by the human user
* **MODEL**: Responses generated by LLM models
* **AGENT**: Responses from AI agents

**Message Structure:**

* Content (text/media)
* Role (USER/MODEL/AGENT)
* Parent message ID (for threading)
* Model or Agent ID
* Prompt/Instruction ID
* Metadata (knowledge base IDs, file references)
* Timestamps

**Message Threading:**

```mermaid theme={null}
graph LR
    U1[User Message 1] --> M1[Model Response 1]
    M1 --> U2[User Message 2]
    U2 --> M2[Model Response 2]
    M2 --> U3[User Message 3]
    U3 --> M3[Model Response 3]

    style U1 fill:#bbdefb
    style U2 fill:#bbdefb
    style U3 fill:#bbdefb
    style M1 fill:#c8e6c9
    style M2 fill:#c8e6c9
    style M3 fill:#c8e6c9
```

### 3. File Uploads

The chat system supports rich media attachments.

**Supported File Types:**

* **Documents**: PDF, TXT, DOCX, XLSX, CSV
* **Images**: JPG, PNG, GIF, WebP
* **Audio**: MP3, WAV, M4A, OGG
* **Video**: MP4, WebM, MOV

**Upload Process:**

```mermaid theme={null}
sequenceDiagram
    participant User
    participant ChatAPI
    participant GCP
    participant Database

    User->>ChatAPI: Upload File
    ChatAPI->>GCP: Store File
    GCP-->>ChatAPI: Presigned URL
    ChatAPI->>Database: Save Upload Metadata
    Database-->>ChatAPI: Upload Record
    ChatAPI-->>User: Upload Response (ID + URL)

    Note over User,Database: File can now be attached to messages

    User->>ChatAPI: Send Message with file_uploads
    ChatAPI->>Database: Link Upload to Message
    ChatAPI->>ChatAPI: Extract File Content
    ChatAPI->>ChatAPI: Include Content in Prompt
```

**File Processing:**

* Files are uploaded to GCP Storage
* Presigned URLs generated for secure access
* Content extraction for text-based files
* Image/video processing for multimodal models

### 4. Chat Settings

Per-session LLM parameters that override defaults.

**Available Settings:**

* `temperature`: Controls randomness (0.0 - 1.0)
* `max_tokens`: Maximum response length
* `top_p`: Nucleus sampling threshold

**Settings Hierarchy:**

1. Request-level parameters (highest priority)
2. Saved session settings
3. Model defaults (lowest priority)

## Chat Features

### Real-time Streaming

Responses stream token-by-token for immediate user feedback.

**Streaming Flow:**

```mermaid theme={null}
sequenceDiagram
    participant Client
    participant ChatAPI
    participant LLMProvider
    participant Database

    Client->>ChatAPI: POST /send_message
    ChatAPI->>Database: Create User Message
    ChatAPI->>Database: Initialize Billing Hold

    ChatAPI->>LLMProvider: Stream Request

    loop Token Streaming
        LLMProvider-->>ChatAPI: Token Chunk
        ChatAPI-->>Client: SSE: {message: "token"}
    end

    LLMProvider-->>ChatAPI: Usage Data

    ChatAPI->>Database: Save AI Response
    ChatAPI->>Database: Update Usage Metadata
    ChatAPI->>Database: Finalize Billing
    ChatAPI-->>Client: SSE: {message: "DONE"}

    opt New Chat
        ChatAPI->>ChatAPI: Generate Title
        ChatAPI->>Database: Update Chat Title
        ChatAPI->>Client: WebSocket: Title Update
    end
```

**Stream Format:**

```
data: {"message": "Hello"}
data: {"message": " there"}
data: {"message": "!"}
data: {"message": "DONE"}
```

**Special Stream Types:**

* **Text Tokens**: Regular chat responses
* **Reasoning Steps**: For models with reasoning capabilities
* **Image Generation**: Progress updates and final URLs
* **Video Generation**: Progress updates and final URLs
* **Error Messages**: Graceful error handling

### Knowledge Base Integration

Chats can leverage knowledge bases for enhanced responses.

**Integration Flow:**

```mermaid theme={null}
graph LR
    A[User Query] --> B[Search Knowledge Bases]
    B --> C[Retrieve Relevant Chunks]
    C --> D[Build Enhanced Prompt]
    D --> E[Send to LLM]
    E --> F[Context-Aware Response]

    style A fill:#e3f2fd
    style C fill:#f3e5f5
    style D fill:#e8f5e8
    style F fill:#fff3e0
```

**How It Works:**

1. User specifies knowledge base IDs in the message
2. System performs semantic search on user query
3. Top relevant chunks retrieved (configurable limit)
4. Context injected into the prompt
5. LLM generates response using both its knowledge and KB context

**Enhanced Prompt Format:**

```markdown theme={null}
[System Prompt]

KNOWLEDGE BASE CONTEXT:
[Knowledge Base Context]: Retrieved information from KB 1
[Knowledge Base Context]: Retrieved information from KB 2

Use the above context to answer the user's question when relevant.

USER QUESTION: [User's actual question]
```

### Audio Transcription

Convert speech to text for voice-based interactions.

**Transcription Features:**

* **Multiple Formats**: MP3, WAV, M4A, OGG
* **Language Support**: Multi-language transcription
* **Streaming Support**: Process audio files of any size
* **High Accuracy**: Powered by advanced speech recognition

**Usage Flow:**

```mermaid theme={null}
sequenceDiagram
    participant User
    participant ChatAPI
    participant SpeechService

    User->>ChatAPI: Upload Audio File
    ChatAPI->>SpeechService: Transcribe Audio
    SpeechService-->>ChatAPI: Transcribed Text
    ChatAPI-->>User: {text: "...", status: "success"}

    Note over User,ChatAPI: User can now send transcribed text
```

### Prompt Generation

AI-powered prompt enhancement and generation.

**Prompt Types:**

* **Creative**: Generate creative writing prompts
* **Task**: Create task-oriented prompts
* **Question**: Generate insightful questions
* **Continuation**: Suggest conversation continuations

**Streaming Prompt Generation:**

```mermaid theme={null}
sequenceDiagram
    participant Client
    participant ChatAPI
    participant PromptEngine

    Client->>ChatAPI: POST /prompt
    ChatAPI->>PromptEngine: Generate Prompts

    loop Token Streaming
        PromptEngine-->>ChatAPI: Prompt Token
        ChatAPI-->>Client: SSE: {content: "token"}
    end

    ChatAPI-->>Client: SSE: {content: "DONE"}
```

### Multi-Modal Support

Advanced models can handle various content types.

**Capabilities:**

* **Text Generation**: Standard chat responses
* **Image Generation**: Create images from text descriptions
* **Video Generation**: Generate short video clips
* **Vision**: Analyze uploaded images
* **Document Understanding**: Extract and analyze document content

**Model Selection:**

```python theme={null}
# Text-only chat
model_id = "gpt-4-turbo"

# Image generation
model_id = "dall-e-3"  # Supports image generation

# Video generation
model_id = "runway-gen3"  # Supports video generation
```

## Billing and Usage Tracking

### Credit System

Every chat interaction is tracked and billed.

**Billing Flow:**

```mermaid theme={null}
stateDiagram-v2
    [*] --> Hold: Initialize (qty=1)
    Hold --> Processing: Send to LLM
    Processing --> Calculate: Receive Response
    Calculate --> Debit: Finalize Charge
    Debit --> [*]

    Processing --> Release: Error Occurs
    Release --> [*]

    state Calculate {
        [*] --> CountTokens
        CountTokens --> ApplyPricing
        ApplyPricing --> WeightedTotal
        WeightedTotal --> [*]
    }
```

**Token Calculation:**

```python theme={null}
# Calculate weighted tokens based on model pricing
input_tokens = len(message.split())
output_tokens = response_token_count

# Get model-specific pricing
pricing = {
    "input": 1.0,   # credits per 1000 input tokens
    "output": 2.0   # credits per 1000 output tokens
}

# Calculate total cost
weighted_total = (input_tokens * pricing["input"]) + (output_tokens * pricing["output"])
```

**Metadata Stored:**

* Input tokens
* Output tokens
* Total tokens (weighted)
* Cached tokens (if applicable)
* Pricing ratios
* Message IDs

### Usage Metadata

Chat sessions track cumulative usage.

**Tracked Metrics:**

```json theme={null}
{
  "usage": {
    "input_tokens": 1500,
    "output_tokens": 3000,
    "total_tokens": 4500,
    "cached_tokens": 500
  }
}
```

## Chat Operations

### Creating a Chat Session

```mermaid theme={null}
sequenceDiagram
    participant User
    participant API
    participant DB

    User->>API: POST /chats
    API->>DB: Create Chat Session
    DB-->>API: Session Created
    API-->>User: {id, title, status}
```

**Auto-Created Sessions:**

* If no chat\_id provided in send\_message
* System creates session automatically
* Title set to "New Chat"
* Status set to ACTIVE

### Updating Chat Sessions

**Updatable Fields:**

* Title (manual or auto-generated)
* Status (ACTIVE, ARCHIVED, DELETED)
* Settings (temperature, max\_tokens, top\_p)

**Auto-Title Generation:**

```mermaid theme={null}
graph LR
    A[First AI Response] --> B{Is New Chat?}
    B -->|Yes| C[Generate Title via LLM]
    B -->|No| E[Skip]
    C --> D[Update Database]
    D --> F[Broadcast via WebSocket]
    D --> G[Update Firebase]
```

### Deleting Chat Sessions

**Single Delete:**

* Soft delete (status change to DELETED)
* Cascade deletes messages
* Removes file upload links

**Bulk Delete:**

* Delete multiple sessions in one request
* Validates ownership
* Returns count of deleted sessions

## Advanced Features

### Agent Integration

Chats can interact with deployed AI agents.

**Agent Chat Flow:**

```mermaid theme={null}
sequenceDiagram
    participant User
    participant ChatAPI
    participant AgentRuntime
    participant Database

    User->>ChatAPI: Send Message (agent_id)
    ChatAPI->>Database: Create User Message
    ChatAPI->>AgentRuntime: POST /{slug}/{version}/invoke

    loop Streaming Response
        AgentRuntime-->>ChatAPI: Chunk
        ChatAPI-->>User: Forward Chunk
    end

    AgentRuntime-->>ChatAPI: DONE
    ChatAPI->>Database: Save Agent Response
    ChatAPI->>Database: Finalize Billing
    ChatAPI-->>User: {message: "DONE"}
```

**Agent Requirements:**

* Agent must be active
* Agent must be deployed
* User must have access to agent

### Instruction/Prompt System

Use pre-defined prompts to guide AI behavior.

**Prompt Integration:**

```mermaid theme={null}
graph TD
    A[User Message] --> B{Instruction ID?}
    B -->|Yes| C[Load Prompt Content]
    B -->|No| D[No System Prompt]
    C --> E[Set as System Prompt]
    E --> F[Send to LLM]
    D --> F
    F --> G[AI Response]
```

**Prompt Components:**

* ID: Unique identifier
* Title: Display name
* Description: Purpose description
* Content: Actual prompt text

### WebSocket Updates

Real-time updates for chat state changes.

**Events Broadcasted:**

* Chat title updates (after auto-generation)
* New message notifications
* Status changes

**WebSocket Message Format:**

```json theme={null}
{
  "data": {
    "id": "chat-uuid",
    "title": "Updated Title",
    "user_id": "user-uuid",
    "org_id": "org-uuid"
  }
}
```

### Firebase Integration

Chat updates sync to Firebase Realtime Database.

**Firebase Path:**

```
{org_id}/chats_write/data
```

**Data Structure:**

```json theme={null}
{
  "id": "chat-uuid",
  "title": "Chat Title"
}
```

## Best Practices

### Chat Management

1. **Session Organization**: Use meaningful titles and archive old chats
2. **Settings Optimization**: Adjust temperature and max\_tokens per use case
3. **Knowledge Base Selection**: Only include relevant KBs to reduce noise
4. **File Management**: Clean up unused uploads regularly

### Performance Optimization

1. **Streaming**: Always use streaming for better UX
2. **Context Window**: Monitor message count and summarize long conversations
3. **Token Limits**: Set appropriate max\_tokens to control costs
4. **Caching**: Leverage cached tokens when available

### Error Handling

1. **Billing Failures**: Handle insufficient credits gracefully
2. **LLM Errors**: Display user-friendly error messages
3. **File Upload Limits**: Validate file size and type before upload
4. **Timeout Handling**: Set appropriate timeouts for long operations

### Security Considerations

1. **Access Control**: Use RBAC to control chat access
2. **File Validation**: Validate file types and scan for malware
3. **Content Filtering**: Implement content moderation as needed
4. **Data Privacy**: Handle sensitive conversations appropriately

## Common Use Cases

### Customer Support Bot

```mermaid theme={null}
graph TD
    A[Customer Query] --> B[Search Knowledge Base]
    B --> C[Retrieve Help Articles]
    C --> D[Generate Response]
    D --> E{Issue Resolved?}
    E -->|Yes| F[Close Chat]
    E -->|No| G[Escalate to Human]
```

**Configuration:**

* Temperature: 0.3 (more deterministic)
* Knowledge Bases: FAQ, Product Docs, Support Articles
* Prompts: Customer service instructions

### Code Assistant

**Features:**

* File upload for code review
* Multi-turn debugging conversations
* Code generation with context
* Documentation lookup via KB

**Configuration:**

* Temperature: 0.7 (balanced creativity)
* Models: GPT-4, Claude-3.5-Sonnet
* File types: .py, .js, .java, etc.

### Research Assistant

**Features:**

* Document upload and analysis
* Knowledge base integration
* Citation tracking
* Long-form responses

**Configuration:**

* Temperature: 0.5 (factual)
* Max tokens: 4096 (long responses)
* Knowledge Bases: Research papers, articles

### Creative Writing

**Features:**

* Prompt generation
* Story continuation
* Character development
* Style adaptation

**Configuration:**

* Temperature: 0.9 (highly creative)
* Prompt type: "creative"
* Models: GPT-4, Claude-3-Opus

## Performance Metrics

### Quality Metrics

* **Response Relevance**: Accuracy of AI responses
* **Context Retention**: How well context is maintained
* **Knowledge Integration**: Effectiveness of KB usage
* **Error Rate**: Frequency of failed interactions

### Efficiency Metrics

* **Response Time**: Time to first token
* **Streaming Speed**: Tokens per second
* **Token Usage**: Input/output token counts
* **Cost per Chat**: Average credits consumed

### Usage Metrics

* **Active Sessions**: Number of ongoing chats
* **Message Volume**: Total messages sent
* **File Uploads**: Number and size of uploads
* **Knowledge Base Hits**: KB search frequency

## Troubleshooting

### Common Issues

**Slow Responses:**

* Check LLM provider status
* Reduce max\_tokens
* Optimize knowledge base searches
* Monitor network latency

**Billing Errors:**

* Verify sufficient credits
* Check transaction logs
* Review usage metadata
* Contact support for discrepancies

**Context Loss:**

* Keep conversations under context limit
* Implement conversation summarization
* Use parent\_message\_id correctly
* Check message ordering

**File Upload Failures:**

* Validate file size limits
* Check GCP storage configuration
* Verify content type
* Review network connectivity

## Next Steps

Explore related concepts and start building:

* [**Knowledge Base**](/pages/concepts/knowledge-base) - Enhance chats with domain knowledge
* [**AI Agents**](/pages/concepts/agents) - Build autonomous conversational agents
* [**LLM Models**](/pages/concepts/models) - Choose the right model for your use case
* [**Tools**](/pages/concepts/tools) - Extend chat capabilities with custom functions

Ready to start chatting? Check out the [**Chat API Reference**](/pages/api-reference/chat-service) or our [**Getting Started Guide**](/pages/getting-started/quickstart).