Yuclase Software Design

High-Performance Message Queue Architecture & Component Design

System Architecture Overview

High-Level Architecture

Loading diagram...

Core Component Architecture

Storage Engine Components

Append-Only Log

• Sequential write optimization
• Message format: [timestamp][size][data]
• Atomic write operations
• Configurable sync intervals

Segment Management

• Log file segmentation
• Configurable segment size
• Automatic cleanup & retention
• Efficient storage utilization

Index Engine

• Optional message indexing
• Fast random access
• Configurable index intervals
• Memory-efficient design

Queue Management

// Go Component Structure
type QueueManager struct {
    topics    map[string]*Topic
    consumers map[string]*ConsumerCatalog
    storage   *StorageEngine
    config    *Config
    mu        sync.RWMutex
}

type Topic struct {
    name      string
    log       *AppendOnlyLog
    index     *IndexEngine
    segments  *SegmentManager
    stats     *TopicStats
}

type ConsumerCatalog struct {
    offsets   map[string]int64
    lastSeen  map[string]time.Time
    mu        sync.RWMutex
}

type StorageEngine struct {
    dataDir   string
    segSize   int64
    syncInt   time.Duration
    retention time.Duration
}

Network Protocol & Communication

RESP Protocol Implementation

Command Structure

• Redis-like protocol (RESP)
• Text-based command format
• Simple request-response model
• Error handling & status codes

Connection Management

• TCP socket server
• Concurrent client handling
• Connection pooling
• Graceful shutdown

Command Processing Flow

Loading diagram...

Storage Engine Design

Append-Only Log Architecture

Log File Format

• 8-byte timestamp (Unix nanoseconds)
• 4-byte message size (big-endian)
• Variable-length message data
• Sequential write optimization

Segmentation Strategy

• Configurable segment size (1MB default)
• Automatic segment rotation
• Time-based retention policy
• Efficient cleanup operations

Directory Structure & File Organization

// Storage Directory Structure
data/
└── topics/
    └── <topic-name>/
        ├── 000000000000000000.log  # Log segment files
        ├── 000000000000000001.log  # (sequential numbering)
        ├── 000000000000000002.log
        ├── index.idx               # Index file (optional)
        └── consumers.json          # Consumer offset catalog

// Message Format in Log Files
type LogEntry struct {
    Timestamp int64  // 8 bytes - Unix nanoseconds
    Size      uint32 // 4 bytes - Message size
    Data      []byte // Variable - Message payload
}

// Consumer Catalog Format (JSON)
type ConsumerCatalog struct {
    Consumers map[string]ConsumerInfo `json:"consumers"`
}

type ConsumerInfo struct {
    Offset   int64     `json:"offset"`
    LastSeen time.Time `json:"last_seen"`
}

Performance Optimizations & Design Patterns

Write Performance

Sequential Writes

• Append-only log design
• No random access writes
• OS page cache optimization
• Minimal disk seeks

Batching & Buffering

• Configurable sync intervals
• Write batching for throughput
• Memory-mapped files
• Async I/O operations

Concurrency Design

// Concurrency Patterns
type SafeQueue struct {
    topics map[string]*Topic
    mu     sync.RWMutex  // Reader-writer lock
}

// Lock-free read operations
func (q *SafeQueue) GetTopic(name string) *Topic {
    q.mu.RLock()
    defer q.mu.RUnlock()
    return q.topics[name]
}

// Write operations with exclusive lock
func (q *SafeQueue) CreateTopic(name string) error {
    q.mu.Lock()
    defer q.mu.Unlock()
    // Topic creation logic
}

// Per-topic locks for fine-grained concurrency
type Topic struct {
    log *AppendOnlyLog
    mu  sync.Mutex  // Per-topic synchronization
}

// Atomic operations for consumer offsets
type ConsumerCatalog struct {
    offsets sync.Map  // Concurrent map for offsets
}

Client Architecture & SDK Design

Multi-Language Client Support

Go CLI Client

• Interactive mode
• Command-line interface
• Connection management
• Error handling

TypeScript Client

• Promise-based API
• Type definitions
• Connection pooling
• Async/await support

Protocol Abstraction

• RESP protocol handling
• Command serialization
• Response parsing
• Error code mapping

TypeScript Client Architecture

// TypeScript Client Design
interface YuclaseClient {
  connect(): Promise<void>
  disconnect(): Promise<void>
  ping(message?: string): Promise<string>
  createTopic(topic: string): Promise<void>
  enqueue(topic: string, message: string): Promise<number>
  listen(topic: string, consumerId: string): Promise<string[]>
  getOffset(topic: string, consumerId: string): Promise<number>
  setOffset(topic: string, consumerId: string, offset: number): Promise<void>
  listTopics(): Promise<string[]>
  getStats(topic?: string): Promise<QueueStats>
}

class SimpleYuclaseClient implements YuclaseClient {
  private socket: net.Socket
  private config: ClientConfig
  private connected: boolean = false
  
  constructor(config: ClientConfig) {
    this.config = config
    this.socket = new net.Socket()
  }
  
  async connect(): Promise<void> {
    return new Promise((resolve, reject) => {
      this.socket.connect(this.config.port, this.config.host, () => {
        this.connected = true
        resolve()
      })
      this.socket.on('error', reject)
    })
  }
}

Configuration & Deployment Architecture

Configuration Management

// config.yaml structure
server:
  host: "localhost"
  port: 8080
  max_connections: 1000
  read_timeout: "30s"
  write_timeout: "30s"

storage:
  data_directory: "./data"
  log_segment_size: 1048576  # 1MB
  index_interval: 1000
  sync_interval: "1s"
  retention_duration: "168h"  # 7 days
  compression: false

logging:
  level: "info"
  file: "./logs/yuclase.log"
  max_size: "100MB"
  max_backups: 5
  max_age: 30

performance:
  buffer_size: 4096
  worker_pool_size: 10
  gc_interval: "5m"

Deployment Patterns

Single Instance: Simple deployment for development and small workloads
Docker Container: Containerized deployment with volume mounts for persistence
Kubernetes: Scalable deployment with persistent volumes and service discovery
Load Balanced: Multiple instances behind a load balancer for high availability

← Back to Yuclase Project