Quick Start

This guide will help you understand Kasho’s architecture and get it running in your environment. While we use Docker Compose as an example, the concepts apply to any production orchestration system like Kubernetes.

Production Deployment Overview

Kasho consists of three required containers that work together:

1. Redis - Used for distributed state management and caching
2. Change Stream Service - Captures changes from your source database (pg-change-stream for PostgreSQL, mysql-change-stream for MySQL)
3. translicator - Applies transformations and writes to your target database

Container Images

All Kasho components are packaged in a single Docker image available from Docker Hub:

• Image: kashoio/kasho:latest (or specific version like v0.3.0)
• Commands: ./pg-change-stream, ./mysql-change-stream, and ./translicator

Prerequisites

Before you begin, ensure you have:

• A Redis instance (or Redis-compatible service)
• Container orchestration system (Kubernetes, ECS, Docker, etc.)

[mysqld]
server-id = 1
log_bin = mysql-bin
binlog_row_image = FULL
gtid_mode = ON
enforce_gtid_consistency = ON

Docker Compose Example

Step 1: Example Directory Structure

For this example, create a directory structure:

mkdir kasho-example
cd kasho-example

Step 2: Example Docker Compose Configuration

Create a docker-compose.yml file:

services:
  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    volumes:
      - redis-data:/data
 
  pg-change-stream:
    image: kashoio/kasho:latest
    command: /app/bin/pg-change-stream
    environment:
      KV_URL: redis://redis:6379
      PRIMARY_DATABASE_URL: ${PRIMARY_DATABASE_URL}
    ports:
      - "50051:50051"
    depends_on:
      - redis
 
  translicator:
    image: kashoio/kasho:latest
    command: /app/bin/translicator
    environment:
      CHANGE_STREAM_SERVICE_ADDR: pg-change-stream:50051
      REPLICA_DATABASE_URL: ${REPLICA_DATABASE_URL}
    volumes:
      - ./config:/app/config:ro # Contains transforms.yml
    depends_on:
      - pg-change-stream
 
volumes:
  redis-data:

services:
  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    volumes:
      - redis-data:/data
 
  mysql-change-stream:
    image: kashoio/kasho:latest
    command: /app/bin/mysql-change-stream
    environment:
      KV_URL: redis://redis:6379
      PRIMARY_DATABASE_URL: ${PRIMARY_DATABASE_URL}
    ports:
      - "50051:50051"
    depends_on:
      - redis
 
  translicator:
    image: kashoio/kasho:latest
    command: /app/bin/translicator
    environment:
      CHANGE_STREAM_SERVICE_ADDR: mysql-change-stream:50051
      REPLICA_DATABASE_URL: ${REPLICA_DATABASE_URL}
    volumes:
      - ./config:/app/config:ro # Contains transforms.yml
    depends_on:
      - mysql-change-stream
 
volumes:
  redis-data:

Step 3: Configuration Files

Create a config directory for your transforms:

mkdir config

Create config/transforms.yml with your transformation rules:

version: v1
tables:
  # Example: Transform user emails in the public.users table
  public.users:
    email: FakeEmail
    phone: FakePhone

See the Transform Configuration guide for all available transforms.

Step 4: Environment Variables

Create a .env file with your database connections and service configuration.

# Source database (read-only access needed)
# Replace with your actual source database connection string
PRIMARY_DATABASE_URL=postgresql://kasho:password@source-host:5432/source_db?sslmode=disable
 
# Target database (write access needed)
# Replace with your actual target database connection string
REPLICA_DATABASE_URL=postgresql://kasho:password@target-host:5432/target_db?sslmode=disable
 
# Redis connection
KV_URL=redis://redis:6379
 
# Change stream service for translicator
CHANGE_STREAM_SERVICE_ADDR=pg-change-stream:50051

# Source database (read-only access needed)
# Replace with your actual source database connection string
PRIMARY_DATABASE_URL=mysql://kasho:password@source-host:3306/source_db?tls=false
 
# Target database (write access needed)
# Replace with your actual target database connection string
REPLICA_DATABASE_URL=mysql://kasho:password@target-host:3306/target_db?tls=false
 
# Redis connection
KV_URL=redis://redis:6379
 
# Change stream service for translicator
CHANGE_STREAM_SERVICE_ADDR=mysql-change-stream:50051

Step 5: Run the Example

docker-compose up -d

Verify all services are running:

docker-compose ps

You should see:

• redis - Running
• pg-change-stream or mysql-change-stream - Running on port 50051
• translicator - Running

Production Considerations

When deploying to production:

1. Container Orchestration: Translate the Docker Compose configuration to your platform:

   • Kubernetes: Create Deployments, Services, and ConfigMaps
   • ECS: Define Task Definitions and Services
   • Other platforms: Follow your standard container deployment practices

2. Redis: Use a managed Redis service or ensure high availability

3. Networking: Ensure containers can communicate:

   • translicator must reach pg-change-stream service
   • Both services need access to their respective databases
   • Consider service discovery mechanisms in your environment

4. Configuration Management: Mount transform configurations using your platform’s methods:

   • Kubernetes: ConfigMaps or Secrets
   • ECS: Parameter Store or S3
   • Docker: Volume mounts or config management tools

Bootstrap Existing Data

If you have existing data in your source database, you’ll need to run the bootstrap process:

# Run bootstrap process (will prompt for confirmation)
docker exec -it kasho-example-pg-change-stream-1 /app/scripts/bootstrap-kasho-pg.sh

# Run bootstrap process (will prompt for confirmation)
docker exec -it kasho-example-mysql-change-stream-1 /app/scripts/bootstrap-kasho-mysql.sh

2. The bootstrap process will:
   • Create a consistent snapshot of your source database
   • Start capturing changes from that point
   • Load the snapshot data
   • Transition to streaming mode

If you prefer, you can manually bootstrap things with a bit more effort. See the Bootstrap Process guide for detailed information.

Verify Installation

Check that replication is working:

1. Check change-stream service status (using gRPC):

   # Using grpcurl (install with: brew install grpcurl)
   grpcurl -plaintext localhost:50051 change_stream.ChangeStream/GetStatus

2. View logs:

docker-compose logs -f pg-change-stream
docker-compose logs -f translicator

docker-compose logs -f mysql-change-stream
docker-compose logs -f translicator

3. Make a test change in your source database:

   -- In your source database
   UPDATE users SET updated_at = NOW() WHERE id = 1;

4. Verify the change appears in your target database

Next Steps

• Configure your databases for production use
• Understand configuration options
• Learn about transforms

Troubleshooting

Services won’t start:

• Ensure your environment variables are set correctly, especially the database URLs.
• Verify transforms.yml exists in the config directory
• Check logs: docker-compose logs [service-name]

Connection errors:

• Ensure databases are accessible from Docker containers
• For PostgreSQL: Verify wal_level = logical
• For MySQL: Verify binary logging is enabled with SHOW VARIABLES LIKE 'log_bin'
• Check firewall rules allow connections