DevOps & SREFeatured Case Study

Chronos: Distributed Cron & Workflow Orchestration

Fault-tolerant cron scheduler executing over 1.2M automated batch jobs daily with exactly-once execution guarantees.

The Engineering Challenge

Problem & Root Cause

Periodic batch tasks were handled by server-local crontabs that failed silently during AWS instance reboots, causing missed data warehouse aggregations and duplicate billing triggers.

Architectural Solution

System Architecture Strategy

Constructed a leader-elected scheduler daemon using Etcd leases, distributed lock fencing tokens, and automated heartbeat retries across worker pools.

Core System Implementation

Detailed Architecture Overview

Etcd election assigns master duties for dispatching job tasks into priority queues. Worker nodes acquire fencing tokens before executing jobs to avoid split-brain execution.

Critical Technical Highlights:

Leader election and auto-failover via Etcd v3 distributed leases

Fencing token validation preventing zombie workers from modifying state

Dynamic cron expression parser with timezone-aware execution offsets

Dead-letter queues with exponential backoff and Slack alerting webhooks

Personal Engineering Contribution:

Designed fault-tolerance semantics, authored worker execution agent, and implemented web-based execution graph monitor.

Verified Production Metrics

100% SLA adherence over 14 consecutive months
Processed 1.2M jobs daily with 0 duplicate executions
Cut incident triage time from 2 hours to under 3 minutes with unified job tracing

Edge Cases & Lessons Learned

Network partitions could allow stale leaders to dispatch work. Solving this required monotonic fencing tokens validated by recipient services.

Applied Technologies & Tooling

Go / Golang(Backend)

Etcd v3(Infrastructure)

PostgreSQL(Database)

Redis(Cache)

Prometheus(Observability)

Terraform(DevOps)

Explore More System Blueprints

Cloud Architecture

Nexus Stream: High-Throughput Event Mesh

Distributed real-time event broker processing 180k msgs/sec with sub-5ms p99 latency across multi-region Kubernetes clusters.

View Case Study

Full-Stack System

Aurora Core: Double-Entry Immutable Ledger

Financial transaction engine supporting strict ACID guarantees, cryptographic audit trails, and multi-currency exchange balancing.

View Case Study