Houston

Houston is a tool for orchestrating data pipelines and microservices. It lets you define and run complex workflows, called 'missions', that coordinate different tasks and services. Instead of relying on a heavy central server, Houston uses a lightweight API and a pub/sub messaging system to manage these workflows. This means your actual task code can run on serverless platforms or anywhere else, making the entire system more flexible and resilient.

What sets Houston apart is its focus on being simpler and cheaper than alternatives like Airflow. Its serverless approach can cut costs significantly, and because the central coordinator isn't under heavy load, it can handle hundreds of concurrent missions. You can write task runners in any programming language, and multiple teams can share the same runners, which encourages collaboration and avoids vendor lock-in. Setting it up is straightforward, with a quick installation process and a demo to get you running in minutes.

This tool is a solid fit for data engineers and development teams who need a scalable, cost-effective way to manage data workflows. It's particularly useful for organizations with large data teams looking for a scalable service mesh, teams that need to get code into production rapidly, and anyone who values the flexibility to connect services without being tied to a specific cloud provider's ecosystem.