Software Engineer2009-2010
Broadhop logo

Broadhop

MongoDB in Production Before DevOps Had a Name

Debugging bleeding-edge technology directly with the founders—and beating Oracle

20M
Active Mobile Sessions
28K
Transactions/Second
v1.4
MongoDB (Barely Beta)
Beat
Oracle RAC

Before the Patterns Had Names

In late 2009, I was at Broadhop in Denver, deploying the Quantum Network Suite for international telecoms like Saudi Telecom. Quantum ran on MongoDB—which was still in beta. This was before "DevOps" was a common term, before infrastructure as code was standard practice, before anyone had written the playbooks we all follow now.

MongoDB version 1.4's high availability was, to put it charitably, "completely broken." When v1.5 introduced replica sets, Saudi Telecom asked me why the logs said "1.5 for beta use only." My response? "Just ignore that." We were shipping beta software to production for international telecoms because that's what the business needed.

What I didn't realize at the time was that I was about to start debugging production issues directly with the co-founder and CTO of the database we were betting the company on.

Part 1: The Guy on Google Groups

When you're running bleeding-edge technology in production, you don't have Stack Overflow answers or well-documented runbooks. You have Google Groups and the hope that someone who knows the code will respond.

I started reporting bugs and asking questions about multi-master replication issues. Some guy named "Eliot" kept responding—within minutes, not hours. He'd suggest patches, point me to new builds, walk through the replication logic with me.

// Google Groups - April 9, 2010
Brian: "the java driver we are using apparently doesn't like this because it can't tell which mongo server is the master"
— 11 minutes later —
Eliot: "Not to change the subject, but the repl pair change went in 1.5.0 Do you want to try it?"

Eliot Horowitz was the co-founder and CTO of 10gen—the company that built MongoDB. I didn't know that at the time. I was too busy shipping to notice. To me, he was just the most helpful person on the internet, someone who clearly understood the codebase intimately.

Part 2: Beating Oracle

Broadhop was competing against solutions built on Oracle RAC—the enterprise standard for high-availability databases. Our competitors had decades of Oracle expertise, proven deployment patterns, and enterprise sales relationships.

We had a beta database and a willingness to work directly with the people building it.

Production Results

20 Million
Active mobile user sessions
28,000 TPS
Transactions per second

We won deals against Oracle RAC. Not because MongoDB was more mature—it wasn't. But because we could move faster, iterate on problems in real-time, and had a direct line to the people who could fix issues at the source.

Part 3: The Pattern Before the Name

Looking back, this engagement embodied patterns that wouldn't have names for years. We were doing "shift left"—bringing operational concerns into the development process. We were practicing "you build it, you run it"—I was both deploying and supporting the infrastructure. We were doing "continuous feedback"—shipping to production and iterating based on real-world behavior.

But we didn't call it any of those things. We called it "getting the job done."

Patterns We Were Using (Before They Had Names)

Shift Left

Operational concerns in development

You Build It, You Run It

Full ownership of the stack

Continuous Feedback

Ship fast, learn from production

Inner Source

Direct collaboration with tool creators

This was my first taste of what I'd later understand as the DevOps mindset: technology decisions aren't just technical—they're about feedback loops, relationships, and the ability to solve problems faster than your competitors.

Key Lessons

Bet on Velocity

Mature technology with slow feedback loops loses to immature technology with fast ones.

Know the Builders

Direct access to the people who build your tools is worth more than any support contract.

Ship First, Name Later

The patterns that matter emerge from doing the work, not from reading about best practices.

The EANTC Benchmark

I was on site with EANTC running the official performance benchmarks. The results validated what we'd built—enterprise-grade performance on technology the industry said wasn't ready for production.

Broadhop EANTC Performance Benchmark Results

Further Reading

Evaluating Emerging Technology?

Sometimes the right choice isn't the mature one—it's the one with the fastest feedback loop. Let's discuss how to make that call.

Schedule a Conversation