37signals leaves the cloud for Deft's data centers, saving millions per year

37signals — the company that makes Basecamp and HEY — is known for setting trends in the business and tech worlds. One of its co-founders, David Heinemeier Hansson (DHH), wrote the application framework a lot of the web runs on. He and the other co-founder, Jason Fried, literally wrote the books on managing a remote workforce.

So when DHH announced the company was leaving the cloud, people paid attention. They paid attention because 37signals is a big company providing services that a lot of other companies rely on. They paid attention because the founders of 37signals have been known for calling it early. And they paid attention because their own cloud bills weren’t looking like what they had been promised, either.

Update: Hear how much 37signals expects to save over the next five years.   

People should also pay attention to all the reasons 37signals laid out for leaving the public cloud in favor of the data center. This wasn’t a reflexive retreat to the way things were. It was a forward-thinking move to infrastructure that will support their business today and far into the future.

As 37signals’ data center provider since 2010, Deft has had the benefit of learning from and supporting the 37signals team for well over a decade. We spent some time with Eron Nicholson, Director of Operations at 37signals, to get a deeper look into what the company is up to. Eron manages the company’s team of site reliability engineers. His team is responsible for ensuring Basecamp and HEY are always on and meeting customers’ expectations.

37signals’ cloud infrastructure was always hybrid

Like many things in the cloud, it helps to start with a metaphor. In 37signals’ telling, the cloud is to tech companies what excavators are to construction companies. If you only need an excavator for a few jobs a year, you should rent one. If you need to use one every single day, then it makes more sense to buy your own.

Same thing goes for the cloud. When spread across an entire year, the cost of borrowing public cloud resources is often more expensive than owning your own.

This was the logic that 37signals used to approach its infrastructure decisions. After trying different colocation partners, the team tapped Deft to build a private cloud for all of the instances where owning made sense. Then, we set up secure connectivity to the necessary public clouds for extra burst capacity they may not need long term.

Basecamp, the team’s long-running project management software, was already living in bare metal servers. It has a high volume of users with relatively stable usage needs. Demands do grow over time, and with a solid handle on the predictability of their business, Deft and 37signals have time to assess and adjust before additional capacity is needed. Recently, Deft and 37signals’ teams overhauled the bare-metal infrastructure, setting up a new, high-powered environment architected to last for the next 10 years.

A decade of stable growth, and then a hockey stick — in both users and cloud costs

For new products, it’s hard to know what’s going to take root and grow and what will teach us lessons before fading away. When the company went to launch its new email product, HEY, it relied on the cloud to serve as a testing ground. It wasn’t expecting massive overnight success — most of its applications grew gradually. But the launch of HEY in 2020 hit at a time when more people than ever were trapped in their homes and tied to their inboxes. Customers signed up in waves, maxing out resources as hundreds of thousands of people jumped on board.

“The public cloud was a great place for overnight success to happen. But it wasn’t sustainable. The sharp peaks in growth made it clear that we had a potential hit on our hands, and that cloud hosting was not going to be tenable long-term.”

Eron Nicholson
Director of Operations at 37signals

While the team didn’t expect so much attention at launch, it did expect the one thing anyone in tech can rely on: change. As the 37signals team was building for AWS, it was also keeping an eye on what future infrastructure might look like. “We’ve learned that these things aren’t always forever,” says Eron about 37signals’ infrastructure. Having made last-minute moves onto and off of public cloud before (and dealt with the pain of rearchitecting to accomplish it), the 37signals team now builds pretty much everything with an exit plan.

HEY was built on AWS but was never dependent on AWS-native functionality. Sure, unwinding from the cloud will take effort. The 37signals team still needs to figure out the best way to run cloud apps on its own servers and find a failover solution as elegant as what they built into AWS. But these are discrete challenges with available solutions. There’s no major rethinking — or rebuilding — of the app itself because the team runs on a simple philosophy:

“No decision is forever,” says Eron. “We can always decide to reevaluate and do things better.”

What a fully managed infrastructure provider enables for a medium-sized business

Mural at 37signals’ headquarters (Source)

When DHH was talking about leaving the cloud to be self-sufficient on 37signals’ own servers, he was never talking about going it alone. The reason the 37signals team can do so much with such a small team — including launching new products like HEY — is because they’re not bogged down actually managing bare metal. That’s not their job. Their job is to run the company. Deft’s job is to make sure the infrastructure it runs on is always available.

When 37signals was hit with a massive DDoS attack seven years ago, Deft was there to help the team through it — not just advising, but doing the work along with them. Other companies caught up in the same attack weren’t so lucky.

A solid partnership isn’t just beneficial during dramatic moments. Deft’s support through the boring, day-to-day stuff helps keep 37signals’ team focused. As DHH himself pointed out, so many people hesitate to move from public cloud to owned hardware, thinking that they’ll have to have people nearby to fiddle with it all the time.

(Source)

A good infrastructure partner will have a remote hands service available 24/7/365 to make physical changes to your physical infrastructure so you don’t have to. At 37signals, the team works on the business, not the hardware, and they do it from anywhere in the world they please — a prerequisite for a company that also wrote the book on remote working.

How 37signals thinks about future-proof infrastructure and future-proof products

When you operate with the understanding that nothing is permanent and everything is up for reevaluation at any moment, suddenly future-proofing isn’t something you have to plan for. It’s just … how you work.

Moving totally out of the cloud will take Eron and his team the better part of the next year or two, even with all their pre-planning and flexible architecture. When it happens, though, it will give them many things they don’t have now: more predictable bills, total control over the infrastructure, and maybe most importantly, less mental load. After almost 20 years of running Basecamp, plus the addition of HEY and a handful of other products throughout the years, Eron and his team have to keep tabs on too many environments.

“We have a lot of technology spread. Basecamp 4 is running on bare metal, we have apps running in the cloud right now, we have some running in VMs,” he says. “I want us to streamline those … to run as much as we can in the same way so the cognitive load on my team is just a little bit lower.”

By reducing the technologies, the team will be in a better position to expand the product offerings. At least one new tool is already in the works, and it’s being built entirely in the data center. If it experiences the same spike in interest that HEY had on launch, the team is prepared to deal with it, without the help of the cloud.

At 37signals, “It’s not quite the move fast and break stuff style of work,” Eron says. Instead, the team focuses on incremental changes. Experiment, sure — but make it small enough that all the apps customers depend on will stay running.

With that ethos, owned hardware is the natural choice for Basecamp. It may literally be more hands-on than a public cloud, but it gives the team all the stability and control they need to always — however subtly — make things better.