A Conversation With Tom Mairs and Mike Hillyer of KumoMTA

Ever make a connecting flight through Atlanta - Hartsfield? While it’s the busiest airport in the US, moving between flights is easier than in most airports. There’s a fascinating analysis about why Hartsfield works well, based on the (proposed) constructal law of physics.

That’s not what this episode of the Future Of Email is about, but it’s close!

An email MTA – Message Transfer Agent — does some of the same jobs as an airport. MTAs hand off email messages — to mail delivery agents (think Gmail), or possibly to other MTAs in between. At scale, a busy MTA could be juggling a million email messages. Like a million cranky fliers at Hartsfield, each message could have its own complications. Flight delays = slow domain lookup. Address not found = lost my ticket. You get the idea.

KumoMTA is a new open-source MTA, created by 3 founders with incredibly deep experience in the world of email. Two of the founders, Tom Mairs and Mike Hillyer, joined host Matthew Dunn for a Future Of Email conversation about MTAs, open-source, founding companies, creating niches, finding traction, the virtues of Rust and scriptability, and quite a few more topics. Actually, the conversation was as busy as Hartsfield on a Friday!

It’s easy to take email for granted, whether you just get too much of it or you’re professionally responsible for that. This is a great conversational glimpse into the sheer complexity involved in making a Hartsfield-for-email work well, at scale — and at the surprisingly calm, smart business model to releasing the code that does that job for free, as open-source software. A super (and technically deep) conversation — stay tuned for a followup in a year!

TRANSCRIPT

A Conversation With Tom Mairs and Mike Hillyer of KumoMTA

Matthew Dunn: Good morning. This is Dr. Matthew Dunn, host of the Future of Email. My guests, plural, today, two gents, uh, heavily involved in the Kumo MTA project, Tom Mares and Mike Hillier. Welcome, guys. Thank you. Nice to be here. And we say MTA and 99 percent of the world hears, like the adults in Charlie Brown. What's an MTA, Tom?

Tom Mairs: Yeah, it's funny if you do a search for it, most of the time you'll find out, you know, Metro Transit System or something like, you know, it makes SEO very difficult. Yeah. Yeah. Uh, but it's a an M. C. A. Is an on premise mail transfer agent takes mail from one place and sends it to another. And in our case, it takes it from one or two places and send it to billions of other places.

Matthew Dunn: And why, why on [00:01:00] premise in your qualification?

Tom Mairs: Good question. There are, there are really two different ways to get mail from, from its source to, to inboxes, uh, one is to use an on premise MTA, which is, uh, a physical or a cloud based server that you own. You download the instant software, you install it and you manage it.

The other way is to do it with a cloud service where somebody else has done that and you're just using their service. Yeah, exactly. And somebody, so an MTA is still an MTA, whether you're using it in a cloud. Or, uh, in your own system. Okay. But an MTA takes it from one place to another. We specialize in that on premise, you own it to software, you, you own it, you deploy it, you manage it, and you're responsible for it.

Mike Hillyer: Oh,

Matthew Dunn: okay. Okay. And, and Kumo specifically, open source. Absolutely. As I said, it was one of the things that, that really caught my attention. Now, just for background, both of you guys have like, Terrifically long history in the email space. X acquired by MessageBird in [00:02:00] your case, Tom?

Tom Mairs: Yeah, actually, uh, I've been working in that space for, I had been working in that space for about 15, 16 years.

Mike actually pulled me into that world, uh, out of, out of, uh, I was a, um, I was a self employed consultant for, for a very long time doing Uh, exchange, um, and, uh, and send mail type implementations for, for companies with the inbound. Uh, working on the outbound was a new experience, uh, 15 years ago. Yeah, and then Mike's

Mike Hillyer: fault.

Socket labs and what else? So I, uh, I go way back. So I actually started out in open source, so I did you, okay. Um, actually first job I ever had outta school beyond the one that I met. So I met Tom at a previous job, and then I went off to work in open source. So I worked for ISQL, which is an open source database company.

That one eventually got acquired by Sun who got acquired by Oracle. Oracle, yeah. Um, just before it got acquired by sa I was recruited over to what at the time was message [00:03:00] systems. Okay. Because. Message systems. The founders were very active in open source and I met them because we were both speaking at open source conferences at OSCON and some other, some other events and, and in meeting with them, they said, Hey, you know, you're really good at explaining things in your conference sessions.

Could you, um, could you come and do that for, for our, um, other, uh, you know, for our other. Customers, right? For people who are trying to use the software for email. And I was able to say, sure, let's, let's give that a go. And so I started off, you know, in email in 2006, working for message systems that eventually became SparkPost that eventually got acquired by MessageBird.

Although I actually left before the MessageBird acquisition, I always leave just before the big acquisition. That's my, that's my system. So you think

Matthew Dunn: I wasn't paying attention, but I'm going to hook back and say, did you ever end up getting calls from lawyers when there was the whole Oracle Sun. What's going to happen?

Mike Hillyer: No, I was, I was, I was asked at that point. [00:04:00] So like I said, I left, I left MySQL just before they got acquired by Sun. And then I left MessageBird just before I left SparkPost just before they got acquired by MessageBird. So I like to just get out right before the acquisitions. But, um, But with that, then I, I was there till 2018, went off and started a consulting company called Email Ninjas, worked with Socketlabs for a bit, and one of the things I, I saw, especially in the last couple of years, as I would, as I would talk to different ESPs, I, I've worked with most of them at this point, just because.

A lot of them use message systems, SparkPost infrastructure. And, uh, and as I would talk to them about, you know, what they were looking at in the future with on prem, the one answer I kept getting back was, you know, we're, we're kind of tired of the vendor relationship. And, and with that, a lot of them were building their own MTA.

They were like, you know what, we're just going to, we're going to make our own going forward, which is a very ambitious goal, but I've seen, I've seen multiple companies do it. And, but that became very clear to me. They were [00:05:00] looking for something other than the commercially licensed MTA with its changes and fees and terms and everything else.

And the reality is, is that, I mean, this is the future of email, but the history of email is actually open source. Most...

Matthew Dunn: I took the words out of

Mike Hillyer: my mouth. Yeah, absolutely. They started out in Postfix. They started out using Sendmail, something along those lines. And then eventually... When their volumes got high, that's when they moved to the original open source app or the original commercial options, which was back in the day, there was iron port, um, then there was, um, message systems momentum.

There was port 25 power MTA. They went to those options because open source was never built for, for sending at scale. It was built for, for smaller senders. And you'd go into some of these companies and they'd have like, I'd go into a place that had like a whole rack of postfix servers running and they had ingenious little scripts that would say, Hey.

Postfix can only send for one user at a time. So we're going to spin this one up as customer five, send their, their campaign, spin it down, turn it into customer 13, [00:06:00] send customer 13. Right? Like they, they do all of that because they didn't have the multi tenant, the high volume, whatever. We went into that place to replace it with two servers of momentum back in the day, right, like from, from a whole rack to two machines because open source wasn't doing what they needed.

So that's, that's where my history has been is first of all, working in open source and databases, then placing a ton of it in email and now, and now hopefully bringing open source back to large scale centers.

Matthew Dunn: What would I be describing the challenge that MTA has to tackle at scale? Somewhat accurate. If I said, look.

When you start going to millions of messages, billions of messages, in some case, out the door, this sort of completely unpredictable handshake, hi, I've got this for you, got this for you. It's just a different engineering problem than sending one email message, right? And, and all of the, you know, monitoring, maintenance, performance, like to make all of [00:07:00] that work, that's a non trivial task.

That's why my eyebrows went up and you said, right there on MTA, like, whoa, not easy. What am I missing in that equation?

Tom Mairs: It is a very complicated situation. You know, the understanding how email works is, It takes a little bit doing, doing, um, a, a single message. You know, you can follow that conversation when you start sending hundreds of thousands of messages.

That's also relatively easy to juggle when you get into the millions of messages or billions. Yeah. You know, there, there are people out there sending billions of messages every single day. Sure, sure. You're starting to bend the laws of physics at , so, so you have to get really crafty about how you manage things like.

garbage collection and caches and, and that stuff becomes really, really important, uh, when you're going that fast. Uh, so when you, it's really interesting to see people, I'm just, I'm just going to build my own MTA. Well, yeah, you understand the [00:08:00] concepts, but you don't really understand what happens when you're, when you send it at light speed, when you start going really, really, really fast.

Really strange things happen. So you have to look at things like, can you parallelize those things? Can you make things happen in a different concept? Not just having, you know, one of the big major downfalls of of some of those other systems is a single queue. So, for instance, postfix, you know, you've got a single queue, you can put mail into it and you have to wait for that mail to go before you can send the next set of mail.

Yeah. One of the things that we've done is we've fanned that all out. So you don't just have one queue, a single campaign might end up having 50 or a hundred or a thousand parallel queues that all split out the mail in a fig fan, which is an interesting trick, but that actually adds more complication to the

Mike Hillyer: engineering part of it.

Wow. Okay.

Matthew Dunn: So I'm going to spin up a metaphor. Maybe it'll be useful in this conversation. Maybe not, because I think we're going to get technical, but, um, the, I'm going [00:09:00] to build my own MTN. I'm not knocking someone who wants to tackle that, but That's like, I'm gonna, I'm gonna build a, I'm gonna build a jalopy out in my garage.

You guys are dealing with traffic engineering for the interstate system, which is a different problem than the car. Yeah.

Mike Hillyer: Yeah. Yeah. Yeah. I mean, one of the, one of the, and it's interesting because this is not the first modern architecture, open source MTA that's, that's being developed. There's actually five or six really popular ones right now.

Yeah. But most of those projects and including actually a lot of MTAs that are built in general, they usually start out as a receiving tool. And so when you do receiving and you think about it, if I'm a receiving MTA, what I need to be able to do is accept a lot of connections from a lot of different places and be able to make decisions based on each connection, each message.

Did this come from a trusted IP range? Does this message trigger any filters? All of that has to happen. In parallel, as far as the receiving side goes, and then once. They're vetted and they're let in, [00:10:00] they are put into a single queue because most MTAs, especially receiving side, just drop that message then into the, the mail server for IMAP for, for whatever you use exchange, whatever that may be, where it then gets, you know, looked at by the end user.

And I know that that machine outside of it failing. We'll accept every message I hand it as quick as I can hand it to it, right? So I don't need a queue to pass onward because my one queue is just there in case that machine goes down. And if it's down, then I'm not trying anything, and as soon as it's back up again, I just dump it all into the IMAP box.

Right into the mailboxer. That's it. That's a, that's a sending MTA. That's why even though there's been several popular, um, open source MTA projects, they're still not seeing any use by email service providers or large senders because that, that's not the problem they're trying to solve. Um, when it, when it comes to queue management, it really is the make or break of a high volume service, you know, for sending.

And you think about it like, like an airport, right? If you had an [00:11:00] airport, if you were running, I live in Atlanta, if you were running Hartsfield and it had one x ray machine, one queue to stand in in order to get your mail checked, it would be a nightmare. It's already not exactly great getting in line for security at Hartsfield as it is, but what if there was only one x ray machine, right?

What we have instead is the ability to say, you know, we don't just have a line, and it's like airlines, we don't just have... A delta line. We don't just have a line specifically for, you know, whether it's, it's going to this place, that place, we have the equivalent of saying. We actually have for every airline for every city they go to, we have a separate security line.

So if you're on Delta to Atlanta or you're in Atlanta, if you're on Delta to Orlando, that's one security line. If you're on Delta to New York, that's a different security line. If you're on American Airlines to Orlando, that's a separate security line. And that means that no two passengers. Going to Orlando through those two airlines are standing in the same line.

If, if American airline [00:12:00] slows down, the Delta passengers can still get through the line to Orlando. If Delta slowed down to New York, that doesn't affect Delta to Orlando, right? That kind of queue management is what makes a solution like this effective at, at extreme scale is because we never queue. Two messages to the same destination from the same channel into the same queue.

So slowdowns never become a problem as far as that goes.

Matthew Dunn: So you, you, unlike a receiving MTA, if I'm understanding what you just explained correctly, you've sort of got a multiplex queue problem, right? Queues in as well as queues out that you've, you've figured out how to manage. Wow. That's, uh, That's hardcore.

Mike Hillyer: Yeah. Cues in isn't actually the thing. Cues in is actually about being able to do, um, a large number of parallel connections across a large number of threads, uh, all running on separate schedulers, that's what a good receiving MTA needs is very multi threaded connection handling on the inbound. And then an outgoing MTA [00:13:00] still needs that because there's a lot to manage, but it's the cues that make it on the outbound.

So, so a good sending server has to be capable of both.

Matthew Dunn: Right. Okay. Okay. That makes sense. Even though it'd be very hard to draw, uh, draw on screen, um, slightly different question, but entirely related. How did Kumo come together first? Like you've got multiple people with a heck of a lot of experience involved.

This didn't just happen over, you know, one frosty malt beverage. How did this, how did this start coming together? Oh,

Mike Hillyer: this is the virtual age. It happened over no beverages at all. It was all online. So this comes back. Um, actually, it's one of those, it's one of those things where, you know, because of what the industry looked like last year, and especially what happened in the industry.

So I started out, um, First of all, I've worked with the two co founders. So, so like, like Tom said, I brought him in at message systems because I wanted an additional person to help me out. And I knew Tom was the guy. And then, um, and then Wes, our [00:14:00] engineering co founder was the chief architect of the Momentum MTA product at message systems when I joined.

So I knew Wes from the very start of my career. And then I think it was two years later, I brought Tom in and then Tom knew as, as well. So that's where we all knew each other from. And here's the thing. When you start something, when you're older, you get an advantage that like, you know, you hear all these stories about startups and they're all 20 something and, and they've got.

Incredible creativity and energy. And they come up with things that you just would never have imagined, but you're really, when you start something up in your middle age, you're on like easy mode, you get certain cheat codes. And one of them is I don't have to like, you'll see all these things on LinkedIn, how to find a good folk co founder.

You need this and this and this and a good co founder, and you better get it right early on because have these agreements in place, or you wait until you're 45 and. You know, all the best people and you just go straight to the best people. Cause they're also your friends. Right. And so that was essentially how that all came about.

[00:15:00] And Wes had gone from message systems to Facebook, where he built some of the extreme scalability components over there for various things that they work on. And, uh, he was getting to the point where he's like, all right, I've done my Facebook time. I want to try something different. And I knew that cause I talked to him and I'm like, why not try this?

And then Tom, I was like, we need the perfect customer success person. They need to understand how to explain all this to customers, keep them all happy, keep them all taken care of. That's what Tom has been doing for years at SparkPost, um, then MessageBird. And so, yeah, you bring together, uh, you bring together three people who all know each other well, and all have their own piece of the pie that they contribute.

And, and like I said, it's almost like cheating because we just didn't have to do this whole search of co founders and everything else. Well, if I, if I

Matthew Dunn: had a Frosty Malt beverage, I would raise it and toast you because I couldn't agree more about the, the sort of cheat codes of having, having some experience.

When you start a new thing experience in relationships when you start a new like it really is a [00:16:00] different ballgame doesn't mean it's easy, but it is a different ballgame.

Mike Hillyer: It was kind

Tom Mairs: of a, an interesting perfect storm you know where the three of us had worked together we know each other quite well we worked very well together when we work together.

And then we all found ourselves you know temporarily unemployed. Um, and, and literally just ended up having a conversation over a frosty beverage. And, and the idea just kind of percolated as part of a side conversation, you know, how would you do things differently if you could do it again? That kind of thing.

Yeah. And honestly, I thought we were, you know, we were just having a really interesting conversation as old friends, uh, that had the time to talk. Never really thought about it at the beginning, turning into a business and then things just kind of percolated and it's like, Hey, this is actually a really cool thing.

This could actually go somewhere. And that, and that, you know, organically grew with three people that are, I couldn't think of a better way to, to. Have three co founders come together as people that already knew each other and had a [00:17:00] common goal and common philosophy to start with.

Matthew Dunn: Yeah, and and probably a common some a lot of commonality in the map of relationships and knowledge since you both since you all three.

Worked in a lot of overlapping industries and worked together in overlapping industries and we could go on and on about how the email space is Kind of particularly interesting and connected, but if we get there tell me about the decision to make this an open source Project mike you said you spent a bunch of time in open source But that didn't mean you had to had to go there with this

Mike Hillyer: It it more meant that I knew the answer to the to the problem that I actually encountered.

So, um I've always been what I like to call Mr. On prem. So I, I've always done on prem email and on prem can mean server Nicola. It can mean I have a server in AWS. It means I'm not using SAS. Right. And, and as I would, as I would, you know, stay in the industry and talk to people, I talked to a couple of CEOs at some of the larger ESPs and I'm like, you know, are you interested [00:18:00] in looking at another, you know, another vendor MTA to, um, to work with and.

The answer was, we don't want to start another relationship like that. Like, some of these companies have been very long term with a commercial vendor, and they've seen, you know, changes from a perpetual license to a volume based license to a tier based license, annual license. All these things keep happening, and they're like, we don't want to start over with that.

We don't want to go to another vendor, start another relationship, where everything seems great, but then... Lo and behold, over time, the, the terms change and they, they, they wind up feeling held hostage because if they stop paying the, the, the annual fees that they're, that they're obligated to, well, then the servers just shut off.

Right. And they said, so we, you know, half of them would say like, we're going to stick to the devil we know. Cause even though they, you know, we've had all these changes and you know, the, well, what's the old star Wars quote, I've changed the agreement, pray I don't change it further kind of thing. Right.

They're like. Yeah, but you know, we can't stop paying or our servers turn off and we don't want to start the [00:19:00] cycle somewhere else. So this is the devil we know we're going to stick with it. Plus. Changing out an MTA is a huge undertaking. And then some of them would say, well, we're going to build our own, right?

Which is also a huge undertaking. Um, but they, at least at the end of that undertaking, theoretically own an MTA that will never have their license terms changed underneath them. We'll never, you know, be, be subject to, you know, all they have to do is maintain a team that knows the product. Um, there's risks there.

Stuff tends to. I mean, for a large ESP, my customer facing stuff is what's important. My, my stuff under the hood will only get updated after that if, if there's something critical. And so those teams tend to get reassigned to do other stuff. And it kind of goes into not shelf ware, but it doesn't get iterated on.

Like, It happens all the time. Like there's, there's orgs I've seen that, you know, say, Hey, we're going to use it. We're going to build our own WYSIWYG editor for our emails. Others say we're going to use Bfree, right. Or B, uh, B. io and the ones that use B have an org that is constantly improving that WYSIWYG [00:20:00] editor, the ones that don't.

Have to have a lot more diligence to make sure they keep improving. Cause it's really tempting to say, and it's working and let's work on something else. Right. So, so that's, that's the risk of the ones that try to build their own, assuming they have the skillset to do it in the first place. So with all that though, what I understood was having come from open source.

Like at my SQL, it is a commercial open source company. The whole idea of that organization is we're creating an open source MTA, and we know that people are going to use it. It got paired up with PHP, became the most popular database for quite a while there. Um, and, and, and they knew that certain CTOs would show up and go, Hey.

What happens if that breaks? Well, we go to the forum, we go to the chat, we try to figure out an answer. He's like, that's not going to work. We need to know that there's a throat to choke. We need to know there's someone we can phone. And that's how MySQL got its business was selling support to, to companies that want to use it, but need to have the insurance of, I can phone support and have this problem fixed.

Yeah. [00:21:00] And so I knew that that model was exactly what I was hearing from these CEOs of these ESPs was, they weren't saying it because they weren't thinking of that, right? But they were saying essentially, I want, I want an, uh, uh, infrastructure that I'm not held hostage by. I want to know that I can, that I can use it, that if I don't like the license terms, that I can stop paying and that it's going to keep running, that I've never held.

Captive by by that vendor, and that's what open source as a company allows us to do is to say here it is. Here's the source code. If anything goes wrong, if we're all hit by buses, there it is. You can continue moving forward with it. If you decide you don't want to pay us support. You never have to start if you, and we have to earn your business every year.

If you don't like what we're doing for you, there's nothing keeping you from stopping paying for support. Cause all you lose out on is the ability to contact us for support. The software is going to keep right on working. Right. And, and the best part is like Tom, I think was around [00:22:00] even in the early days when this was the case, but when we were selling, you know, a commercial MTA message systems, and we'd get clients who would say, I'm going to need code escrow, because the whole idea was you're a little company, you may make it, I hope you do, you may not, and I need to know that if you fail, that I don't lose use of this software.

And so there was escrow agreements in most of those early, early customer contracts to say, we keep pushing the code to a third party organization, if we ever go under, that code gets released to those customers. We just skipped the middleman and just released it immediately. Um, which not only means nobody's worried about the risk of us going under that they won't have access to the code, but it also is very transparent for security.

If you're worried about how this is implemented, if you're worried that there might be things, take our public code and, and get somebody to do an audit of it. Feel free. You know, it's there. It's, it's wide open for you. Would it

Matthew Dunn: be, would it be a fair broad strokes, uh, [00:23:00] statement to say that adopting an open source model, like what you've described, Kumo, MySQL, whatever, requires enough expertise on the, on the customer side, the one who's not paying for licensing, maybe paying for a throat to choke, right?

They've got to, you said, read the code, like they got to know how to do that to get the benefit out of that, don't they?

Tom Mairs: You know, that raises the other thing I was thinking about while Mike was talking about this. From a customer success point of view, it lends to this very, very nicely. I've been running customer success and solutions engineering teams for a very long time.

Um, and one of the things that always gets in the way, more so in the last three to four years, has been the licensing conversation. You know, a customer success person is just trying to help the customer be successful and the licensing thing. So, well, my, my license comes up or after we deal with the [00:24:00] renewal or, you know, make sure that you pay your bills so that we can actually help you out more.

I just want to help you. I want to help you be successful. I want to build a solution and help you get your messages out and, and not have that problem, but I've got this licensing thing in the way I have to, I have to worry, worry about. From a customer success point of view, not having any licensing to worry about is a gold mine because now I can actually just help the customer and now it's a conversation about support as opposed to licensing a piece of software.

So now one of the things that we have been doing with, with the, the, the current customers is, um, helping them actually get deployments, actually building implementations for them. Mm-Hmm. and, and wrapping support and professional services around that. That's really our whole model. You can have the software for free and you have access to the code anytime you want, but if you really want to be successful, you'd be crazy to do that without a support agreement or somebody to reach out to for help.

It's not just about having a throat to [00:25:00] choke. Oh, that's a, that's a really important piece of two o'clock in the morning when everything breaks.

Mike Hillyer: Um,

Tom Mairs: when you've got a new idea and you just want to bounce something off, I want to have a piece of code that I just want to test. Yeah, if you're paying for licensing, there's always that nagging feeling that they're gonna increase my fees or they're gonna they're gonna charge me extra for this.

When it's all wrapped into a support agreement, it allows their solutions and customer success people to just help the customer.

Matthew Dunn: Yeah, yeah. And I'm a huge fan of open source. I think it's one of the most. Intellectually, and I don't mean this word the way I'm going to say it politically, eco politically significant, um, things in the last 30 years, like it's astonishing what it's enabled.

But it's not necessarily, it's not necessarily for the faint of heart and with the skill set that you guys pulled together at the table that became Kumo, um, saying let's go that direction. I don't know if you've done it, if you hadn't had experience doing it in the past.

Mike Hillyer: One of the [00:26:00] interesting things to me, and something to keep in mind with this, is that our initial target customer is Literally an email service provider, an organization where there are people dedicated to mail ops, to running, running the servers, keeping them configured.

Often there's an overlap of deliverability. Sometimes there's not some, some of these larger ESPs have a deliverability team and a mail ops team and mail ops is about keeping the servers going. Deliverability is about tuning and everything else. But, um, one of the things that we didn't expect early on, like you, first of all, you have to have a unicorn developer.

Wes is, Wes is absolutely incredible. He, like I said, he's. built some of the key components to keep Facebook fast and running. Right. And so now, and this is the third time he's built an MTA actually. So it's not even like that's new to him. Um, and because of that, one of my initial expectations was that yes, it's open source, but we I was like, but we're doing this as a company and we're not expecting initial contributions.

Like this isn't yet a community project. This is a commercial product that is open [00:27:00] source. Yes. Uh, the, the unexpected surprise for me, the pleasant surprise was we've already had of the, of, of our customers and the evaluators, we've already had. For outside individuals who have contributed to code. So as an example, we had, um, we had somebody who's, who's working on our software, um, who was testing out and they're like, we, uh, our initial implementation of DKM signing didn't expose settings for what's called canonicalization for headers.

And for the body of the message, uh, we figured, you know, most people are just going to want to use the standard relaxed, relaxed. So we're just going to deploy that for now. Next thing, you know, we get a pull request into our repo saying. Here's a block of code. I want to be able to, to do header and body canonicalization differently than what the default is.

And they wrote code, they wrote the modifications to our code to make that available to them in their config and then submitted it to us. And Wes gives it a little feedback. We do a couple of tweaks in it goes. So that's the other amazing thing is like. With [00:28:00] open source and especially in this case, rather than saying, Hey, I need this, when can it be on the roadmap, especially if it's a little change like that actually was, I don't have to say, Hey, I need this.

When's it going to be on the roadmap? I, this person modified their copy to do it and then sent us the, um, the code modification so that we could, you know, add our choice incorporated into the. Into the official mainline release, which we did. Um, and so they got what they needed immediately. And then we were able to take that contribution and make it better for everyone as a result.

So. That's the other beauty of it is that people don't have to be like, Hey, I really need this when can it be on the roadmap if they want to, they can actually, I mean, we're always happy to do, we're happy to do paid mods again. You know, we're not charging for licenses. So we make our money where we can, which is professional services, but, um, but they don't have to wait for us.

They can, they can. You know, make that contribution themselves, keep it for themselves if they want to, or push it back to us to, to incorporate into the, into the [00:29:00] larger release.

Matthew Dunn: Into the, into, wow, that's, uh, that's terrific. Do you mind if I asked what, like, what the sort of stack underneath is, language, uh, data storage and so on?

Like, because writing it, writing it, especially for the third time. You've got a unicorn developer going, I'm going to use this to tackle this big problem. Like, what are the key pieces? It's all Rust

Tom Mairs: and Lua. So, um, That's interesting. Rust is, um, It's like a better version of C, right? It's what, it's what it should have been.

Um, it's, it's incredibly fast. It's, it's, uh, very malleable if you know how to, how to deal with it. Um, it, it manages the hardware very, very, very well. And then on top of that, from a configuration point of view, we use Lua. Lua is, um. It's, it's, unfortunately, people don't know a whole lot about Lua, but it's a fantastic language.

It's, it reads almost like English. It's, if you've, if you [00:30:00] ever learned Basic when you were in high school, it's, it's so incredibly simple to, to write Lua. But it also has the power of, of directly, um, uh, working with at the sea level. So Lua and Rust work really, really well together. You never have to touch the Rust.

It's the basic engine that goes very, very, very, very fast. Lua allows you to do the configuration. And we intentionally didn't write a static config, we did the whole thing in Lua, so you have total control over how you manage the message as it comes in, which adds some complication for us from a support point of view, because there isn't a standard config, you know, the config is literally I posted it earlier in a post on LinkedIn, the config is about three lines of Lua, you know, Start a listener and send the mail.

Matthew Dunn: I'm not familiar with Lua, but in my, in my understanding, it's accurately that it gives you a smart. [00:31:00] When people ask for script ability or a script, they's like, I wanna do programmatic stuff. It doesn't mean I wanna undo the engine that's running it, but I wanna get beyond, I've gotta do it that way. Yeah,

Mike Hillyer: yeah, yeah.

You can do

Tom Mairs: things like if, if the else conditions you can, you looping, you can do iterations, um, you can reach out to databases and pull things back in. Yeah. So. Something I just published this morning is, you know, the ability to say, okay, if this message has this username, go out to the database, pull this back, see whether or not it validates.

Then if it's authenticated, then go ahead and send it. But before you do that, strip all these headers out. Cause I don't want that going out to the public and then sign it with Lua and do it in all that millions of messages an hour. And you can do that. Lua allows you to expose just, you know, ten lines of instructions, if this, then that.

And you don't have to worry about the thousand lines of rust underneath it that's actually doing the heavy

Matthew Dunn: lifting. Oh, that's cool. Boy, no wonder people are starting to bang down the door already. The mod you [00:32:00] talked about earlier, Mike, was that done in Lua?

Mike Hillyer: So, so yeah, so what happens that the I will say one more nice thing about rust is it's all memory safe out of the box and it has so many built in capabilities like just as a quick little anecdote with, um, with momentum, which was message systems.

It was C plus plus and it was Lua as a scripting language, but not a config language. Just just to draw a line there. So your configs were still text files in the text file. You could say at this point in the flow. I want to call this Lua script and run it and then get back to it. But you couldn't, you couldn't configure it using Lua.

Um, but, but like one of the biggest changes that happened in momentum during my tenure there was it was a scheduler based approach, which is, I'm not even gonna. Deep explain it. But instead of having a thread for every connection, it has a thread that is constantly watching everything that goes on and acting on everything.

And it uses extra threads to do like the slow work. Um, when, when the number of course, like when there was only two cores in a server. Best ideal [00:33:00] approach as there were more and more cores in, in a server. It was not as ideal as multi threaded. And so Wes, one of Wes's largest projects was changing that scheduler approach to be multi threaded schedulers, where there's multiple schedulers watching different areas and reacting to all the different events.

It made it incredibly scalable over 10 million messages an hour in some circumstances per box. Um, when he went to Rust. Which he did back at, uh, again, he did that back at Facebook. He became a big Rust advocate part way through, um, and he was teaching people how to use Rust and they were migrating a lot of what they did to Rust over at Facebook.

But when he comes here to do that MTA, to do Kumo MTA, multi threaded scheduling is just a... Crate, it's an already existing feature and he just plugged it in and decom signing was a feature and he plugged it in. It wasn't complete. He did submit some contributions back to that project, but a lot of what he did was just grabbing these crates that are already prebuilt and gluing them together with code.

So instead of building it all from the ground up, I mean, [00:34:00] there's a reason why we've only been at it since February and we're pretty much code complete is because rust adds all these capabilities that used to be hand coded in C. So. With, with Lua though, yeah, I mean, the benefit is the heavy lifting's in Rust, the, the telling it what to do and how to do it is in, is in Lua, and when somebody wants an integration, like, for example, saying, hey, I want to, I want to validate, like Tom said, I want to validate auth off of, off of a data source like sqlite, normally that's a config file.

And then call up Lua here and then do Lua. Our config file is a script. So if somebody says, Hey, um, like normally most of these ESPs, they're using power MTA, they're using momentum, what have you text config files. And they write little scripts that. That take a data source with their DKIM signing and they write a file in the format and they place it in the right place and they tell the server to reload the file and the server reloads the [00:35:00] file.

And now I know the newest rules for, for signing DKIM with us. We write a Lua script that says when it comes time to do a DKIM signature, this is where it is in SQLite, go ask it and then do the signature and that's it. There's no, there's no static text file that needs to be updated. It just. Asks the server, right?

And so with momentum and even power MTA, one of our challenges used to be, well, I've got to allocate enough memory space. Cause I've got to take, like, let's say I've got 4, 000 IPs I'm signing for and 5, 000 customers and all these different throttles, there were customers so big. We had to increase the memory limits of the, of the config store in memory to fit it all these files would get to be.

The hundreds of megs, even with this approach, we don't store the config in memory. We're actually saying at the moment, you need to know what DKIM signature to use. Go ask for a minute in case you're signing a lot in Lua, right? And then go cash that information in case [00:36:00] we need to do that signature a thousand times or whatever, because it's a big cat, you know, campaign.

And then forget about it until the next time you need to know that scenario, right? What IP should I use? Most of the time you pre, you predefine all the IPs. You have to schedule the reloads because you got to balance the server to reload all the data here. We're saying, Hey, if a tenant comes in that you haven't seen before, ask the data source and then cache that info.

And what IP does go out of ask the data source, which could be a file. It could be a text file on the disk. You don't have to stop using text files, but now the text file isn't a config file. It's a data source that's being queried. Yeah. Yeah. Right. It's now a text database. You're handling instructions.

Yeah. And so now everything is just loaded as it's needed. We don't have a server. We don't have a config reload command because we're constantly reading what we need to know when we need to know. Yeah.

Matthew Dunn: Yeah. I gotcha. Five minute hookback question. For my two minutes of looking at Rust. Crate is sort of the equivalent [00:37:00] of library in Rust land?

Effectively, yeah. Okay. Okay. Okay. That's, that's, that's what I thought. Wow. Wow. That's a, that's a powerful combo. You know,

Tom Mairs: Mike actually hit on something that's actually a fairly major problem, a hidden under, underlying problem in, in this industry for, for mail ops people, that isn't really well known, I don't think, in that, in that all of the major, um, MTAs have to do this reload.

Reloads are a pain. Yeah. So you make a configuration. Marketing wants something slightly different or you've got a new key that you have to add. Yeah. You have to make that change in your config and then you have to reload the server. Yeah. Which means empty

Matthew Dunn: the queue. Right? Well,

Tom Mairs: not only that, but if, what if you have 40

Mike Hillyer: servers Right.

And three different data centers. Right.

Tom Mairs: Now you gotta, it's, it's an engineering problem in itself. Yes. Just reloading all of that. Yes. Yes. Sumo MTA is cluster aware and you can actually have centralized management and you don't have to do any of those reloads because it's just reading data when it's needed.

So all you have to [00:38:00] do is update the backend data and keep on going. There's no reloads, there's no waiting, there's no requeuing, there's no dumping stuff back out into the wild and having possible deliverability issues because you've now you've Pushed all that queue back out again when you didn't want to.

Yeah. You don't have to wait for those. You don't have to wait for a whole bunch of maintenance windows to make changes to your config. It just makes life so much easier from a, from a management point of view for an, for a mail ops engineer.

Matthew Dunn: Yeah. Yeah. And, and much more, uh, DevOps way of running. The MTA, as opposed to stop, wait, reboot, new config, got to not make a typo in the config file, right?

Mike Hillyer: Right. And honestly, that DevOps mail op approach is, is also core to what we're doing. Like Kumo is by the way, Japanese for the word cloud. So we, right from the beginning, we wanted to know that this was going to be cloud friendly. Um, because everybody wants to move to their own [00:39:00] cloud, right? Like they, they want to use, they don't necessarily want a cloud, like as in a third party SAS vendor that does their email relay, but they want their servers in their own cloud and sometimes they want to take some of their traffic and send it off through somebody else's cloud service and maybe support that.

But from the mail ops perspective, like historically, the MTA has been a black box, right? It's you set it up, you put the config file in, you inject a message in over SMTP. Maybe it's got an API or something. And then you've got a log file that you read for what happens on the other side. And it's. Never been an integrated element of an environment, even when you integrate.

Well, how do we integrate it? Well, you're going to write these include files to this location from whatever data source you have using a Perl script, and then tell the server to reload it. And then here's how you read the logs. And, you know, if you want to get to close real time, you read them on a regular basis.

And there's been advances in some of the commercial tools, especially since I left, cause I wrote some of that roadmap before I left, never saw it see fruition. But, um, that's not how it works anymore. People [00:40:00] want APIs to talk to servers, they don't want to sit there and do command lines. We used to do We used to set up for people where we'd have cron jobs run a command line that would then do something to the server on a regular basis, or we'd have a little script that would catch a hook and use it to issue a command line and send it back as a response, like all that.

Um, so they are getting better at implementing management APIs and injection APIs. And, but the modern thing is, well, we don't want, you know, logs anymore. I need a web hook. I need a, I need to get this data back in real time to the rest of my environment. So now we're looking and saying, okay, and this again.

We also cheated in the sense of we didn't have to have a product team go out and do a whole bunch of research because we've been doing this for so long that we know what we would do better the next time. And so it's like this time right out of the gate. It's got to have webhooks. It's got to, you know, push data out over AMQP.

It's got to take injection over an API. It's got to be managed over an API, right? Like we don't. We have a command line thing. All it does is call the API. It doesn't do anything directly. It's just there [00:41:00] to call the API on your behalf. Like we, we knew that this time around it had to be an integrated part of an environment, not, not a black box that you talk to in a specific way.

So that's why. You know, Lua lets it reach, it can take its config from any data source and just use it directly. It can take API. Uh, I think we're putting Kafka for sending out, uh, log entries next. Cause somebody asked for that. But like, you know, AMQP was a big one. They want that real time data coming out of the server and, you know, to as many different destinations that we had.

One thing we didn't expect this early on is like we knew people would want webhooks and somebody's like, I need a webhook for each tenant. So I need each tenant to get a webhook with their information to their webhook endpoint. And we're like, Oh yeah, I can see why you'd want that. And then we did it in Lua.

It, we didn't modify the rust code because the Lua was capable of doing that. Right. So it's like, all right, well, here's a Lua script. That's going to do that. That's. That's the beauty of like config as code is, you know, like Tom and I used to [00:42:00] love momentum because it's like, we didn't have to know the use cases in advance to make them happen.

And even then we were writing config files and then bits of Lua. Here we're like, we really don't need to know the use cases in advance for most things because it's all Lua scripting on top of that Rust. So it's fast and it's flexible at the same time. Because you can never predict what somebody's going to do when you hand them a

Matthew Dunn: toolbox.

Right. No, this is like, this is, this is actually a fascinating conversation, a bunch of levels. Did want to ask you about something. You just gave me the opening to ask about it. Um, Kumo, Japanese for cloud. And you were planning from the get go for. You know, cloud implementations, whatever that means. You look at the decision that, uh, I think it was ElasticDB made a year, year and a half back about changing terms because basically they felt like AWS was taking them for a ride.

Did you guys have conversations about that? And have you grappled with it? Yeah,

Mike Hillyer: we've had licensing [00:43:00] conversations for sure. There's, there's, um, yeah. So one of the things, this actually ties into another, another area. We, there's been an example, there's been multiple examples over time of companies that started out with open source and then changed their license.

Mostly they change it to be like, all right, well, if you compete with us, you can't use the open source or if you host it or what have you, um, because of that. But. If you look deeper, what you almost always see under the hood of all of those companies that make those decisions is investors. There's VC money.

There's ROI. Like one of the things that we wanted to get away from is ROI. We wanted to get away from this idea. Well, not ROI. We, we want ROI, but hockey sticks, right? Like every time, like every time I've worked for a company that has VC money or has back backing by investors, everybody keeps saying hockey stick, hockey stick, hockey stick.

Tom and I agreed early on. The only time we want to ever say hockey stick. [00:44:00] Flames are doing and whether or not Atlanta is going to get a team for a third time so I can watch in person again. That's it. Unless we're talking hockey, no hockey stick. But the idea is that these, these, when you have that investor money, you have this pressure to, you've got to, you've got to increase revenue.

You've got to find new tans. So you wind up with products being introduced that don't necessarily benefit the customer decisions being made that don't benefit the customer. And it all comes because there's this. You know, there's this demand for that return from the investors. Yeah. We, we agreed and we have not, we are not taking investor money because we don't have an exit plan, right?

We're not here to, to sell this to someone. We're not here to go public. This is, this is a good little, little market to work in, but it's not big enough to be a public company in. And, and getting acquired is never good for the customer in the longterm, right? That whoever acquires you is there to find a way to get more money out of it.

Then they were getting out of it so that they can get that investment back. So most of those decisions that you see [00:45:00] are based on that concept of, we need to get a bigger return. The other thing is. We're never going to, um, we're never going to have that Amazon concern because we also have no desire to be a cloud relay provider, right?

Like we're not, we're not looking to, cause then you have to have, well, first of all, you have to have investors to be successful because you got to market it like crazy. I have sales teams cause you're going after individual. You know, cloud users, um, and then you'd have to have compliance teams and deliverability teams, because now you're hosting it.

Now you're providing it. You got to manage all the IPs, all the servers, all of that. We're a software company. We're here to provide this out for people to use, to build those things for themselves. And that's great. If they want to go and they want to, you know, give SendGrid a run for their money with our software, by all means, go for it.

But we're not here to build that kind of a business. And so we're not worried if Amazon were to take it and use it to run SES. Yeah. Because we were never planning to introduce our own SES in the first place. Like, like you look at [00:46:00] HashiCorp, right? Oh, Amazon's taking it and they're hosting it for us or for people.

And now they're taking profits and we're, you know, now they're not using our hosted version because they're using Amazon's. Yeah, we're not doing it in the first place. So you, the only way to compete with us would be to build an MTA and try to sell support for it. Right. And even if you took ours and tried to sell it.

Yeah. Good luck supporting it. Right? Like nobody's going to go to you for support when they can go to us. So, the thing that we do, we're not expecting anyone to compete with, and, and using our, using our code. So, we just aren't worried about that particular scenario, really. Because we don't have, you know, we don't have investors saying, find more

Matthew Dunn: revenue.

Yeah, yeah. Um, my, my sense is that the, um, the clients of open source who build, derive businesses, By adding something to that mix, whether it's, you know, cloud hosting or something like that, they seem to have learned the lesson that, that they've got to, they've got to in some way [00:47:00] stay connected with the source of the source for everyone to stay healthy, viable, etc.

in the long run. I mean, you look at the big, the big cloud players, and I think Microsoft particularly has had a major heart, heart transplant on in this front, like, They're, they're quite pro and quite supportive of open source, broadly speaking, because they can't afford not to be. P. S. they're all running on Linux boxes too.

So, like,

Tom Mairs: right, it's got to work. That's a relatively recent change in the life of Microsoft as well. And I think that is, I think that is an indication that, that we're I want to say we're in, in, at the beginning of a revolution, but I think we're kind of in the middle of it and it's being kind of quiet.

Um, you know, the, there's a ton of companies out there running on open source and, and they don't even know it. How many Docker containers are out there and people think that they actually own that? It's like. No, you're actually running on

Mike Hillyer: open source right

Matthew Dunn: now. Yeah. Well, I mean, you've got, you've got [00:48:00] audits to make sure you have some freaking idea of whether or not you are.

You can say Mike. Oh,

Mike Hillyer: I was also going to say like support for us as a product, but support goes both ways. Like, like I've had people say, well, what if we only wind up like, they're like, well, we only

Matthew Dunn: call our current vendor like twice

Mike Hillyer: a year at most. What should we even be paying support? And the answer is, well, it's support in two ways, right?

It's, yes, you have our support to use the product. And yes, it's going to be very stable. So once it's up and configured and running, odds are you're not doing call us with a SEV1 issue because it's just not going to break like that. Um, but the other is to say, well, support support also means you're supporting this team, right?

If you build your own MTA, you would pay salaries for people to constantly, well, first to build it and then to keep it going because you would never run the risk. Of, of having an un, you know, unmaintained map in the middle of, I mean, their email service providers, the word is email. They are like, it's amazing how many people don't realize you'll, you'll, I'll walk into an office of a big ESP and I'll pass like [00:49:00] 50 designers.

And then 30 copywriters and then, you know, 20 marketers and the whole wing of salespeople going to the basement where there's two guys that are running all the MTAs and anyone else in this building can, can get sick. But if the servers go down, the whole business stops, like half the, half the graphic artists can be out if it's a full service shop or the support people can be gone.

And it's still not the end of the world, but there's only one or two guys at the bottom that are running the servers that actually. Make that whole business tick. And so part of it is that, yeah, they're going to support open source because the great thing is, is that our users want us to succeed, right?

They want us to keep doing this and keep making it better and keep it going. And so part of paying support is also just a commitment to say, our business depends on this thing. And that means we depend on these. I mean, yes, it's nice to know there's source code out there, but the reality is we need these guys.

to continue to, to do this thing and to build out more people to help us with it and, and to make sure that that [00:50:00] organization continues to do that indefinitely, which is our whole plan. Because we depend on that piece of software more than anything else to keep our business going. Yeah. Like that's a big trust they put on us, but they also understand that for that to work.

You know, yes, they have the option to use it for free and a lot of people do. Um, but it's, it's the, and some of them don't like, I mean, if a guy's running a website and he needs a mail server, the mail server isn't critical to his own business. But if you're an email service provider, your whole organization counts on this.

So. What's throwing some support money at it. And we tell people like, look, take what you pay for support now to your current vendor. And we'll take that. We don't, you don't need to spend new money. There's no new license fees. Just take your existing spend, move it over. And in return, this thing keeps going.

And their software that they're going to be depending on continues to be evolved and improved because this is all we do. We're not. You know, even if you build it yourself, okay, how long can you keep your focus there before it's working good enough? And you, you move those people to [00:51:00] something else, keep one guy there just to keep an eye on it here.

This is all you do. And so for that money, you get to keep free, you know, three and eventually more people just, just doing this one thing.

Matthew Dunn: Awesome. I'm, I'm keeping an eye on the clock on the top right here. And I, cause I don't want to use up your time. We should probably think about wrapping, but I wanted to ask you how, what the.

The state of the business of Kumo MTA. com. Is he guys happy with where you are? Well, what are the immediate steps you want to see in terms of growth and reaching more customers, like how's things looking? You

Tom Mairs: know, I, I have to tell you, you know, you touched on this a little bit about the, the concepts of going into open source and selling all of that.

Um, I, I, I did have some fear trepidation at the very beginning of this whole conversation, not this one, but, you know, the, the conversation about Kumo, um, and, uh, and as we talked through it, uh, the, the idea of trying to sell. Open source software [00:52:00] grew on me and I started thinking about all the amazing possibilities that were there, but I was thinking we would work on some foundation for a year and maybe June or July in 2024, we'd be able to launch a product and and be able to, you know, get out into the community.

And I, I was amazed when, you know, like Mike said, we kind of started this in the conversation in February, we actually had production install running in July. And that was a, that was a moment for me to think like, we actually have something really special here and we've got, it's not just a piece of software, it's a whole community that we're building and they're contributing to this.

And it's like, Um, I couldn't have dreamed of a better collaboration or a better, um, uh, way that this has all flowed together. I'm, I'm super excited about where we are right now. And I'm, [00:53:00] I'm having a hard time actually thinking where we're going to be in three months because we've kind of done everything we planned on doing.

I thought we would be here a year from now and we're well past where I thought we would be even, even a year from now. So I'm very excited about where the future is and how we can help the community grow and help, um, mail ops people get, you know, better quality of life,

Mike Hillyer: better tools to work in the basement.

Yeah.

Tom Mairs: And, um, I'm, I'm super excited. I love our partnership. I think we work really well together and I love the community that we're, we're helping to build.

Mike Hillyer: And, and even, um, yeah, everything exceeded. Like we went to mock for the, uh, you know, for the, well, not for the first time, but the first time as Kumo MTA, um, you know, in, in last month in, in Brooklyn and, um, we had like four or five conversations lined up and we thought, okay, that's pretty good.

Maybe we can talk to a couple more people while we're there. We wound up talking to like. Well, 13 different [00:54:00] organizations and half of those additional conversations were people telling other people go talk to those guys. And that is not something we were expecting, uh, you know, that early out of the gate.

I mean, it's like, I mean, it feels like if somebody said, Hey, if anyone ever asked me for business advice. I'm going to be like, do not try to use this as an example. Um, because you know, you, again, you read your LinkedIn, find your ICP, work on it, you know, do experiments. It's like, no, we knew our ICP was so down that every time I talked to someone and I say, look, imagine a high performance sending MTA, but it's open source.

Everybody's just like, I'm in, right? That's it. I'm in the only challenge we have is making sure people hear about it. And even that's actually been a lot easier than we were expecting. Cause again. Turns out word of mouth is, is way better than we would have thought for something like, and it doesn't hurt that we know absolutely everyone in the industry at this point almost, but like word of mouth was not something I was expecting so early on for people to say, Hey, talk to those guys.

That was not part of the plan. And, and yet [00:55:00] here we are. So that's, that's been, like, like Tom said, I think that's the most amazing thing is like, we had no idea we were going to be here this quickly either. I didn't think we'd build it that quickly in the first place, but it turns out was Rust is very, is perfect for this.

And, and on top of that, we didn't expect, you know, this much uptake. Yeah. But yeah, when I talked to the original people, it's true. They're like, I don't want another vendor relationship like this. And they don't, but they don't want to build their own. And so here we are.

Matthew Dunn: So my parting thought is I really want to talk to you guys again in about a year.

Like this is such a great story. It's such a great conversation. I really appreciate you both, uh, you know, sharing it and kind of unfolding the thinking behind it. I don't think it's just accidental. I don't think the velocity you've got is as accidental as it may feel. I think that's some of your cheat codes, Mike.

I told somebody,

Mike Hillyer: I told somebody this is a overnight success. 17 years in the making. Yeah, yeah,

Matthew Dunn: exactly. Yeah, exactly. [00:56:00] Well, we should wrap. My guests have been Tom Ayers and Mike Hillier of KumoMTA. com. Correct, Mike? Correct. Correct. Thanks again, gents. I really enjoyed the conversation.

Mike Hillyer: It's

Tom Mairs: been great. Thank you.