Goodbye SimpleNexus

Introduction

When I started working at SimpleNexus six years ago, I had no idea how much personal growth I would experience during my time there. Over the last six years, I have learned a great deal and have been able to develop both personally and professionally. This chapter in my life has now come to a close. I wanted to write this blog post as a bookend to reflect on all the great memories and experiences I've had at SimpleNexus, as well as the software engineer I have become.

Personal Growth

One of my favorite things I have taken away from SimpleNexus is the new hobbies that coworkers have shared with me and the friendships that I developed with those coworkers. In 2019, I ran a marathon with a lifelong friend and coworker Chris Egan. My boss Steve Gardanier and his son Michael Gardanier who also works at SimpleNexus set up two different water stations for us along the marathon path because COVID canceled the marathon two weeks before we were going to run it. It is super rare to work with people who would take time out of their Saturday to come and support me with my personal goals. Later in 2022, Chris, Michael, and I all completed our first sprint triathlon together. That started our journey into biking and in 2023 Michael and I completed our first century ride.

Next, I have to give a shoutout to my friend Robert Montgomery who got me into mechanical keyboards. He helped me hand-wire my first split keyboard and since then I’ve built a couple more. It is a whole new hobby and I appreciate all the time he took helping me get into it and answering my many questions. Building a keyboard is like a Jedi building his lightsaber. Are you even a software engineer if you haven't built a mechanical keyboard?

Formula1, ThePrimagen (my new favorite YouTuber), home automation, vim (the editor I use as my daily driver), all sorts of new restaurants and foods I had never tried before, golfing, gaming (big Warzone and Rocket League fan here now), and so many great book recommendations that have changed how I view the world were introduced to me through different people at SimpleNexus. I hope that anywhere I go I can l learn and grow and pick up new things from the people around me. I have been lucky to work with many amazing people over the last 6 years. Those people are hands down the number one reason (among many others) I’ve stayed at SimpleNexus as long as I have.

Work is work, but it has also been a lot of fun if you work with people who can make it fun. I will always remember the late night deploys made better with episodes of Seinfeld and Brooklyn 99, the annual ski trips, watching BYU bowl games in the office around Christmas time, game nights, Battle Ball, escape rooms, Friday afternoon basketball games, and more Super Smash Brothers than I care to admit.

My Professional Journey

One of the things I appreciated most about SimpleNexus was the consistent mentoring and support I received from my colleagues. Dave Stevenson and Steve Gardanier were instrumental in my growth in engineering. They trusted me with projects and responsibilities that allowed me to grow my skills and gain valuable experience. This trust and support allowed me to push myself and take risks that I might not have been able to at another company.

As SimpleNexus grew and evolved, so did my role within the company. I had to take on new challenges and responsibilities outside my comfort zone. While this was intimidating at times, it helped me develop new skills and become a more well-rounded employee and person.

Intern Life

I started at SimpleNexus back in October 2017 as an intern. It was a small startup. I was the 25th hire and the 10th engineer. There was no intern or mentorship program at SimpleNexus yet. As an intern, I worked just like any other software engineer. We all worked off the same Kanban board on the same projects and tickets. It was the perfect environment for me. For that first year, I was a sponge. I was able to dive into every aspect of the code base, learn several new languages and frameworks, and watch and learn from all the other engineers around me. On my first day, the engineer helping me set up my dev environment asked what my favorite editor was. I had no clue. I had only ever used Intelij and didn't know what else was out there. I asked him what his favorite was, which was Atom, and then I told him that Atom was mine too and installed it. I have come a long way since that first day. I miss those early days, when live coding on a server to fix a bug was possible (and, dare I say it, not even frowned upon), and deploys happened from developer machines with a script. I miss the robotic voice that said “deploy to production complete” when it finished. It was a time of crappy internet and eating at R&R more than anyone should in their whole life.

I can remember my first release of a big project. It was December 2017. We had a client willing to pay the big bucks for us to implement a whole spreadsheet of enhancements to the customization functionality for the emails sent through our platform. Up to this point, the emails were configured in HTML, and a 1000-line recursive method performed replacements on the email body based on the context of the message. I researched and built a UI with an open-source WYSIWYG editor and introduced templating language called Liquid (by Shopify). Rolling out the feature involved several deploys and tasks to convert all the existing templates. I remember staying at work until around midnight, with two other engineers who had volunteered to keep me company and bring dinner and snacks. That night was the first time I got that feeling that comes from merging (or enabling a feature flag) for a big project and watching it go across the finish line and into the hands of real users. I have loved that feeling ever since.

One project from my intern days that has stuck around with almost zero code changes is the integration between our core system and Netsuite. I think this was one of those projects given to the intern because no one else wanted to do it. I wrote the code for both our system and Netsuite to handle pretty much any contract sales could think of. We had flat contract rates for some customers, some customers had per-user licenses, and in some contracts, the customer was billed for certain features being turned on for everyone, or billed for features turned on at a per-user level. We billed on some features transactionally, some in tiers of usage, and stored all that history in Netsuite. This system worked for us when were making little over $200,000 MRR and still worked well beyond the $50 million ARR we were making when we were acquired by nCino.

Platform Team Lead

I graduated from Brigham Young University in April 2019 and started full-time at SimpleNexus as the Platform Team lead. The Platform Team was created in the first batch of teams once we had grown to the point where teams were necessary. My friend Zack Ormond, myself, and our product manager Ryan Chadwick were the initial members of the team. It was an internal tools team for developers and other departments. We built tools like a framework to handle email, sms, and push notifications and an event system built on top of Apache Kafka. After about a year, my team absorbed the Infrastructure Team as well.

Looking back my two biggest wins from this time were an infrastructure migration to Kubernetes (K8s), and a major system refactor we refer to lovingly as the remote loans project. When I started getting into the infrastructure side at SimpleNexus we were running on 10 different EC2 instances. Back then we were serving around 15 million requests a week, when I left the Platform Team we were serving just under a billion requests a month. We were starting to grow and our traffic was doubling every couple of months. I wasn’t the one who decided to switch to Kubernetes but I inherited the project in the earliest stages and myself and one other engineer did all the work over a couple of months to fully migrate production 100% over to K8s. There was a large learning curve but I have grown to love it and I’d say I’ve even done some pretty cool things with it. One of my favorites is something I call the SSH Proxy Controller. When we flipped over to K8s we didn’t want to interrupt developers’ normal workflows which often involved SSHing into our staging or production environments. I wrote a controller that would accept SSH requests and then for each one, it would provision a new Kubernetes pod for the user with the docker image from the service they were trying to access.

The remote loans project was a critical refactor of our core data model. Essentially we had about 40-50 models and database tables that were (mostly) duplicated data and the code for maintaining both sets of models made the system overly complex. The project itself took around eight months and we touched around 90% of the files in our codebase to some extent. If we hadn’t done this refactor I don’t know how development would have been able to continue to add features at the rate we did. Over those eight months, we had to move over the data that wasn’t duplicated and drop all those database tables while refactoring code to align with those changes. These database tables were our most trafficked, largest, and mission-critical so the stakes were high and we spent many a night at 3 AM running database migrations and data cleanup and deploying code changes one model at a time as we rewrote major parts of our codebase. The project was a huge success and along with reducing complexity in the app and database, it also improved our average response time by ~20ms when we deployed the largest portion of the project. Somehow we all survived the remote loan apocalypse.

It's funny that a lot of my favorite, or at least longest-lasting memories, have come from production incidents. I can remember a night a couple of months after we had fully moved onto Kubernetes when I was at home around 8:00 PM and was cleaning up some unused AWS resources in a region where we didn’t host anything production related. I saw three or four IAM roles and couldn’t find anything using them so I went ahead and deleted them. Of course, within two minutes my phone was ringing and the whole site was down. It was in that moment of sinking dread that I realized my mistake. IAM was a global service NOT a regional service. Sure, nothing in us-west-2 used those roles but they were the main instance roles used on the managed nodes in our production EKS cluster. Although I was the team lead, I was never the smartest or most experienced on the team, and in previous incidents, I’d always benefited from watching those more experienced engineers take charge and make things happen. This night was a turning point for me. I had the most Kubernetes experience on the team and knew the most about how everything worked and not a single one of them had a clue about how to proceed. I was able to figure out what needed to be done and organize the team to monkey patch production back together but the real fix required us to completely recreate our production cluster from scratch. All in all the site was down for maybe 20-30 minutes and since our app was used mainly during business hours we didn’t have any large customers yelling at us.

I’ve broken production more times than I can count and probably hold the record for most time at SimpleNexus. I also probably hold the record for fixing the production as well. That said, one of my favorite parts about SimpleNexus is the no-blame culture they had. We were encouraged to take risks, fail fast, fix fast. We accomplished some major wins for the company because we were willing to take risks. When things did go wrong we responded as quickly as we could and always adjusted our processes based on what we learned from those incidents. Semi-related, the other night I was grilling some burgers and set my backyard fence on fire. I came inside and announced to everyone there was a fire and asked my friend to get me the fire extinguisher. At the time I had a ton of adrenaline rushing through me, but no one believed me because I was asking so calmly. I’ve responded to so many incidents now (the Platform Team always got roped into all incidents whether or not it was related to anything we did) that I think the same principles we followed during those incidents have leaked into other aspects of my life as well. If you looked back at my first incidents, you wouldn’t see someone calm, collected, organized, and in control but luckily I’ve been able to practice and develop skills over the years in this area.

It has been amazing to help take SimpleNexus from those early manually managed EC2 instances into a large enterprise world. We now have a very mature infrastructure all written in code that supports many microservices, developers, clients, and compliance requirements.

Engineering Manager

After spending a couple of years leading the Platform Team I started to worry that I was developing too much depth in one area too early in my career. I was a developer at heart and was worried that I wouldn’t only be able to find Ops or Sys Admin-type positions in the future. My boss was looking for engineering managers and I decided I’d give it a shot. It was a great experience for me to try my hand at management. I like working with people but I had hoped I’d get to spend more time in the code than I did. At one point I was managing 30 different people across five teams and that took all of my team leaving me further from the code than ever. It was great to see what happens in management and I learned a lot of things that I could apply as I transitioned back into an individual contributor role but the most important thing I learned about myself is that I want to spend my time coding. At least for the next several years of my career.

There are still several things that I am proud of from my time as an engineering manager. One of the teams I managed was our QA team and I worked hard to put all the QA Engineers out on a team as automation engineers and introduce integration tests in our pipelines. I think that developers as a team should be responsible for their quality and hate the back-and-forth that happens between a developer and a manual QA. I developed an incident response plan for the organization as a whole so that the Platform Team would stop getting pinged on every incident and instead the team responsible would get the on-call notification. During the time I was an engineering manager our department grew from 40 developers to 80. It involved a lot of interviewing and hiring and work to try and keep the amazing culture we had developed. The last thing I remember from my management days was helping the company survive the launch of a new mortgage application form called URLA and be ready to help all our clients go live with it. The software itself was more than ready for the new application but it was a huge effort to help each of our customers configure it in a way where they would be compliant with the new rules.

Principal Engineer

After being an engineering manager, I transitioned into a role as a team lead of our Architecture Team. It started as a team made up of each of our guild leads (Android, iOS, Frontend Web, and Backend Web) and a couple of other strong principal engineers. We spent our time advocating for the technical side of development, we highlighted areas of tech debt and maintenance and updates of our systems that needed to be addressed, maintained the technical roadmaps, facilitated communication between disciplines to align on a single technical vision, and pass on best practices and patterns, and spent a lot of time working on projects and jumping between teams helping where it was needed. Our CTO had retired earlier that year and we tried our best to be a hive-mind CTO and push the company in a good technical direction. I learned a lot from those engineers and will always be grateful to Lee Hester who has changed everything about the way I look at software design. I'll even forgive him for making me read Domain Driven Design by Eric Evans (a great book, aside from it being a textbook).

I attended AWS re:Invent several times while working at SimpleNexus. I have great memories from going there with coworkers (shoutout to Massiel Islas and Spencer Fitzgerald) and I also learned so many amazing things. re:Invent was always a source of inspiration and kept our Platform Team roadmap full. During this time I helped them migrate our single AWS account into multiple accounts following the best practices outlined by AWS in this white paper, we put into place a solid disaster recovery plan and spent a lot of time setting up the infrastructure to be able to support cross-region failovers. I worked with several teams to set up the first microservices.

I also helped solve lots of problems that came from scaling up both the system and our development organization. After doubling the number of engineers we had, our current workflow, which had every developer merge their changes into a single staging environment before letting QA test it out, wasn’t working. Staging was constantly broken, and it was hard to even get code out to test. When it was tested and reported broken, developers would blame someone else’s code and ship it to production anyways and then cause some sort of major incident. I decided it was time to fix and built something called Feature Environments. If I had to pick one project that I am most proud of from my time at SimpleNexus it is this. I completely changed our git workflow and pipelines. Instead of a single environment, I wrote all the necessary pipelines and infrastructure to build and deploy an ephemeral environment per merge request. I introduced seed data scripts that would allow teams to test their features as quickly as possible, we ran integration tests in those environments and then took advantage of Gitlab’s Merge Train and Merge Results features to merge straight into master with high confidence that our code was working, and if tests failed the merge would get kicked off the train and no developers would be blocked from getting their code out. Our number of high-severity incidents dropped to almost zero after switching to this new workflow, we increased the number of deploys we could do in a single day and had higher quality in the features we released. It was a fun problem and I feel like it had a major impact on the future of SimpleNexus’ development.

My last stop on my SimpleNexus journey was on a team integrating SimpleNexus and nCino’s systems. Our work was demoed at nCino’s conference nSight 2023. I got to work and learn from two extremely smart and talented engineers Michael Gardanier and Rich Corbridge and we knocked that initial MVP out of the park. I learned a lot from working with nCino’s engineering and operations teams as well. nCino has about 10x as many developers as SimpleNexus did so there was a lot to learn about how to manage larger applications, infrastructures, and development organizations. Through SimpleNexus, I’ve got to work at a company at the seed level to a large public company.

Conclusion

Working at SimpleNexus has been an incredible experience, and I am grateful for the opportunities I have had to grow and develop both professionally and personally. I can’t say I’ll miss the SOC2 and SOX audits, being back full time in the office, or trying to understand every possible way you can configure Kafka; However, those things along with the consistent mentoring and support I’ve received, the challenges that pushed me outside of my comfort zone, and the amazing people that I’ve been so lucky to meet and work with have set the bar extremely high for any future job.