The Problem of Poor Tech Leadership at Bank of America

At Bank of America, I ran into the worst codebase that I had ever seen. It was written in the Scala programming language, and I love Scala, but this codebase was like a wooden ship that had gone without maintenance for so long that the wood was rotting and it had taken up water. Morgan Stanley also ran into issues with its Scala codebase, but this was worse. Java and Scala are two different programming languages and the way you run a Scala project is different than the way you run a Java project. Java is backwards compatible and Scala is not because Java is fully matured and Scala is not. I believe this delay in maturity was a deliberate design decision made by the creator and head of the Scala programming language, Martin Odersky, because without backwards compatibility, it is possible to retroactively fix flaws in the programming language. That being said, without backwards compatibility, and with a totally different programming paradigm, a certain additional degree of care must be taken (including additional maintenance), and the people who handed me this Scala codebase DID NOT take this additional degree of care and DID NOT do any maintenance. Imagine a president who ran up their country’s debt up to the absolute max before handing it over to a new president from another political party. Or imagine that you bought a used car that was on the verge of breaking down. There was a lot of technical debt to be paid back and a lot of maintenance and updating that needed to be done. The old ship (codebase) must be repaired before it can set sail again (i.e. before new features can be added), and this codebase was in a complete, utter state of disrepair. It also had many security holes, which to me went part and parcel with how flawed and shitty it was from a software perspective.

Looking at the logs, people who originally built this codebase no longer worked at Bank of America. I looked one person up on LinkedIn and he apparently left to work at Facebook. The last code change in the build was over three years ago. Apparently this codebase had been built sloppily and neglected for years before being handed off to unsuspecting me (plus some other people), and I couldn’t even contact the people who wrote the code that I was working on because they either did not work here anymore or they were in India. Basically, they left a complete and utter dysfunctional mess, and now it was my job to continue where they left off. Oh boy.

The codebase was the worst thing that I had ever seen. None of the tests ran and passed. The build was completely messed up. Everything was deprecated and needed to be replaced. This codebase was like a car where the engine, headlights, oil, carburetor, and every major component was broken and needed to be fixed before the thing could make forward progress again. The fact that they left the keys in the ignition (a massive security hole) was the least of my problems – if they knew exactly what they were doing the security hole wouldn’t have even been there, but they winged it – poorly. I wing things, but I first do my research, plan it out, and execute it meticulously, but these people were sloppy and didn’t do their reading. My main concern at the time was that the codebase was so bad that it was virtually impossible to make changes or updates to this codebase without redoing almost the whole thing. The codebase was built on top of an HTTP library that was deprecated (Spray), and they were supposed to have switched to a different one (Akka HTTP) years ago, but apparently these people did not get the memo. The memo was posted on the Spray website for years, like so:

If you look at the website, it is clear that you were supposed to migrate from Spray to Akka HTTP, but they didn’t migrate. The people who created Spray tweeted and posted in multiple places that all the users should migrate, but the people at Bank of America apparently didn’t get the memo. The Bank of America firewall blocked all the social media websites, so one possible explanation is that the people who created Spray were posting and posting, but nobody at Bank of America got the memo. This codebase was built on top of rotting, deprecated shit, the people who built it were no longer available, and now it belongs to me. Oh boy. On top of that, the bank employees who used this codebase had no idea that it was supposed to have regular maintenance and updates. The people before me didn’t really know what they were doing and they just left the thing, and now I am supposed to add new features to it. This never goes well.

One difference between Java and Scala is that because Java is almost totally matured, it doesn’t change as quickly, so things don’t get old or deprecated as fast. It doesn’t change or move as quickly, so it doesn’t need as much updates and maintenance. Bank of America has these ancient codebases written in very old versions of Java, plus they had mainframes that were running code written in COBOL. They were very behind the curve technologically, and this particular project was constructed by a startup like organization that was acquired by or inside of the Bank. The startup type people apparently picked the most experimental, bleeding edge, pre-release technology and then abandoned it and then the bank people took it over. The only problem was that the startup people were good enough to be hired by a company like Facebook, and the bank people were in India and I could barely understand their English. This codebase was in such bad shape that it crashed constantly, ran terribly slowly, and was impossible to work with. The people who worked on it (including the guy who left to work at Facebook) didn’t know what they were doing (perhaps because there is a learning curve, the functional programming language Scala is not meant to be coded in like C++ or Java, and also because the documentation was blocked), and it was the worst software system that I had ever seen. At least at Amazon someone code reviews your code before you push changes, so if one person disappears you can always talk to the person who did the review, but here there was zero review. The bank just kind of paid money for a thing and the people who were in charge of this thing didn’t what they were doing. For the tech people it was like “the wild west”, and for the bankers, it was constant frustration because the software kept breaking and crashing, and it took FOREVER for them to see new software features.

A while back my mom told me a story about a guy who was assigned a software system to work on. This software system was like a trading program that had previously been built by one guy. The trading program made a lot of money for the bank, but then the guy suddenly quit. They needed to make changes to this trading program, so they grabbed this other guy and told him to continue where the previous guy left off (this isn’t exactly what happened, but you get the idea). I could imagine sitting down at this other guy’s trading program and feeling like it was written in Chinese and also busted in 50 different places. No, it wasn’t literally written in Chinese, but it was unintelligible. There was no code review, no oversight, and no responsibility. The previous person just abandoned the thing. Now you have to either learn Chinese and fix all the things that are broken or start over and create a new trading program that trades just as well as the old one. Normally you would do the former, but in Scala there is no backwards compatibility, so you can’t just abandon it for years, especially if the stuff that it was built on is already rotten and deprecated. The whole thing needed to be migrated to a new system and redone, and apparently the only person at Bank of America who had any idea how to do this was me. I was now leading the charge (without issuing a single verbal order), and people almost twice my age were coming to me.

Scala is a bigger, harder programming language than Java. If you’re not a great programmer, you could write Apache Spark (data processing script type) jobs because you don’t actually have to fully know the Scala programming language to do that, but you’re going to run into problems if you try to do a big, complicated backend system with no guard rails. These people who came before me tried to do something big and complicated, and they wrecked the thing and left it in god awful shape before abandoning it. The setup instructions hadn’t been updated in like four years and I basically wrote entirely new setup instructions and provided them to everyone, bought everyone technical books, and planned out a migration path. At Bank of America, there were people of all skill levels from people who had no idea what they were doing up to me (and some people who I think were pretty close to me but didn’t work as passionately), and these people were spread all over, from New York City to India. They totally screwed the whole thing up. Heck, in order to talk to another person who was working on the same code as I was, I had to wake up early in the morning and get on Skype because I was in the US and they were in India. Apparently Bank of America never heard that coworkers are supposed to be in the same time zone.

Ideally, you have someone running the show who is really smart, thoughtful, can lead without having to yell orders, and who knows what they are doing (i.e. the technical lead or software architect). Bank of America didn’t have someone like that, at least not on the codebase that I was working on, so I kind of filled in for the absence of that person. Before I worked at Amazon Web Services, and everyone was good or better and they were all in the same building. Here there were people ranging from “I have no clue what I am doing” to “I have this super detailed plan to fix everything, but I don’t have approval because the person who is supposed to give me approval doesn’t actually understand the problem”. You can guess which one I was.

In software engineering, this state of being stuck in a problem is called “The Tar Pit”, a term taken from the book “The Mythical Man-Month“. This group of programmers was in “The Tar Pit” because there was nobody in charge who knew what they were doing and the thing was in a total state of utter disrepair. There was nobody with prior knowledge and experience to enforce structure and order. Nobody read the manual, and because of the firewall, the website with the manual on it was blocked anyway, so I had to use my phone to read it at work. I used my phone, but apparently the people before me did not. They just winged it, and they did it terribly. I took control of this situation, and then my recruiting company, TekSystems, kind of mysteriously fired or transferred me from Bank of America, and then the Russia Investigation happened, and I got thrown into some stuff. From there my life became crazy.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s