Behind The Tech #1 – Talos ATS Introduction

I wasn’t there, thankfully, for the conception (they say that 3 is a crowd) but in the autumn of 2015, mere months after the initial glint in Ste Rundell’s and Tom Coulter’s eyes, I helped to deliver Talos ATS into the world. 

Like all responsible guardians we had plans for its future and how to guide it through the perilous early years of its life, and I’m proud to say Talos is now 6 years old and maturing into an incredible product. 

This blog is about the principles, philosophies, challenges and solutions that enabled this success, and this, our first post, is a brief introduction to those concepts. 

In future posts, we hope to go into much more detail on many of these points, and do deeper dives into some of our tech. For now, though, I hope this post whets the appetite a little, and I hope you find some of it interesting. 

Features over Infrastructure 

Developer effort is not a limitless currency, and it needs to be spent wisely. Luckily, there are only 3 products it can purchase. We can spend it on infrastructure, we can spend it on fixing, and we can spend it on creating. 

At the beginning of a product’s life, there are few features and few users and so there are few things to fix and often little urgency to do so. This affords developers working in the early lifecycle the opportunity to spend much of their time focussing on either the creation of new features or the building of infrastructure. The question is how much to spend on each. 

Beyond the simplest systems infrastructure is necessary. If it were a house the infrastructure would be the foundations, and the beams, and the plumbing, and the electricity, the holes for the windows, the driveway, and the scaffolding providing temporary rigidity. Such considerations are not required for a short-lived ‘den’ but essential for a long-lived house. 

Infrastructure can be thought of as the skeleton of the system, those hidden beams and foundations that give the system its ‘shape’. The infrastructure will determine how different components of the system communicate, the types of data structures that the system will consume, the constraints on how the system can grow and scale, and the limits placed on the evolution of new features. 

But here is the fundamental difference between software and housing: software grows, and changes, and evolves. Software is more like biology than construction. A house, once built, can possibly be extended, or have some internal structural changes, but it cannot become a tower block. Software needs to evolve, the structures that hold it together need to flex, or be replaced, all whilst maintaining the integrity of the system. Once the software is out in the world it is ‘live’. We can’t kill the software to change it. 

So how much time should we spend on getting this infrastructure correct? As an ambitious company we planned for success. We planned for Talos to grow large. But large-scale infrastructure costs a lot of development effort. 

On the flip side of the equation, we needed features. A strong, robust, scalable infrastructure that takes a year of development effort will be of little use if the product has no features and therefore no sales. 

Features deliver commercial value; infrastructure does not. This is the quandary of early development, and the trick to solving it lies in treating infrastructure as an evolving feature in itself. 

What if we could find an architecture that encased each feature, rather than an architecture for the entire system. What if our architecture could be composed in a modular and consistent fashion, so that a small robust architecture grows and matures into a large robust architecture as a consequence of adding new features? 

If we could find such an architecture then we wouldn’t need to spend development effort building enterprise-level infrastructure from the outset. We could focus on building new features. 

In future posts, we will delve deep into our architecture and show how we achieved this. For now, suffice it to say that our first principle became ‘features over infrastructure’. 

Pragmatism over Perfection 

There is an old saying that suggests that if we shoot for the moon the worst that will happen is that we will land amongst the stars. Not only does this betray a staggering misunderstanding of astronomy, it completely neglects the effort required to shoot for the moon. If all you need is to get to the end of the road then building a rocket is overkill. 

When we design software, we imagine hitting the moon. Unfortunately, if we fail, we won’t land romantically amongst the stars, we will burn up on re-entry. Again there is a tension between our long term moon landing goals and our short term goals, and again we need to find a way to resolve this tension. 

The approach we adopted is to solve only those problems that are in front of us right now, to produce only the features that are required right now. This means that we don’t chase the optimal solution for every problem, rather we solve the immediate issues with a constant view on what may be required in future. 

We do just enough engineering to allow us to create a maintainable solution without burdening ourselves with a level of engineering that would be burdensome for our current team and overkill for our current demand. 

For example, we don’t currently use microservices even though there are a great many books and articles and videos proclaiming microservices to be the solution to all life’s ills. I mean, Netflix is built using microservices so they must be great right? But we are not Netflix. We don’t have a team of hundreds of developers building a system to provide hundreds of millions of users across the globe the ability to stream constantly from our servers. So why would we use engineering practices suited to large teams building highly scalable B2C software? 

We are a small team focused on building B2B software primarily for UK businesses, so our engineering practices are currently aimed at that level. When we have Netflix style problems then we will consider Netflix style solutions. 

This pursuit of pragmatic solutions leads us to our 3rd principle…  

Iteration over Planning 

I once worked at a company that had been planning a large systems upgrade for years. Senior tech people had come and gone, and analysed the migration from every angle and floundered on one problem after another. No migration work was ever started because every plan seemed to have problems. But sometimes action trumps planning, and action often just needs a general direction. Imagine being in unfamiliar territory and only having a rough idea of the terrain. 

You can see mountains and valleys and woods, and you know the rough direction in which you wish to head. The best approach is to take a couple of steps, evaluate what happened, assess if you are now closer to your goal and if you are then take the next couple of steps. 

Sometimes you will need to backtrack, and there might be the odd occasion where you go around in a circle. But surely enough, eventually, you will overcome or avoid each of the obstacles on your path and reach your goal. 

Detailed planning, therefore, should be focussed on immediate priorities, but as the planning moves further out into the distance, and the resolution of detail becomes blurry, all you need is a general idea that looks promising from the perspective of your current position. 

In this way, we iterate our way to a solution. We pick a direction, plan small, and move. Then repeat. Over and over. Ultimately we will get to where we want to be, and when we get there we can review the journey as a whole and learn from any mistakes we made and identify any decisions we could have made better. 

Autonomy over Authority 

This level of iteration requires a lot of detailed decisions to be made and rescinded very quickly, and an impediment to this would be a command and control structure where every decision requires senior authorisation. 

So rather than a top-down approach, we aim for a bottom-up approach where shared ideas, shared understanding and shared goals can drive individual autonomy.  

Caricatures of software engineers often involve people sat typing furiously at keyboards, headphones on, hipster coffee in hand, occasionally having a break to play Fortnite – and some of that is true, some of the time. But engineers are problem solvers, and most of the enjoyment of engineering is in solving problems. Engineers can be seen as glorified typists who convert solutions into code, but we see engineers as people with the ability and the autonomy to creatively solve problems using code. 

It’s a subtle but important shift of emphasis, for it puts the control back in the hands of the engineers who can then make decisions, create solutions, and move quickly. 

But with so many decisions being made so quickly, how do we ensure consistency? 

Knowledge over Process   

One way of course would be to provide rigid constraints, in the way of processes, that must be adhered to. Processes however are just one way, albeit the most common way, of ensuring conformity – but there are others. One other way is to surround your team and business with a shared culture, give people knowledge and options and then empower them to make decisions that align with commonly understood shared goals.  

Here is a trivial but illustrative example. Like most software companies we have a ticketing system on which active issues are raised. Some of these issues will eventually reach 3rd line support, which is to say they will reach an engineer for resolution. 

Now the first thing an engineer will ask for on receipt of a ticket will be the reproduction steps of how to recreate the issue. Or at least that’s the first thing that the engineer will ask for with many issues. But for others, reproduction steps are not necessary. Perhaps the ticket is asking for a specific action that only an engineer can do. Or perhaps the issue relates to a question from a client. Or perhaps there are many such reasons why a ticket may not require reproduction steps. And we could try to enumerate all such reasons and encompass them in a ‘ticket creation process’ that people raising tickets must learn and adhere to. Or we can teach people why engineers need reproduction steps, and what the ticketing system is trying to achieve and then let people work out for themselves whether a ticket needs them or not. We choose the latter approach. 

This leads on to our 6th principle…

Thinking over Heuristics: ‘Shuhari’

There is a concept in Japanese martial arts called Shuhari, which roughly translates to “to keep, to fall, to break away” and this concept is used to define a student’s level of martial arts mastery. “Shu” means to “obey”, and students at this level are beginners not yet equipped with enough understanding to make wise choices. Such students are best improved by repetition of good practice guided by simple heuristics. 

“Ha” means to “digress”, and students at this level have learned that sometimes the heuristics can be ignored and rules broken. They have enough experience to realise that simple rules of thumb cannot cover every situation and that certain situations are best dealt with by departing from the guidelines. 

“Ri” means “separate”, and this is the level of a master where the heuristics can be discarded entirely, the master has fully absorbed and internalised all of the lessons and is wise enough to make the correct choices alone. 

Software is awash with rules of thumb, often abbreviated further into acronyms: DRY, SOLID, KISS, YAGNI, etc, and worse (much, much worse) these phrases are often used as the sole justification for a decision. On your intellectual journey to a solution these acronyms should be thinking guideposts, waypoints that need to be considered but (and here is the important bit) can be ignored and decisions should not be based solely on the heuristic but by understanding the lesson it contains. 

Thinking is hard, and heuristics provide the comforting allure of someone else already having done that hard work. But unless that other person was sitting where you are now, faced with the same problem, in the same programming language, with the same team, and the same goals then there is simply no way that the guidance can be guaranteed to be applicable to your situation.  

So we encourage our engineers to think through solutions, and to diverge from accepted wisdom if that is the solution we require. We aim to be Ri, not Shu.  

Simple over Easy 

Rich Hickey, the inventor of the Clojure language, once gave a talk where he contrasted the concepts of simple and easy. Rather than being synonymous he elucidated the following distinction: 

Easy focuses on effort. An easy solution minimises effort. Simple focuses on complexity. Simple solutions minimise complexity. 

There is a misunderstanding that many people (including, sadly, software engineers) have about the goals of software development, which is that the goal is to produce working software. Of course, that is a goal, but it is not the only goal. Many people can produce working software, because getting something to work is often quite easy. The challenge lies in getting it to work in a way that is readable, maintainable, performant, can evolve with the changing needs of a system, and that can scale as the demand on the system increases. 

So, sometimes simple can be hard. Minimising complexity can be difficult but we should search for the least complex solutions because such solutions, once found, are almost always easier to deal with in the long term.

We aim for simplicity.

Check out tech careers & other vacancies with Talos360

Want to join our amazing team?

Talos360 is an exciting, forward-thinking community of tech, recruitment, and marketing experts. We have a range of fantastic opportunities for talented people like you to join our fast-growing team.

From our modern base-camp offices in Warrington, we deliver our bespoke talent technology, tailored hiring campaigns, and employer branding packages to over 600 happy customers nationwide. Our workplace is an exciting, relaxed atmosphere, with its own gym facilities, bar, and multiple breakout rooms – and what’s more – we’re pet-friendly!!

Discover your future career with Talos360 – check out our careers page below: