Climbing the data career ladder, and descending it again

First-hand experiences from a data professional that climbed to director level and went back to being individual contributor.

The fields of data, machine learning (ML) and artificial intelligence (AI) have made incredible leaps over the past decade. Having worked in these fields long enough, I've witnessed first-hand what it can mean for a data professional. It largely shaped my career path. One moment I was a fresh graduate in a junior role, the next I was deploying state-of-the-art ML models adding hundreds of millions in USD revenue to leading global merchants (Spotify, Microsoft, Meta, etc). Further down the line I was a director of engineering setting engineering strategy and responsible for multiple teams.

But what happens when you realise the view from the top is not as fulfilling? I went through the experience of both climbing the data career ladder and descending it again. This is my story and my perspective, a climb to being a manager of managers and the perhaps unexpected, but ultimately rewarding, descent.

TL;DR

Are you just interested in a summary of the learnings of this newsletter, here you go:

• When starting your career, getting experience is most important. It might not even matter in which industry or the depth of the role. If it aligns with your interests and you can grow, go for it.

• Working across multiple different data roles can be invaluable. Working across data science, data engineering and ML engineering can give a unique perspective that not many others have. It makes you a more attractive hire and especially in smaller teams this can be quite helpful.

• Climbing the career ladder might not give satisfaction. While many strive for climbing the career ladder one way or the other, in my case, climbing the managerial career ladder did not just result in satisfaction. In some cases going back go being an individual contributor (IC) might make sense.

• Managers play a large role in employee satisfaction. Being a manager brings quite some responsibilities, and not everyone is cut out for it. While I enjoyed the people part of being a manager, there are managers which are just after power. Before becoming a manager yourself make sure you have the trust of higher management for them not to question your every move.

1. Starting my career in the banking sector

My journey began at one of the largest financial institutions in the Netherlands; ING. Starting a career in such a big and well-established organisation can be a blessing and a curse. In bigger organisations things are usually arranged well, from learning budgets to well defined career paths. On the other hand, you might have to deal with complex hierarchies and work at a slower pace than you'd like. But first, how did I end up there?

At university I obtained an undergraduate degree in mathematics after which I got a graduate degree in financial mathematics. During my studies I specialised in probability theory and statistics but also did various courses in econometrics and quantitative finance. The latter helped me establish a network in the financial sector, as a few professors were also working in this sector. Ultimately my network helped me land me an internship at Rabobank. At this internship, I learned how it is to work for a corporate and decided that a career in banking wasn't so bad.

My first role in model validation

Using the connections I made during my studies, I landed my first full-time role in model validation at ING. Model validation, what does that even mean though? My job was to assess, analyse, and ultimately approve or reject machine learning models that others built. Think of it as being a quality control expert for algorithms, with the power to say "yay" or "nay" to someone else's work. The work mainly evolves around writing detailed model validation reports that are used for internal, primarily model improvement and model quality assessment, and for external reporting, auditors and external parties like the European Central Bank might go through these reports.

Already from the start, it was quite an overwhelming experience. I was pretty much the only one in my team without a PhD degree, and everyone had heaps (think decades) of experience. This forced me to quickly develop a hyper-critical eye and, crucially, I honed my writing skills. Crafting reports for regulators and internal stakeholders demands precision and skill. I learned a lot during this time, from a people perspective (how hierarchy works and how political games are played) and from a technical perspective (how ML models are built and to judge them).

Nevertheless, after a year I was itching to build my own models, not just critique. I transitioned internally to a team that builds and maintains ML models, diving headfirst into building and coding them. This is also the time where I transitioned from mainly using Matlab and R to Python, turning my mostly academic knowledge into real-world solutions. What I take from my first two years of working experience

• Try to establish connections already before trying to find your first job. This can significantly help you find a first job. Internships, applied courses and boot-camps are a good way to do this.

• When you strive for a career in data it might not matter a lot which type of role you land. Typically it's most important that you get experience first from which you can determine which industry or role might suit you best.

• It's okay if you don't know certain things in your first job. The key is to stay curious, take initiative and keep asking questions. It's also pretty vital to find a workspace where you can learn from others (preferably more senior) colleagues.

2. The transition to ML generalist

During my first two roles, I became much more interested in the world of data science (DS) and machine learning (ML). It was also during this time that I was further developing my coding skills, from “okay at solving university problems” to “can write a decent Python script”. But I also realised that to accelerate learning and getting a better grasp at coding, a switch in role or company might make sense. When I started to research options, I quickly realised there's an abundance of data roles

• Did I want to mostly perform data analysis and be Data Analyst?

• Would it be wiser to dive more into the world of data pipelines before going deeper into ML and become a Data Engineer?

• Do I continue the path of developing ML models and become a Data Scientist?

• Is coding what I enjoy most and should I attempt to become a ML Engineer?

How did I make this choice? I didn't really. I looked at opportunities in which I could explore what suited me best. I interviewed at a few different companies and made a choice.

The world of Fintech

The choice fell on a Fintech company that was scaling a lot; Adyen. It combined the world of tech and finance and it had heaps of transactional data which were largely untapped. Back then it was a company of about 600 people just fresh off its IPO. During the interviews it was clear that they were still defining their data strategy and there wasn't a very established data culture yet. This appealed to me, as I also was still determining which role suited me best.

But what I didn't realise: It was a completely different beast than the traditional banking world. And much of it was for the better, there was a lot more pace and engineering/tech wasn't seen as a cost centre. This is a fundamental difference that tech companies have compared to many non-tech ones. Often in non-tech companies, money spend on engineering resources is highly scrutinised over and closely budgeted as often it's not deemed "core business". While tech companies have the realisation that investing in engineering shouldn't be seen as a cost centre but rather as a profit centre.

A recommended read on the topic comes from

Gergely Orosz

in which he describes the difference how various companies look at investment in software engineering:

Profit Centers vs Cost Centers at Tech Companies

At the time Adyen had just launched its first Proof-of-Concept (PoC) for a data platform, a single yet powerful on-premise machine running Spark. Adyen was (probably still is) fully on-premise with its own data centres and hardware. But that single machine, already showed great potential. It gave me my first glimpse of the transformative power of using better data tooling (such as “distributed” data processing). Queries that once took forever now returned results much quicker, offering 5x, 10x, or even 100x speed improvements.

It's around this time when I learned a lot about data platform engineering. For a while I helped maintain the first iterations of the data platform, and while it sometimes meant working through the night, I definitely enjoyed it. One of the biggest undertakings was to migrate from a custom data orchestrator built on Jenkins to Apache Airflow. It's where my preference and experience with Airflow started. Aside from the data platform engineering work I was also doing quite some analytics engineering for product teams. Showing teams how they can leverage their data by building their first data pipelines and dashboards can feel quite rewarding.

Becoming a ML “generalist”

But my passion still was machine learning. At the time, anyone in DS wanted to become a "Data Scientist Unicorn", a data scientist that combines soft skills with hard skills in programming and ML. This wasn't exactly my aim but it did make it more difficult to move to a DS role as many were striving to become the next DS unicorn. Eventually I made the leap to the DS side, just as the first "real-time" ML projects were kicking off. An intern had already laid the theoretical groundwork, which was invaluable (Boris Mathijsen, thanks a lot!).

But aside from that it was baptism by fire. Almost everything had to built from scratch. This is when I learned the core principles of ML engineering and MLOps. Given the lack of a MLOps team or even a MLOps platform, a lot of the foundation had to be built from the ground up. In my case I handled everything from building the data pipelines, making a notebook PoC, handling model deployment, to live performance evaluation.

The results of the ML project spoke for itself: It boosted the authorisation rates by as much as 10% compared to rule-based systems. For Adyen's merchants, the likes of Microsoft, Spotify, Meta, and more, this meant hundred's of millions (measured in USD) in additional revenue. Real-time scoring of ML models was a game-changer for Adyen, though now everyone just calls it “AI”. It's because of this experience I call myself a ML generalist, having practised hands-on DS, MLEng and MLOps.

If you're curious to my experiences during this time in more detail, here's a blog post and talk at Spark & AI Summit (now known as Data & AI Summit by Databricks). Here are a few key learning's during this time

• When unsure which data role to pick, try to find a company or role that let's you explore different types of work. Companies that don't have an established data culture yet, like start-ups or scale-ups are great places for this.

• Many companies that are dealing with big datasets can benefit a lot from a data platform. Nowadays we have tools like Snowflake and Databricks to handle most of the heavy lifting, but there are still many companies that do not fully harness the power of a well-designed data platform.

• Working hands-on on different parts of a ML project can be invaluable. It can give you a unique perspective that in many companies is separated into many different teams and roles. This also makes you a lot more attractive to hire and opens a lot more possible roles you can apply for.

3. Climbing the ranks: From Tech Lead to Director

When the value of building out Adyen's ML capabilities was clear, the team I worked in went from two data scientists embedded in a back-end-heavy team to a dedicated multi-disciplinary five-person ML product team. But before diving into my experiences, let me first explain the engineering team structure and why this was a big step forward.

Engineering leadership and team structure

Many starting tech-oriented companies focus primarily on hiring back-end or full-stack engineers (source). If a company is successful and is to scale it's often these engineers which take up engineering leadership roles. This can lead to situations where engineering leadership primarily comes from a back-end specialisation. It's rare that data, AI/ML, front-end or web engineers take up these roles.

“When you look at most CTOs and engineering executives at most tech companies: their engineering background is more often than not backend engineering.”
-

Gergely Orosz

(source)

For data & AI/ML engineers specifically this might attributed to the fact that many companies only start hiring for data-oriented roles at a later stage. Which also means that many of these companies will only built a data culture at a later stage. In other cases companies entirely split their data org from their engineering org, which in my opinion is the wrong choice as there's is a lot of overlap between data, AI/ML and traditional software engineering.

This is probably one of the reasons that engineering teams at tech companies are often multi-disciplinary in nature. This means that a team usually consists of multiple different engineering roles, e.g. front-end, back-end, data, AI/ML, MLOps, SRE, database, etc.

My advice is, it's important to know back-end engineering if you want to increase your odds of getting executive (CTO) or engineering leadership roles (VPs, Director). Being more of a generalist than a specialist data or AI/ML engineer definitely can increase your chances.

Becoming a Tech Lead Manager

I was asked to lead the newly established ML product team as a Tech Lead Manager (TLM). This is a dual role that combines technical leadership with people management. For the same reasons as explained before, at the time, in Adyen almost every manager was coming from a back-end background. Therefore this was a significant moment for me and for people in data roles at the company. It took some convincing, but at the age of 27, I became one of the first TLMs from a data background in the company.

The TLM role gave me the opportunity to roll up my sleeves and do technical work, while also mentoring and guiding my team. We made great progress in migrating the previous rule-based logic to real-time ML. The team also made huge strides when it came to building an experimentation platform, which eventually even won the internal innovation award. I genuinely enjoyed the experience, but soon rapid hiring changed everything for me and the team. In a year’s time the team grew from five to fifteen, and my engineering time significantly decreased.

After the first year, I was asked to lead a similarly sized team in a different domain. This meant less hands-on work, more engineering strategy, and a steeper learning curve as I familiarised myself with a new domain and code-base. Managing a larger team, it eventually grew to almost 20 people, also brought new challenges, from personal struggles to interpersonal conflicts. It was also different to manage people I did not directly work with before, as a manager this means you need to both provide a safe environment and obtain the trust of your reports.

Becoming a manager of managers

Two years in, I was promoted to a Director level role. I was no longer managing individual contributors, but managers. It's a completely different ball-game. Influence becomes indirect, strategy supersedes day-to-day tasks, and the political landscape intensifies. Organisational growing pains became obvious, and hasty decisions affecting me and my teams made above my head made me question my own path.

• Had I made the right choice by going fully hands-off?

• How could I stay relevant in a rapidly evolving technical field if I wasn't actively coding?

• How can I manage people that are much more experienced than me?

• Do I enjoy playing along with the political games?

The view from the top is not always pretty they say. Especially the political games, constant course corrections by higher management and decreased motivation in the teams took its toll. It's often a manner of luck whether a manager enables you or holds you back. More often than not, in my experience, it's the people who solely seek power that exhibit the most egoistic and non-emphatic behaviours when they climb the ranks. That's probably a reason that some executives and people in higher management are not the most empathetic people managers.

I learned that it's a red flag when higher management employs fear based tactics together with unreasonable expectations. And I also realised that, to be a successful manager you need to be trusted and backed by management, if they question your every turn and overrule your decisions, you're setup for failure. My experiences were tough but did come with great learnings, retrospectively I should've stand my ground more and evaluate whether taking up a certain role is worth it.

Then, an opportunity arose. Former colleagues that started a new company and needed a founding team member to build out their data capabilities. The decision wasn't easy. It meant a significant pay cut but also a return to hands-on work, and a chance to build something from scratch.

Ultimately, I went for the opportunity.

4. Descending: A return to the roots

The year I spent at the start-up was transformative. I re-learned the joy of coding, built full applications from inception, and architected a data platform from the ground up. It was a reminder of why I fell in love with the coding and data engineering in the first place. It's here where I went back to the basics and was handling a lot of stuff myself, from deploying cloud infrastructure to handling ad-hoc data requests.

If you're interested to what I did during this year, here's a recording of when I spoke about this at PyData Amsterdam last year.

With the first start-up year under my belt, I realised that I wasn't applying a lot of the knowledge I had obtained in the DS & ML field. In this start-up the focus was more on building reporting capabilities rather than AI/ML applications. My personal life also started to play a larger role as just before joining the start-up I became a father. Being at least 4 days but preferably 5 days in the office was no longer preferred from my perspective.

A brief return to the corporate world

I decided to take a leap and went back to the corporate world. On paper, it seemed like the perfect fit. A greenfield ML project, with lots of potential, a 4-day workweek and the choice to either work at the office or from home. But reality hit fast after joining.

After having seen the breadth of things in data and ML engineering, I was now limited to just a small piece of the puzzle. Over the years I had obtained knowledge across different data domains, but the corporate structure demanded specialisation. And while I got a lot of energy in mentoring data professionals, it became mostly a job to "manage up", pushing for things you knew were true while other people needed convincing.

I realised I didn't want to invest yet another few years of climbing the corporate ladder, navigating office politics, and not being able to get most out of my time. I decided I wasn't ready for this type of commitment. Perhaps I should've looked for a role with more influence, which typically in these companies means going into a management role. But I was more keen on keeping hands-on for now.

Back into start-up life

I was fortunate to come across another startup; Palm. This time I joined as an early member of a fully remote team. It's crazy how life sometimes works but at Palm I feel that I can fully harness my experiences in data, ML and Fintech. Palm is all about building the next-generation cash and treasury management solution. A platform which requires heavy lifting in both data, AI and traditional ML. And even though I'm applying my existing knowledge I'm still learning a lot in back-end engineering and the applications of GenAI/LLMs. I'll be speaking about one of the applications at PyCon & PyData DE in April this year.

This is where I am today I'm enjoying every moment. Where will I go next? Time will tell, but I believe that this environment suits me better. Will I go back to management? Probably at some point, I believe the Engineer / Manager pendulum is a powerful concept.

“The best frontline eng managers in the world are the ones that are never more than 2-3 years removed from hands-on work, full time down in the trenches. The best individual contributors are the ones who have done time in management.” - Charity Mayors (source)

There's quite some material written on the Engineer / Manager pendulum. Here's a blog by Charity Mayors on the topic. In this pendulum people would swing being manager and individual contributor roles a few years at a time. This way you keep your technical skills relevant while it enables you to still make career as a manager.

5. The lessons I learned along the way

Here's a compilation of the lessons learned in the past few years

• When going into management, keep in touch with the tech: While becoming a manager can be satisfying from different perspectives, it's crucial to stay up-to-date with technical developments. It's also often easier to find a new role as individual contributor (IC) than as a manager.

• Titles and compensation aren't everything: Fulfilment in your day-to-day work means more than a mere title and a large total compensation.

• Find your environment: Corporate environments can be great for some, but limiting for others. Find the environment that allows you to thrive, I encourage everyone to both experience small companies and bigger ones.

• Manager's have a large role to play for their teams: Manager's can make or break a workplace. Beware of this and the culture that is set. Try to steered away from managers that are just after their own agenda.

• It's okay to change your mind: Your career path doesn't have to be linear. Don't be afraid to pivot and explore different opportunities.

My journey up the career ladder was valuable, but it taught me that the destination isn't always the most rewarding part. Sometimes, the best path is the one that brings you back to your roots. And perhaps I'll attempt another climb of the ladder some day.

Thank you for reading another edition of The Data Canal. To support me in writing my content, please do not forget to subscribe. If you’ve missed previous editions, here you are to must-read editions.