Roles
Every few years someone predicts the end of a role. What actually happens is more familiar: the role gets refactored, restructured without changing what it accomplishes, and new bundles of responsibilities are born.
One note before we dive in: this story is about delivery roles, the people who build outcomes, from product managers to engineers to operations. The people doing AI research and building the models themselves deserve their own post. Though even there the pattern holds: scientist and engineer roles are blending, while new specialties like safety and interpretability are minted alongside.
I've watched it happen four times in one career. The generalist who did everything. The split into specialists. The blend back together when handoffs got too slow. The split again to build depth. The lifecycle never changed. What changed was who did the work, who owned the outcome, and how it got done. We never stopped testing; the tools changed. We never stopped building; programming languages evolved.
Let's look back at how the shape of roles kept changing. Three shapes describe every era:
Four redraws already happened. The fifth is unfolding now, and the pattern is steady enough to predict where it lands.
Programmer/analyst
M shapeOne person did it all: business analysis, design, architecture, coding, and deployment.
A company ran a handful of systems, each serving one department, so IT sat next to the business, close to the customer, and one role could own a system end to end.
Deep ownership and a direct line to the business. But every team ran its own tools and its own approach, and the cost of managing IT kept climbing.
Business analysts and application developers
I shapeRoles split. Analysts no longer coded, and developers no longer gathered requirements. Data split off too: data analysts and data engineers emerged, with tooling and methods distinct from application development.
Decentralizing the work into specialist roles is what enabled standards: consistent tooling and process, built and enforced by people focused on each area.
Depth and consistency brought cost and complexity down. The bill moved elsewhere: end-to-end outcomes now depended on handoffs, and coordination became work of its own.
Cloud and agile
T shapeAnalysis and engineering merged back into a single data engineering role. Everyone was expected to gather requirements, code, and test. New deep specialties were created alongside: ML engineer, platform engineer.
The move to cloud promised agility and faster time to market, and the handoffs between analysts and engineers were the thing slowing delivery down.
Teams focused on end outcomes with less context switching. Not everyone was equally strong at coding or analysis, so it fell to the org to upskill people and organize teams around complementary strengths.
Data as a product
Back to the I shapeData product management emerged, alongside renewed depth in engineering: from the world of pipelines and data marts to reusable data products that bundled metadata, governance, and observability for the whole organization. Product managers faced users and built roadmaps, with no expectation of producing code.
Data became a critical asset in overall company strategy, so managing data as a product became critical too.
Delivering results required handoffs, strong coordination, and intentional tradeoff discussions to stay aligned.
We are here: AI
M shape againRoles are blending again. The lines are blurring, and expectations will settle into an M shape, with one key difference: this time AI can be the specialist, going as deep in an area as anyone.
Just as with agile, the promise of faster time to market is real. AI brings speed inside every specialty and absorbs its complexity: an agent can draft the pipeline, the tests, and the documentation in an afternoon. What it cannot do is agree with your stakeholders on what to build, or vouch for its own work. The handoff, between people, and now between people and AI, is the most expensive thing left on a team.
New roles built around AI itself: AI governance, evals and verification, agent orchestration, human-AI coordination. The early signs are already here: the model-risk teams banks built to independently validate models are the blueprint for AI governance roles, the EU AI Act is creating AI compliance officers, and AI labs are hiring evals engineers. Understanding how AI works, and what it can and cannot do, will be table stakes for every role.
What this history tells us
1It's a spiral, not a pendulum. Each blend doesn't return us to the old generalist; it blends at a higher level, because tooling absorbed the layer below. Data engineering could fold analysis and engineering back together only because the cloud had absorbed the infrastructure work underneath. And the depth doesn't disappear when roles blend; it moves into platforms and new specialties. While delivery roles converged, ML engineer, SRE, and platform engineering were born. Platform engineering exists today precisely so product teams can blend. We are not going backwards. And this isn't just startups blending while enterprises split; I watched every one of these waves happen inside one company.
2The lifecycle never changes; who owns each stage and how it is done does. The stages of delivering something have stayed constant: Idea, Design, Build, Test, Deploy, Support, Scale. What shifts every era is who contributes to each one and the skills each stage demands: from code and product PRDs to prompt and context engineering, from test to evals, from runbooks to guardrails.
3Know the whole, not just one deep corner. Knowing end to end how something gets delivered is what matters, even in the areas that aren't your strongest: regulations, governance, product mindset, design principles, architecture. Breadth is what lets you deliver end to end. The more you know, the better off you are, no matter the role.
4Complexity tracks tooling, policy, and regulation. The complexity of a role is directly correlated to the tooling, policies, regulations, and environments someone in the role has to navigate. AI can orchestrate across them and abstract the complexity, but it does not remove the need to understand how the pieces fit together to produce the solution.
5Regulation resists blending, and AI will mint new specialists. Especially in industries where separation of duties is deliberate. AI governance, evaluation, and verification will likely become specialist roles of their own: the people whose job is to keep AI in check.
So how do you find your role next to AI?
Stop optimizing for one narrow skill. In every blend, the people who thrived understood the whole: how work moves end to end, and how their piece connects to how the business actually makes money. We call those people FDEs now, but every era had a version of them: the person focused on end-to-end outcomes, sitting close to the business, knowing how to piece things together to deliver value.
There is one new thing this time. At every stage of the lifecycle, AI can produce something unexpected, and potentially risky and costly. So the role next to AI is part builder, part supervisor: evaluating what AI surfaces, and judging when to trust the recommendation and when domain knowledge should override it.
In practice, that means:
- Learn the whole lifecycle, not just your stage. The more you know, the better off you are, no matter the role.
- Tie your work to outcomes. Not the outcome of a project or a story, but how it connects to how the company makes money and to its mission.
- Master the handoff. When execution gets fast, the value moves to the seams: connecting the dots across people, systems, and now AI.
- Expect to relearn the same work. Every redraw kept the stages and changed the skills: code to prompts, test to evals, runbooks to guardrails. Today's new skill is AI itself: how it works, where it breaks, and how to verify what it produces.
- Watch where new specialties are minted. Every redraw creates roles that didn't exist before. The next ones will be built around keeping AI in check.
The New Data Engineering Team
Want to talk about this live? Join me on DataCamp's panel with Pradnesh Patil (Altimate AI) and Neha Tharani (SCOR). We'll cover how the data engineering role is evolving and what that means for team structure, where AI agents add the most value in data engineering work, and the skills engineers and their leaders need in an AI-augmented world. Registration is free.
Register free