The role of data engineering post-COVID-19
The corona virus has changed consumer behaviour more significantly than any other event in this century and as a result, the job market has been turned upside down. The demand for certain products and services has vanished from one day to the other, but at the same time a handful of sectors are growing faster than ever before. While in the middle of setting up Pipeline Data Engineering Academy, we've been caught up in this unexpected turmoil as well, so I thought to myself, the only right path forward is making sense of what's next to ensure that the value we aim to provide stays relevant.
Tech is here to stay, but it's going to adapt
First, let's turn to Anil Dash of Glitch for some clarity about what technology really is and how we would expect it to behave during a crisis:
Technology isn’t an industry, it’s a method of transforming the culture and economics of existing systems and institutions.
It is a misconception that tech is an industry by itself, therefore we can't just make predictions or assessments about the companies we consider part of the Silicon Valley ecosystem. All verticals are going to be impacted differently as consumption is quickly adjusting to the new rules of living under lockdown (i.e. travel vs. streaming media), while within those verticals winners will emerge on top of the losers corpses. Technology as a method, the main driver of growth in the last decade is forced to change, there is no way around it: if existing systems and institutions are adapting to the (post-)corona world, the approach of transforming their culture and economics is going to have to find it's new ways as well.
"Nobody ever got fired for buying IBM." Part #2
During the time of economic growth and general positive attitude towards the near to mid term future, corporates and individuals are willing to take more risks when it comes to investments. The pre-corona world at the stock market made a large bet on growth driven by emerging technologies which require a significant financial investment and long-term commitment from organisations, but this type of capital also presumes consumer demand (in the form of disposable income of households) and a general positive trajectory of the markets. This notion resulted in a crazy number of startup unicorns and what felt like the next industrial revolution: things were good working in tech, job security was not a topic on peoples minds. Some signals were there pointing towards an economic bubble waiting to burst, but being bitchslapped by a pandemic was not a scenario to be considered when discussing the upcoming fiscal year in the board room.
In times of financial distress and unpredictability, people start looking at what they have and how this measures up when hardship is knocking on the door. Decision makers are going to have one thing on their minds: avoiding unnecessary risks. In Google-speak:
By dialing back our plans in other areas, we can ensure Google emerges from this year at a more appropriate size and scale than we would otherwise. That means we need to carefully prioritize hiring employees who will address our greatest user and business needs.
A significant part of the innovation economy is high-risk high-reward, and all of us are witnessing the process of sobering/falling out of love with unpredictable results - also referred to as 'market correction'. Spending corporate money without demonstrating hard ROI will be nearly impossible, and this goes for hiring in particular.
Makes sense, right? Basically, your tech job will be only as good as the value of your measurable output, and data related roles carry significant returns for consumers (external) and decision makers (internal) alike.
The case for data and engineering
One of the building blocks of technology itself that enabled/drove digital transformation with exponentially increasing pace is data, and the players in various industries making more and more use of it (whether the economic value provided through leveraging technology and data is adequately measured by the stock market and company capitalisation shall remain a separate conversation). Taking a closer look at the various disciplines within the realms of a cross-functional digital product team (from agile coaches, design thinking facilitators, backend developers, UX designers, devops engineers etc.), it's likely that non-essential roles will have trouble flourishing as a consequence of the above when the main company concern is providing a cost-efficient and technologically robust core service, ideally based on data-driven decision-making. We are going to observe a general structural shift towards the maintenance and operations of existing tech products in exchange for developing new prototypes and "failing quickly".
It's back to the basics, and the mindset of an experienced data engineer is all about that. But don't take it from me, take it from McKinsey Digital recommending CDOs to focus on staying operational and ensuring business continuity as a first instance.
Data engineering is one of the most underrated but essential roles in a digital product team that is going to stay on the hiring list of companies working with tech. Before COVID-19, we've seen a 50% yoy growth in 2019 in the demand for engineers able to work with big data and a corresponding rapid increase of salaries, and although the growth is somewhat slower now, the crisis verified that the demand is robust.
This too shall pass.
If there is something the dotcom bubble and the 2008 economic crisis have taught us it’s that the economy is able to bounce back, it always has in the past. Whether it's a 'V'-shaped return or more like an elongated 'U', we'll learn soon enough. Regardless, when it comes to the years between the high-times, you better stick to the basics... and it seems like data engineering remains one of the essential disciplines companies working with data desperately need.