Pipeline Data Engineering Academy

Pipeline Academy on Hiatus

2022-06-22T00:00:00+00:00

It’s time to share some important news with you: we’re taking time off to focus on our health and families, the launch of new data engineering cohorts is on hold until further notice.

Health and family

Running a bootstrapped company in times of repeated economic crises and data industry vibe shifts is a gift and a curse at the same time. No surprises here: it can be highly rewarding and joyful, but it can be exhausting and stressful as well. It’s June 2022; after two and a half years of pioneering the data engineering and data architecture education, and after celebrating multiple amazing cohorts of freshly minted data infrastructure experts, it’s time for Daniel and myself to turn the heat down and allow ourselves to recharge our batteries.

Not a “Goodbye", but hopefully a “See you soon!”

The current economic shift will continue to have major (and hopefully sobering) impact on the way data infrastructure and data teams are managed, so challenges in this space are abundant. If you’re looking to become the data architect who can make sense of the promises of the “modern data stack”, who can design, build and maintain data infra that just works, and who understands what really matters and what doesn’t, you’re still at the right spot.

If you’d like to get in touch, feel free to email us directly, or if you’d like to be notified when a new cohort launches, please sign up here.

If you are looking for trainings for your organisation or partner up, just send us an email.

We hope to see you soon,
Peter & Daniel

Meet The Graduates: Guoda Paulikaite

2022-05-02T00:00:00+00:00

In this interview series we’ll share some of the stories that Daniel and I get to watch unfold at Pipeline Academy. Check out what our graduates have to say about the course, how they’ve tackled its challenges and what they are doing now with their new data engineering superpowers.

Peter: Can I ask you to please introduce yourself to the readers of Pipeline Academy’s blog?

Guoda: My name is Guoda, I'm a data analyst. I am Lithuanian, I am also a mother. I've learned sociology as my bachelor, which always will stay at my heart. And I'm curious about how to structure all the uncertainty in the unstructured world that we live in - this is something that drives me.

Guoda Paulikaite, Data Analyst and graduate of Pipeline Academy

Just recently I joined a company that was represented by one of the guest speakers at the bootcamp, and that has really given a boost to my career in this field. Now that I'm at Project A Ventures, I am absolutely having a blast, enjoying the work I do and, yeah... I mean… earlier I've used to have a very clear answer to your question, but now I think I'm at the beginning of something new where I'm not only a data analyst or a business analyst. At the moment I am a data analyst that supports many startups and ventures in their requests for data analytics topics. I am an analyst who works on solving data problems to deliver actionable insights for a variety of business goals. But also shaping their strategies of how they're going to scale and what data needs they will have in data analytics, structures or architectures.

P: So if I understand this correctly, you combine multiple skills in your current job: you are a data analyst, a data engineer, a data architect, and you're also involved in the business- and organisational development of various ventures, right?

G: Yes, indeed. Actually, data analyst sounds like I'm given data, I need to analyse it and that's pretty much it. But there is so much more that needs to happen before. And it seems that in the field of data there are many titles being created just to follow up with all new emerging required skills. So this is why I love my organisation for not worrying about this and just keep on focusing on the actual effort required. Yeah, leaving the content behind the titles flexible.

So, you're totally right. I'm part of managing projects, advising ventures on their high-level data architectures, investigating really specific technical solutions, thinking about processes, thinking about blockers. After all the goal is to analyse the data in a way that it's actionable, but it's just sort of the peak of the mountain. And I take a big part of what's underneath, working together with data engineers and different departments internally and externally to get to that that peak.

P: What was your intention behind joining the data engineering course?

G: Previously having worked in IT teams of large companies, my expectation for joining was to gain the confidence that I've always felt some lack of, you know, coming from a different academic background, even if I was very interested in data and I was good at pushing IT projects forward. I had this idea that maybe I could just go ahead and learn something in a non-academic, non-traditional format with more recent technologies.

I really had a good idea from your marketing materials that it will be a course covering many different data fields. Actually, this was my highest wish and expectation that I will get a good understanding on all of them to be comfortable in dealing with different architectures, different situations, different projects without going into extreme depth or getting lost in details of just one field, but having that grand overview.

P: How did three months of learning online feel for you?

G: I would say, I would have preferred a physical classroom much more, but it went much better than I expected, and this was I think due to the group size that we’ve had (P: cohort of 9 individuals). The fear that I've had about the online classroom was that I won't get enough feedback, or no one will notice my progress or when I'm struggling. Or, you know, just miss out on having fun building relationships with your peers.

But dealing with this was so similar to how you go about it in the real world: your colleagues will not be around all the time, they will be somewhere else geographically and you have to figure out how to talk to them effectively.

P: If you have to think back, it’s already been 10 months since you've graduated. What are your most memorable moments? Highlights, lowlights, good things, bad things from the course.

G: I do remember that the course was very challenging. It wasn't something where you sit back and relax and just get information stuffed in your head. It was many times so challenging that I was just asking myself how can I go forward. One of the highlights was how Daniel actually spotted this frustration and was able to manage it and bring me to take one step back together and say okay, this is all gonna be fine. It sounds quite straightforward and simple, but actually it's one of the things that you learn that the level of detail can be so overwhelming that it's hard to avoid getting lost. I think this is one of the skills that I've learned.

The amount of information and amount of complexity makes it difficult for you to grasp how the world can work, how planes can fly and bank transactions can run if everything's so complex. Then I realised that no one actually has a simple answer. It's just a matter of experience and willingness. You know, not to get lost and just continue. Fail and stand up again. And when I realised that everyone is doing the same, like everyone, even the most experienced developers, they do the same thing. They just go on and search for a solution.

P: Did your perception of data engineering, or the role of a data engineer change throughout or after the course?

G: It became much more down-to-earth than what I imagined before. I imagined that you really need to have a degree in computer science. I still consider that people who make that choice to study computer science get a really good advantage, a good basis. But after all, it's all about experience and curiosity — and if you have that, then the rewards are great.

P: Where do you think your career is headed right now?

G: I'm headed to a place where I'm confident with the uncertainty with different architectures that I've never seen before. There's always new emerging architectures, new technologies. Now I am able to have a conversation about them with confidence, and a structured way of approaching implementing them in projects, and managing the stakeholders along the way.

P: Confidence in uncertainty. I love this expression. Data engineers need a lot of this.

There are people out there who are considering embarking on this path of learning data engineering. Do you have an advice for them?

G: I have so many different tips that I could share. Some of them are more specific, some are more about not losing motivation and persistence. But overall, I think my number one tip would be to try to find a problem that you think you can solve, to have one project that you can work on throughout the whole academy. A tangible, small problem you want to solve, not an abstract, academic one. That’s a great starting point.

P: Thank you, Guoda.

Pipeline Academy Setting Trends at the EdTech Awards

2022-04-14T00:00:00+00:00

Finalists and winners for The EdTech Awards 2022 have been announced to a worldwide audience of educators, technologists, students, parents, and policymakers interested in building a better future for learners and leaders in the education and workforce sectors.

The EdTech Awards were established in 2010 to recognise, acknowledge, and celebrate the most exceptional innovators, leaders, and trendsetters in education technology. Celebrating its 12th year, the US-based program is the world's largest recognition program for education technology, recognising the biggest names in edtech – and those who soon will be.

This year’s finalists and winners were narrowed from the larger field and judged based on various criteria, including: pedagogical workability, efficacy and results, support, clarity, value and potential.

I am absolutely delighted to announce that our work has been recognised in two categories this year:

Pipeline Academy as finalist in the category Product or Service Setting a Trend
Daniel Molnar aka The Data Janitor as finalist in the category Educator Setting a Trend

"As events unfold on the world stage that seem to inch ever closer to a precipice unknown, we are reminded that the leaders and innovators of education technology have always worked on the edge," said Victor Rivero, who as Editor-in-Chief of EdTech Digest, oversees the program.

"The future-focused work they do is inspired by the infinite potential of all people to learn and thrive. It's pushed forward by the human spirit. It's the light that even through the darkest times always shines through," Rivero said.

On behalf of everyone who has been supporting the work of Pipeline Academy, we’d like to say thank you for this recognition. It is proof that our efforts for making the education landscape a more transparent and future-proof place for learners does not go unnoticed by the broader innovation ecosystem.

The Data Janitor Letters - February 2022

2022-03-02T00:00:00+00:00

Data engineering salon. News and interesting reads about the world of data.

The Unbundling of Airflow
Gorkem Yurtseven, Co-Founder, Features and Labels

A diverse set of tools is unbundling Airflow and this diversity is causing substantial fragmentation in modern data stack.

Rebundling the Data Platform
Nick Schrock, Founder, Elementl

A fundamentally new approach to orchestration that orients around assets rather than tasks.

Fundraising by data companies in 2021
Bence Arató, Managing Director, BI Consulting

2021 was quite an exciting year in terms of funding for both data startups and established companies. We tracked more than a hundred data-related funding events during the year.

Analytics Stacks for Startups
Jan Katins, Senior IT Consultant/Data Engineer, kreuzwerker GmbH

The stack should be relatively fast to implement (two weeks is possible), so you can quickly reap the benefits of having a data warehouse and BI Tooling in place or upload enriched data back to operational systems.

Make Your Workflows Better With This Classic Unix Tool
Adam Green, Data Engineer, Gridcognition

Make and Makefiles for data pipelines.

GDPR enforcer rules that IAB Europe's consent popups are unlawful
Irish Council for Civil Liberties

Google, Amazon, and the entire tracking industry relies on IAB Europe’s consent system, which has now been found to be illegal.

CNIL decides EU-US data transfer to Google Analytics illegal
NOYB – European Center for Digital Rights

Austrian and French Data Protection Authority: the continuous use of Google Analytics violates the GDPR.

Here's What's Wrong With GDPR
Shoshana Wodinsky, Data Reporter, Gizmodo

The EU's landmark privacy law, GDPR, was supposed to change the world of tech privacy forever. What the hell happened?

Unbiggen AI
Andrew Ng, Founder and CEO, Landing AI

"It’s time for smart-sized, “data-centric” solutions to big issues."

The Data Janitor Letters - January 2022

2022-02-22T00:00:00+00:00

Data engineering salon. News and interesting reads about the world of data.

We’ve only scratched the surface of the full potential for the data warehouse
Mikkel Dengsøe, Head of Data Science, Operations & Financial Crime, Monzo Bank

Why I think the data warehouse will become the control centre for modern companies

Git, SQL, CLI
Vicki Boykis, Machine Learning Engineer, Automattic

I’ve narrowed it down to three basic tools.

One Year of dbt
Adam Boscarino, Director of Data Engineering, Devoted Health

Over the last year, dbt has become a key piece of the data platform at Devoted and lived up to our wildest hopes and dreams. We have gone from a single proof-of-concept to 1,100+ models.

Why OpenMetadata is the Right Choice for you
Suresh Srinivas, Co-Founder, Collate

Is OpenMetadata pull-based, push-based, or hybrid? Again, all systems are mainly pull-based to integrate with metadata sources. We support push-based ingestion when it is possible.

The future history of Data Engineering
Matt Arderne, Product Engineering, Data & Analytics, focal

On Data Engineers and their place in a Data SaaS world.

Airflow Alternatives: A Look at Prefect and Dagster
Pedram Navid, Head of Data, Hightouch

We take a deep dive into Airflow, Prefect, and Dagster and the differences between the three!

Modern Data Stack: Which Place for Spark?
Furcy Pin, Lead Data Engineer, Younited Credit

This makes data-lineage more difficult, since dbt only lets us visualize the BigQuery part, while our internal “dbt for pySpark” tool only lets us see the pySpark part.

Why I Quit Data Science
Nirant Kasliwal, Analytics Engineer, Sundial

Question from a friend: I am interested in knowing how did you come to this decision of moving to SWE from DS/MLE.

Become A Better Data Engineer On A Shoestring (More Free Resources)

2022-02-18T00:00:00+00:00

A bit more than a year ago I’ve compiled an annotated list of the best free courses and learning resources that could help anyone to become a data engineer on a shoestring. We’ve received an overwhelming amount of positive feedback on it, so after a full year of running the bootcamp I sat down again and collected an other bunch of resources we’ve bumped into during the cohorts.

Experienced data scientists, ambitious data analysts, data-obsessed product managers and future-oriented computer scientist have joined Pipeline Academy in the last year with the goal of learning real data craftsmanship, aka data engineering. But even if you don’t identify as any of the above, but you’re planning to become a more well-rounded data professional, you’ve come to the right place.

Expect well-designed learning experiences that are mostly very accessible in terms of pricing, and learning outcomes that are very close to what the market demands.

Some guidance for this list:

If you haven’t read the first edition of our compiled list of learning materials, make sure to check it out first.
There are various different formats included here (video-based courses, books, podcasts, story-based interactive coding tutorials etc.), try to identify which ones suit your learning style the best.
Don’t skip the fundamentals: using Python, SQL and the command line are essential for data engineers.
Integrate learning into your weekly schedule: try sticking to what you’ve started.
Share your experience and recommendations with others, it’s really difficult to find the right courses in the forest of mediocre Medium posts and useless certifications.

General

The Data Engineering Podcast and The Python Podcast.init
Outstanding podcasts of Tobias Macey on the mentioned topics.

Data Engineering, UC Berkeley, Spring 2021
University course ran by industry players with experience, although one has to reflect on the fact that data engineering does not happen in notebooks.

Technically
Technically sends out engaging, simple explanations of technical concepts that are useful for your day to day job and fun to read. Start with the SQL and the AWS explanations.

Explain the Cloud Like I'm 10
Beginners will find the cloud explained from the basics. Little prior knowledge is assumed. You will find lots of pictures, lots of examples, and many somewhat questionable analogies in this book.

Alex Xu: System Design Interview - An Insider's Guide
The book and the course will help you to think about and understand complex integrations.

Python

Data Structures and Information Retrieval in Python
The new Downey book introduces data structures and algorithms using a web search engine as a motivating example.

Real Python
The best Python tutorials — even the free ones are outstanding.

SQL

Update: our advisor, Dr. Martin Loetzsch just shared for free the material he's teaching also at Pipeline Data Engineering Academy.

Database Design
2nd Edition from The BC Open Textbook Project.

oleg-agapov/data-engineering-book
A good, partial start for SQL from Oleg Agapov in the Beginner path.

CMU Database Group
Video series on contemporary databases and datawarehouses.

CLI

The command line can be intimidating if you're starting out with it. Two resources that are nice to novices especially if they don't come from a computer science background.

Programming Historian
Example: Reshaping JSON with jq

Software Carpentry
Example: Automation and Make

veltman/clmystery
The Command Line Murders - an interactive learnbook.

slackermedia / bashcrawl
Learn Linux commands by playing a simple text adventure.

DevOps

A Cloud Guru
Good choice for paid courses on the cloud and DevOps.

Gently Down the Stream
A gentle introduction to Apache Kafka - pairing whimsical imagery with lucid explanations of stream processing concepts, this book will captivate beginners of all ages.

The Illustrated Children's Guide to Kubernetes
Follow the adventures of Phippy the Giraffe, Captain Kube, and Goldie the Gopher as they discover Kubernetes pods, replication controllers, services, and volumes. Get silly and serious at the same time with this lighthearted introduction to core Kubernetes concepts.

Thanks to all graduates and expert guests of our cohorts in 2021 for their valuable feedback.

Happy learning!

What Did You Build at Pipeline Academy? This.

2022-02-18T00:00:00+00:00

Data engineers have to wear many different hats at the same time: they are architects, designers, builders, maintainers, procurement and quality assurance — to just name a few. If you’d like to break into this profession, you need to prove that you can do all of the above, and more.

One of the key assets you can use to do that is a data product that you’ve built with your own hands. Last summer we’ve already covered the expectations towards a portfolio project in detail, focusing on a general outline and approach. Take a good look:

Recommended READ:
The data engineering Portfolio project

If you’ve never built a data stack from scratch, it’s difficult to imagine how it’s done. Especially when you face a timeline of 12 days total without any real buffers.

Meanwhile, we see the following patterns emerge:

some participants come with a clear project idea, they would like to put a machine learning model into production, or would like to revamp the supporting infrastructure.
some come with a data stack problem that they are currently facing at work, and they need professional guidance and clarity about how to go about solving it.
furthermore - and this might be the most frequent setup - some people come without any clue about what they really wan’t to build. And that’s totally fine.

Below you can find seven outstanding examples for the capstone projects of our alumni. They include air quality monitoring IoT solutions, maps for tracking invasive species in Europe, a Billboard chart for the hottest NFTs and so much more…

They are a great showcase of what you can do with the technologies you learn in our course. Without any further ado, here’s a short list of the projects and products our graduates have built during 2021.

Portfolio projects

Kiril Kasjanov: Wikipedia Event Streaming and Real-Time Analytics with Kafka + ElasticStack

Patrycja Ottawa: Wind turbine monitoring

Sarah Ni: NFTcharts: Uncovering Trade Activity & Hot Trends

Widad Iqbal Mogral: Invasive Species in EU Mapper

Lukas Hondrich: Realtime-ish Twitter Radio

Florian Motz: eMobility availability at VBB Points Of Interests

Romina Nikolova: Free Air Quality Monitoring

Interested in building something similar with the guidance of industry experts? Just hit the apply button and let’s have a chat!

The Data Janitor Letters - December 2021

2022-01-18T00:00:00+00:00

Data engineering salon. News and interesting reads about the world of data.

Databases in 2021: A Year in Review
Dr. Andy Pavlo, Co-founder, OtterTune

It was a wild year for the database industry, with newcomers overtaking the old guard, vendors fighting over benchmark numbers, and eye-popping funding rounds. We also had to say goodbye to some of our database friends through acquisitions, bankruptcies, or retractions.

Building an End-to-End Open-Source Modern Data Platform
Mahdi Karabiben, Senior Data Engineer, Zendesk

A detailed guide to help you navigate the modern data stack and build your own platform using open-source technologies.

Postgres, Kafka, and the Mysterious 100 GB
Matt Ritter, Software Engineer, Google

I think the experience shows that PoCs using “production infrastructure” can expose pitfalls that might appear in a real implementation.

Consider SQLite
Wesley Aptekar-Cassels

As long as you don't expect to need tens of thousands of small writes per second, thousands of large writes, or long-lived write transactions, it's highly likely that SQLite will support your usecase.

Five Tips For a Healthier Postgres Database in the New Year
Craig Kerstiens, Product, Crunchy Data

While onboarding customer after customer this year I've noted a few key things everyone should put in place right away - to either improve the health of your database or to save yourself from a bad day.

The Death of Feature Engineering is Greatly Exaggerated
Pete Warden, Staff Research Engineer, Google

I don’t want to minimize deep learning’s achievements in reducing the toil involved in building feature pipelines, I’m still constantly amazed at how effective they are. I would like to see more emphasis put on feature engineering in research and teaching though.

Don’t start with microservices in production – monoliths are your friend
Arnold Gálovics, Delivery Manager, EPAM Systems

I’m gonna hit you with hard truth my friend.

"Open Source" is Broken
Christine Dodrill, Software Designer, Tailscale

Why I don't write useful software unless you pay me.

Our 2021 in a Nutshell

2021-12-20T00:00:00+00:00

It's the time of the year when everybody is trying to summarise what happened in the last 12 months: 'best of' lists, highlights of the year and predictions for 2022 are dominating your inbox. This blog post is not different either.

2020 was definitely eventful, and 2021 came with its own set of surprises. But Pipeline Academy finally managed to get off the ground, we've launched three amazing cohorts and had loads of fun together with people from across the globe — literally. The chat with Tobias Macey on the Data Engineering Podcast serves as memento for how we've felt after our first cohort had graduated.

Other than that, here are just a few of the highlights that shaped our year:

We've spent the year at our temporary home, the CIEE campus, but as you are reading these words we're already moving in to our new office (more on this soon).
More and more organisations realise how difficult it is to recruit people with data engineering competencies, or approach us to support them on their way of building compliant and sustainable data infrastructures: if you can identify with any of this, feel free to message us.
The "here is how it's done, repeat after me" type of education a lot of data courses provide does not work for data engineering. Read some insights on how we think about education and building data products.
While our first graduates are already out and about, we're busy launching our part-time course in a US-friendly timezone!
The winners of this year’s Pipeline Academy Awards (aka The Pipies) have been announced. What the hell is this about? Read here.
But at the end of the day, the kind words from our graduates are what makes us proud.

Data engineering as an economic force and a career path is skyrocketing, thousands of people in tech and data realise how crucial these technical skills are. If you're struggling with half-baked online courses, or if you're looking for a significant increase in your salary... you know where to find us.

Thanks to all supporters, expert speakers, clients and participants for having our backs.

Happy holidays and a happy new year. Stay safe and healthy.

Best,
Daniel and Peter

The Data Janitor Letters - November 2021

2021-12-15T00:00:00+00:00

Data engineering salon. News and interesting reads about the world of data.

Tracking-industry body IAB Europe told that it has infringed the GDPR
Irish Council for Civil Liberties

Google and the entire tracking industry relies on IAB Europe’s consent system, which has now been found to be illegal.

Why the Data Analyst role has never been harder
Petr Janda, CTO, Pleo

The curse of complexity around the Modern Data Stack.

Spreadsheets: The Duct Tape of the Modern Data Stack
Mikkel Dengsøe, Head of Data Science, Operations & Financial Crime, Monzo Bank

Spreadsheets are the interface that allows anyone to quickly and easily bring data into the data warehouse.

Serverless dbt on Google Cloud Platform
Robert Sahlin, Senior Data Engineer, MatHem.se

A serverless solution to run dbt in a self-hosted and collaborative setup and being able to follow GitOps style.

Orchestrating ELT with Prefect and dbt — a Flow of Flows (Part 1)
Anna Geller, Solutions Engineer, Prefect

How to manage dependencies between data pipelines.

Modelling Type 1 + 2 Slowly Changing Dimensions with dbt
Weng-Kin Lee, Associate Consultant, Servian

By following this pattern to create dimension models, it is easy to incorporate both Type 1 and Type 2 changes into the same dimension. Using dbt macros, we have modularised the dimension’s functionality for reusability and provide capacity to add more functionality to the dimension.

Mapping our data journey with column lineage
Borja Vázquez Barreiros, Senior Analytics Engineer, Monzo

At the time of writing, we have over 4700 data models in our production dbt project, and over 800 views defined in Looker 🤯.

Lesser Known PostgreSQL Features
Haki Benita, Development Team Lead, PCENTRA

Most of us are not aware of all the features in tools we use on a daily basis, especially if it's big and extensive like PostgreSQL.

DuckDB-Wasm: Efficient Analytical SQL in the Browser
André Kohn and Dominik Moritz, DuckDB

It is powered by WebAssembly, speaks Arrow fluently, reads Parquet, CSV and JSON files backed by Filesystem APIs or HTTP requests and has been tested with Chrome, Firefox, Safari and Node.js.

From Data Engineer to SysAdmin
Jonathon Belotti, Team Lead Data Platform, Canva

Put down the K8s cluster, your pipelines can run without it.

Will Nix Overtake Docker?
Connor Brewster, Software Engineer, Replit

No, these tools accomplish different goals, however they can be used in combination to provide the best of both worlds: reproducible builds and containerized deployments.

Storm in the stratosphere: how the cloud will be reshuffled
Erik Bernhardsson

Cloud vendors will increasingly focus on the lowest layers in the stack: basically leasing capacity in their data centers through an API. Other pure-software providers will build all the stuff on top of it. Databases, running code, you name it.

The Pipeline Academy Awards 2021

2021-12-14T00:00:00+00:00

The end of the year has arrived, our first full year in operation. 2021 was about the Cambrian explosion of data engineering tooling, yet you don't have to be a data scientist to be certain that 90% of the data tools will be gone in about two years or so, and for a good reason. Like it or not, most of them are solutions for fictional problems.

Whether you are a seasoned data architect or a newcomer in the world of data infrastructure, you already know that a major part of your work is about curating the right tools for your business, your team and your data. One of the key assets that our participants leave the course with is a framework for making informed decisions about data infrastructure tooling. It considers timeless software engineering practices, state-of-the-art-technologies, business goals and even sustainability factors, but under the hood it is very much powered by common sense.

This framework (and the desire to avoid major migration projects) is what prevents data engineers from being misled by the shockwave of marketing messages that aim to convince them that a new and shiny tool is a must have in... wait for it...

THE MODERN DATA STACK.

We deal with the process of making these tooling decisions on a daily basis. We cover a lot of them in the course, teach them, love them, update them and use them. And we have favourites that stand out.

The 2021 Pipeline Academy Awards (The Pipies) are brought to you by Pipeline Academy. It's our way of praising the teams that built meaningful software solutions that serve a real purpose, the ones that are most likely here to stay to make the lives of data engineers better.

Separating the wheat from the chaff

Generally what we value:

the tool is open source, you can run it on your own if you have the resources and the expertise,
the tool has a paid hosted option for the occasion that you don't have the means to host it on your own,
the hosted option has a generous free tier to start with,
good tutorials and documentation is provided,
learning curve is not steep.

Caveat: the Apple M1 hell is real, the support for it is not really there yet.

The 2021 Pipies go to…

Data Acquisition: Airbyte

Self-definition: Open-Source Data Integration Pipelines | ELT. Our choice for EL, plays nice with dbt, bit heavy on the resource side, Docker as a Lambda :)

Telemetry: Quix

Self-definition: Real-time stream processing PaaS. Whatever takes the pain out of Kafka is a friend of ours. A European player with McLaren expertise, supported by Project A.

ETL/ELT: Prefect

Self-definition: The New Standard in Dataflow Automation. "Airflow is ****, yes.", so Why Not Airflow?. A product that evolves in a great way, already at its second-generation workflow engine, Orion.

Data Modeling: dbt

Self-definition: Transform data in your warehouse. The whale that rules the T in ELT that coined the term Analytics Engineering. dbt encodes best practices, a product built on experience solving actual problems; the jury is still out there on how it scales for BIG data though. Still should be good enough for 90% of the BI teams.

Data Warehousing: ClickHouse

Self-definition: fast open-source OLAP DBMS. From Russia with love, written in C, just a magnitude faster than anything else especially if it starts with Apache.

Data Quality: Great Expectations

Self-definition: a shared, open standard for data quality. The Janus-faced power tool, marrying the best of both machine-readibility and human oversight. Outstanding learning curve and workflow, can get heavy on deployment because of its massive dependencies.

Clouds: Pulumi

Self-definition: Modern Infrastructure as Code. Infrastructure as code, not configfiles, no YAML horror - even in Python. Period.

Deployment: Fly

Self-definition: Deploy app servers close to your users. Dark magic to deploy your Docker images in a whim.

Serving Data: Streamlit

Self-definition: The fastest way to build and share data apps. Shiny for Python, straps a beautiful interactive interface on your Python logic.

Honorable mentions:

Computation: Saturn Cloud

Self-definition: Data Science & Machine Learning with Dask & GPUs. I still like the old tagline: It's like Spark. Except you won't hate yourself.

CD/CI: GitHub Actions

Self-definition: Automate your workflow from idea to production. The only thing that I feel “how-could-we-live-without-it” about, also git scraping.

Serving Data: Datasette

Self-definition: An open source multi-tool for exploring and publishing data. Mentioning Simon Willison, let's talk about the power of SQLite, an interface, all possibly Dockerized with a few lines of code.

The Pipies 2021

Big shout-out and thank you to all of the teams who are responsible for these products, it's a pleasure for us to have so many amazing possibilities to choose from.

If you haven't yet, we recommend every data professional to check out these tools and compare them with the solutions you're already familiar with. Let us know what you think about the winners, and feel free to send us your preferred data tools and updates from this year!

See you in 2022, when The Pipies return.

Disclosure: Pipeline Academy is not financially affiliated with any of the above organisations.

Skills Gap in Data Engineering

2021-11-24T00:00:00+00:00

Most data professionals realise very early in their journey that accessing the knowledge that they really need to solve data engineering problems is hard to come by. The other thing they don’t necessarily see is how short-sighted a lot of courses are, and how most of the technical content they provide is going to be rendered useless in a year or two.

Whenever people talk about the so called skills gap, they focus on addressing problems in a systemic scale and try to come up with a solution that is meant to address multiple dimensions at the same time: time, social factors, economic trends, technical advancement, legal changes just to name a few. This is complicated by the conflicting agendas of the involved stakeholders: employers, learners, government institutions and lastly, educational institutions.

I was lucky enough to be involved in an action group organised by Emerge, Coursera, Ufi and Filtered, that set itself the goal of coming up with a green paper that “examines how we can develop the necessary skills and establish new pathways into jobs”. My contribution in this work was very humble, but this should not stop you from reading the study. I don’t want to miss the opportunity to share this with you so everyone can have a sneak peek into the world of structural challenges that educators and edtech entrepreneurs have to tackle in the upcoming years.

READ THE FULL REPORT: Developing skills and establishing new pathways into jobs (.pdf)

You see, there is a strategic and a tactical level to addressing the challenges of the near future, but they can’t and shouldn’t be separated. Otherwise this will lead to fundamentally broken outcomes for learners: using wrong didactical methods, developing meaningless curricula, abusing technology for the wrong purposes, just to name a few. But false marketing messages (“profession of the next decade” — does this ring a bell?) are just as much of a problem, they can lead masses towards an unsustainable career direction.

Long story short, ultimately I tend to come back to this quote from John Alexander Smith, Professor of Moral Philosophy at Oxford from 1914:

“Gentlemen, you are now about to embark on a course of studies that (will) form a noble adventure…Let me make this clear to you. ..nothing that you will learn in the course of your studies will be of the slightest possible use to you in after life – save only this – that if you work hard and intelligently, you should be able to detect when a man is talking rot, and that, in my view, is the main, if not the sole purpose of education.”

The faster the world moves, the more you’ll have to learn to avoid being misled. Lifelong learning is not a trend.

10 Reasons Why Aspiring Data Engineers Choose Pipeline Academy

2021-11-09T00:00:00+00:00

Ambitious data analysts, data scientists trying to take their careers to the next level, product owners aiming to build next-generation data products, software engineers dealing with legacy data stacks... they all are facing the same challenge: how do I get the data engineering skills that enable me to achieve my goals?

This frustration is very real, and it is indeed very common! Plenty of professionals are trying to find a simple and well-structured leaning path for data engineering, but the road is paved with deceptive marketing messages and fake experts.

The students who ended up at Pipeline Academy gave us their reasons for picking us, and this list was born. You can read their reviews on CourseReport. So if you feel that you can identify with any of the below, just schedule a call and have a chat with me!

1. Online courses don’t work for most

Raise your hand if you've finished every online MOOC course you've started. Trust me, you are not alone. It's difficult to filter for the meaningful platforms/courses that deliver value especially when it comes to learning data engineering, but even if you find them (here are some curated suggestions), bringing up the right amount of motivation to finish them can get really tough. Our bootcamp delivers the structure, the methodology and the environment that makes you achieve your goals.

2. Issues and frustration when building data infrastructures

So you are the data person who somehow ended up being the one building, improving and maintaining the data infrastructure at your company. Yet you lack the proper training to make informed decisions about tooling and optimisations? Or you are being held back from implementing your ideas, concepts and models due to the lack of data engineers or infra know-how? This is the thing that leads data analysts and data scientist leave their jobs and look for a more fulfilling path within the realm of data, which we know is more than achievable.

3. Career improvement and salary increase

Many data pros want to get more technical and earn the benefits of doing so (promotions, more job opportunities, higher salary etc.), and realise that a compact yet intensive course would be the right path forward, covering the fundamentals that apply regardless of the industry they are in. When you are stuck at your job as BI analyst, data scientist or product person, improving your technical skillset delivers a high return on investment even in the short term.

4. No fluff, no bs, just straight-talk

Hard facts corroborated by benchmarks, hands-on experience with using tools and approaches at large organisations, and a contrarian yet truthful opinion are not easy assets to come by as a learner when players in the data ecosystem are hard-pressed to avoid them. Branded courses with marketing-as-a-curriculum (aka MaaC) will misguide learners and give them the warm and cozy feeling of certainty. This is the best tool, stop asking questions. We just do the opposite — in all of our work.

5. Data engineer job descriptions can be intimidating

Have you ever read a job description for a data engineer role? Did it read like the line-up of an indie music festival (Great Expectations, ClickHouse etc.), or the roster of an Italian football team (Luigi, Apache Cassandra etc.) or something Elon Musks PR team came up with as a grandiose joke (Terraform, Kinesis Firehose etc.)? If your answer is yes, I have news for you: this is the exact reason why even experienced data folks are often hesitant to apply for these positions. Part of the measurable transformation (before vs. after) that results from doing the course is that you'll feel empowered to apply, and you'll have the confidence to rock the interviews. Right, Michele?

6. Get a holistic & structured view on the data infrastructure ecosystem

In order to get ready to start as data engineer after 12 weeks of training, you need to develop an understanding of the fundamental concepts, of the tooling landscape, of best practices, but also of the surrounding business context. Clouds and virtualisation? Real-life coding challenges from renowned organisations? Recommendations for diving deeper into ML, data modeling and dataops? It's all in our curriculum.

7. Small cohorts and personalised learning experience

Some franchise bootcamps received significant backlash after filling their virtual classrooms with 30-40 students, which rendered organic teacher-learner and learner-learner interactions difficult. Active participation and asking questions should be encouraged all the time, this is how a coding bootcamp experience is supposed to be more than just looking at way too many talking heads in a Zoom window, where you are more or less just a number. Keeping the cohorts deliberately small allows us to focus on individual needs, customise career coaching and deliver an outstanding learning experience.

8. Expert teachers and renowned guest speakers

Daniel and I have done our fair share of shenanigans in the realm of tech, data, education and building teams and digital products. We've made loads of mistakes along the way, and we love to share our war stories so you don't end up repeating them. Our guest speakers share this attitude, and the opportunity to pick their brain is not something you'll get in any other school or conference.

9. Future-proof knowledge

Nobody really knows what the data infrastructure landscape is going to look like in 2030. Yet there are timeless best practices and tools attached to the smart data engineer's belt that allow for dealing with new and shiny trends of the day. There is a reason why we turn to SQL, Python and Makefiles in 2021, and there are reasons why we apply concepts like TCO before making decisions on tooling.

10. Solving Real problems and building new Products is rewarding

It's not easy to stand out from the job seeking masses by doing the same pre-defined exercises as all the others. What would you say if I told you that the assignments at Pipeline Academy are designed in a technology-agnostic and solution-oriented fashion? We care about helping you figure things out rather than giving you 'fill in the blanks' type of assignments. If you like puzzles, you'll love the course.

The Data Janitor Letters - October 2021

2021-11-04T00:00:00+00:00

Data engineering salon. News and interesting reads about the world of data.

Eating the Cloud from Outside In
Shawn Wang, Developer Experience, Temporal.io

AWS is playing Chess. Cloudflare is playing Go.

Why Lightspeed invested in ClickHouse: a database built for speed
Gaurav Gupta, VC, Lightspeed Venture Partners

$250M Series B financing of ClickHouse.

Day In The Life Of A Data Engineer — What Do Data Engineers Do?
SeattleDataGuy

There’s no better time to jump into the world of data engineering.

ROAPI: An API Server for Static Datasets
Mark Litwintschik, #bigdata Consultant

ROAPI is an API Server that exposes CSV, JSON and Parquet files without the need to write any code.

Announcing Streamlit 1.0! 🎈
Adrien Treuille, Co-Founder and CEO, Streamlit

Streamlit used to be the simplest way to write data apps. Now it's the most powerful.

Function pipelines
David Kohn, Developer, Timescale

Building functional programming into PostgreSQL using custom operators.

Implement a slowly changing dimension in Amazon Redshift
Milind Oke and Bhanu Pittampally, Amazon

DBT at HomeToGo. Creating a scalable framework
Gijs de Kruif, HomeToGo

Nothing new under the sun.

How to Execute Pandas Workloads in a Distributed Manner With Apache Spark
Hyukjin Kwon and Xinrong Meng, Databricks

If you have to do it you have to do it :facepalm:

Bash functions are better than I thought
Gary Verhaegen, Senior Software Engineer, Digital Asset

Given all that, I simply do not understand why people keep recommending the {} syntax at all.

The Data Janitor Letters - September 2021

2021-10-15T00:00:00+00:00

Data engineering salon. News and interesting reads about the world of data.

Cloudflare’s Disruption
Ben Thompson, Stratechery

S3’s margin is R2’s opportunity.

Operations is not Developer IT
Mathew Duggan, DevOps Manager, GAN Integrity

It's not their fault, they were told this was easy.

How Big Tech Runs Tech Projects and the Curious Absence of Scrum
Gergely Orosz

A survey of how tech projects run across the industry highlights Scrum being absent from Big Tech. Why is this, and are there takeaways others should take note of?

Who are analysts, technically?
Bobby Pinero, CEO and Co-Founder, Equals

Rather than needing to be an impossible combination of statistician, developer, and business expert, analysts can simply be great critical thinkers.

Re-evaluating Kafka: issues and alternatives for real-time
Olivia Iannone Technical Writer at Estuary

Kafka’s challenges have exhausted many an engineer on the path to successful data streaming. What if there was an easier way?

A very detailed comparison of Python stream processing libraries
Mike Rosam, Cofounder and CEO, Quix

“To successfully use Flink in production you must invest serious resources … estimate more than 18 months.”

The First Rule of Machine Learning: Start without Machine Learning
Eugene Yan, Applied Scientist, Amazon

Having robust data pipelines and high-quality data labels also suggests you’re ready for machine learning.

Reaching MLE (machine learning enlightenment)
Vicki Boykis, Machine Learning Engineer, Automattic

Once, on a crisp cloudless morning in early fall, a machine learning engineer left her home to seek the answers that she could not find, even in the newly-optimized Google results.

Meet The Graduates: Michele Tassoni

2021-10-08T00:00:00+00:00

Peter: Michele, it's great to see you again. Thanks for taking the time to have a chat with me. Can I ask you to please give us a short introduction about yourself?

Michele: My name is Michele. I'm Italian, but I live in Germany for nine years. I am a biologist by education and, after my master thesis, following my passion for technology I decided to continue my studies in Germany. Here I could learn about drug discovery, data analysis and image processing.

Michele Tassoni, Data Engineer and graduate of Pipeline Academy’s founding cohort

This is where my story with data started, and I could see that I could learn and develop even further. Afterwards, I worked for two years at Wayfair as a data scientist. During this time I learned that building a model and making predictions is not enough in the industry. You need to be able to collect all your data, arrange your dataset, build new features, and only after that you can develop your model properly.

Furthermore, models need to go into production and for this I had to learn many new things. Because I like to focus more on the technical aspects of the data processing, I decided to join Pipeline Academy.

P: What were your expectations before you joined Pipeline Academy?

M: Before joining Pipeline, I was looking for a bootcamp where I could grow my skills, and I wasn't sure whether to try to improve my data science skills or to go towards a more technical direction. Having talked to a couple of schools and their alumni, I thought that a data science course wouldn't be the best choice for me, because I wouldn't get that much out of it as from growing my engineering skills. Once I found out about Pipeline Academy I immediately thought that it would be a better option, and after talking to you I was almost immediately convinced to join.

P: What did working through the twelve weeks feel like for you?

M: As soon as I understood how the 12 weeks were structured and by looking at the curriculum I thought that it would be quite difficult to keep up with it. But once we had to choose our portfolio project my goal was to improve it week by week, trying to implement the most important and interesting learnings that were covered during that particular week. That required quite a bit of effort from my side, also during my free time or even weekends sometimes. However, I enjoyed working on it.

P: ...and finally you ended up with a very impressive capstone project. What was it about?

M: I decided to create a product (link to GitHub) that was collecting data from transfermarkt.com, so about football players, mostly. I collected data about individual players (e.g. weight, height, market value, etc.) for the past 50 years for the top five European championships. The data was stored in a database, restructured and modelled. From this little data warehouse, I was extracting the data back and showing it on a dashboard via a web application. But there was a tiny trick about where this data warehouse lived. I wanted to integrate Great Expectations into my project, which was a bottleneck because my development environment became too large to be deployed on PythonAnywhere for example. So I ended up deploying it on my Raspberry Pi!

P: Tell me about the most memorable moments of your time in the course. The good, the bad, the surprising moments, whatever comes top of your mind.

M: Oh, that's hard. I really enjoyed the guest speakers: for example Martin (Dr. Martin Loetzsch) for me was the best, followed by the "Facebook guy", Bence and Daniel from Shopify. They were inspiring because listening to them I understood that to be good at something, a lot of effort is required.

Also, in my opinion if you join this bootcamp without a good base knowledge and the right level of motivation, it's easy to fall behind. That's why I would say miracles can happen in this school, yet a lot comes from you as a person and as a student, and your motivation and your willingness to push for those three months. It's really not that difficult to do it, and at the end of the course you will have learned a lot.

P: Did your perception of the data engineering role change during or after the cohort?

M: So I have to say that probably without the school I wouldn't have applied for the data engineering position that I'm currently having. I would have flagged this particular job as something that I wouldn't have been a good candidate for. But due to the course I was not scared to apply for it.

P: If you would have to summarise, how did the bootcamp contribute to your success?

M: When starting a new job, you have to learn the whole data architecture of your new company as quickly as possible and as well as you can.

During the bootcamp, we covered many times concepts like how a data warehouse architecture looks like, how data is transferred from a backend database into the data warehouse and how the data moves from a warehouse to the business intelligence tool. This has helped me a lot during the first two months at my new job.

P: This brings me to the question: what are you doing now?

M: I am a Data Engineer at Contorion. The team that I joined, maintains and improves the ETL and the data warehouse. This data is meant for our business intelligence and analytics teams who will then work with it developing new business insights. We also develop our own business intelligence tool. I'm looking forward to the time when my team will start to develop new data products, and I can contribute to them. That will be amazing.

P: If you could give advice to any prospective data engineer, what would it be?

M: I always think that I never had a real good mentor, I always had to do everything by myself. But there were some moments during my studies where I could learn from more experienced people, and that always gave me such a boost. My advice is to try to find someone as your mentor who can lead you to become a better professional.

P: Thank you very much for the interview, Michele.

The Data Janitor Letters - August 2021

2021-09-20T00:00:00+00:00

Data engineering salon. News and interesting reads about the world of data.

From Data Driven to Driving Data — The dysfunctions of Data Engineering
MrTrustworthy

Many “data driven” initiatives are failing even though they had the best engineers on the task and picked the “best” stack of technologies.

What's an OLAP cube? 🎲
Claire Carroll, Analytics Engineer, analyticsengineers.club

OLAP cubes were this intimidating concept, and the more they read, the less they understood, but it turns out that they aren’t that confusing in practice. This is a trend that I’ve seen a lot in data engineering / data modeling where jargon is used as a gatekeeper.

Expecting Great Quality with OpenLineage Facets
Michael Collado, Staff Software Engineer, Datakin

Good data is paramount to making good decisions - but how can you trust the quality of your data and its dependencies?

A future for SQL on the web
James Long, Software Engineer, Stripe

Absurd-sql is a persistent backend for SQLite on the web. That means it doesn’t have to load the whole db into memory, and writes persist.

MinIO: A Bare Metal Drop-In for AWS S3
Mark Litwintschik, Big Data Consultant

MinIO offers an S3 gateway service that can allow you to expose Hadoop's distributed file system (HDFS) with an AWS S3-compatible interface.

No, we don’t use Kubernetes
Maik Zumstrull, Site Reliability Engineer, Ably

It also doesn’t make sense for a lot of companies that are currently going all-in on Kubernetes.

Evolution of search engines architecture
Julien Lemoine, Co-founder & CTO, Algolia

We look at some key milestones in the evolution of search engine architecture. We also describe the challenges those architectures face today.

The Data Janitor Letters - July 2021

2021-08-28T00:00:00+00:00

Data engineering salon. News and interesting reads about the world of data.

Building a data team at a mid-stage startup: a short story
Erik Bernhardsson, Working on something, "Bernco"

The data culture is driven both from above (the CEO pushing for it) as well as from below (people in the trenches). It's OK to fail if at least you learned something from it.

Analytics is at a crossroads
Benn Stancil, Chief Analytics Officer + Founder, Mode

The world is full of great analysts. Will we have the courage to go looking for them?

Against SQL
Jamie Brandon, independent researcher

SQL is the only widely-used implementation of the relational model, and it is: inexpressive, incompressible, non-porous.

For SQL
Pedram Navid, Head of Data, Hightouch

Why I'm so over-protective of my data people.

The Untold Story of SQLite With Richard Hipp
Adam Gordon Bell, CoRecursive Podcast

On today’s show, I’m talking to Richard Hipp about surviving becoming core infrastructure for the world.

The Data Janitor Letters - June 2021

2021-08-02T00:00:00+00:00

Data engineering salon. News and interesting reads about the world of data.

The Analytics Engineering Guide
dbt Labs

Collaborating as a data team to produce excellent datasets -- some parts are bullshit, but it's an interesting read.

Welcome to Snowpark: New Data Programmability for the Data Cloud
Isaac Kunen, Senior Product Manager, Snowflake

Two words: Java functions.

Accidentally exponential behavior in Spark
Ivan Vergiliev, Tech Lead, Leanplum

Don't use Spark for tasks that require complex logic.

The ritual of the deploy
Vicki Boykis, Machine Learning Engineer, Tumblr

Deploying is a ritual. It’s a sacred place, a quiet place, and a dangerous place, where anything can happen. In deployment, the system is in a fragile state, and you are in a fragile state.

Gently Down the Stream
Mitch Seymour, Illustrator, Author, Founder, Round Robin Publishing

A gentle introduction to Apache Kafka.

What's Kafka and what does Confluent do?
Justin Gage, Technically

Help with solving Kafka-esque data problems

Cloudera to go private as KKR & CD&R grab it for $5.3B
Ron Miller, TechCrunch

Cloudera was once one of the hottest Hadoop startups, but over time the shine has come off that market, and today it went private.

Meltano Spins Out of GitLab, Raises $4.2M in Seed Funding Led by GV to Enhance Open Source Data Integration

"Meltano aims to bring the entire data lifecycle into the DataOps Era." Wut?

How to Teach Data Engineering

2021-07-26T00:00:00+00:00

Frequently we receive positive comments about our curriculum and our general approach to data engineering, especially in the light of the hype vs. the realness that surrounds it. However, the selected content would be ineffective and empty for our students without the right delivery. Selecting the fitting pedagogical approach and combining it with the most effective didactical methods are key to running a successful course, so here are the whys and hows behind what a Pipeline Academy student is dealing with at the bootcamp.

As pointed out when talking about data engineering portfolio projects, not having best practices in place forces you start from scratch and figure out yourself what would make sense in terms of teaching this constantly evolving subject matter. But what can go wrong — you ask?

Well, it might happen that the learning process is simply ineffective, and you just don't get the gist.
You might leave a school and realize that whatever you've learned is totally detached from real life, and you as a result fail to connect the dots and the "career kickstart" does not happen.
Potentially you could lose interest during the course and drop out.
Not being upfront and honest about the learning process and what kind of engagement and mental effort studying takes from students can be extremely misleading, and is something that we see a lot of coding bootcamps struggle with.
Making sure people can access the course fairly easily independently from their socioeconomic background is also something we talk about a lot internally.

On top of this, data engineering is kind of a special case: you have to have strong principles and fundamentals to be able to tackle random incoming unstructured tasks and issues when working as a data engineer, and they do help when you're trying to maintain your sanity in general. This becomes very clear when you realise that a controlled environment is rarely something that these professionals deal with, and accepting this is an essential step in the right direction.

So you need some experience and the attitude to get it right, and design a meaningful and engaging learning experience - a pedagogy market fit.

Daniel + Zoom + Chewbacca mask

Not our first rodeo

I have spent over seven years in the e-learning industry. I have worked in higher education, in vocational education and have contributed to the digitalisation of the traditional school system's teaching methodologies. I've presented conference papers on learning theory and practice at conferences like Online Educa Berlin (2004, 2005) and Microlearning Innsbruck (2006) on connectivism, on edutainment and on social software like wikis.

The other co-founder and career coach at Pipeline Academy, Peter holds a double-degree in business administration and teaching, and he has taught marketing research at his alma mater while studying there. He has worked in various environments mentoring and supporting junior talent and helped navigate them through the realm of emerging technologies and the related competencies. He has been in charge of one of the world's top rated data science bootcamps and has coached various students transitioning to the field of data and supported them in the process of finding jobs and leveling up their careers.

I guess it's safe to say that we're professionally well-equipped to make the curriculum work for our students, but it's crucial to follow the latest research and digital tools in order to succeed on the long run.

Pedagogical concept

Contextual learning is based on the constructivist theory of teaching and learning. Learning takes place when teachers are able to present information in such a way that students are able to construct meaning based on their own experiences.

Contextual learning has the following characteristics:

emphasizing problem solving,
recognizing that teaching and learning need to occur in multiple contexts,
assisting students in learning how to monitor their learning and thereby become self-regulated learners,
anchoring teaching in the diverse life context of students,
encouraging students to learn from each other,
employing authentic assessment.

As opposed to the more traditional "I'll show you how it's done, repeat after me, now you should be able to do it too"-kind of teaching and learning environment — not that there is anything wrong with that when used in the appropriate context.

Why constructivism? The answer is easy: engineering is about solving problems. Data engineering is about solving (mostly) unstructured problems while filtering the massive hype around tooling and the ecosystem in general. There will be no handbooks, only a compass and a map. You have to build yourself an internal tool for navigating, while we help you sketch your interpretation of the world; that's the best you can do.

Every interaction (or the lack of it) permeates the curriculum with these paradigms. Assignments, teamwork and learning to fail and try again. That will be it. We've tried and tested this approach: it's tough and it works.

Trust the process.

Based on the experience of ourselves and our graduates we think these are the major considerations you should take when thinking about whether we are the right choice for you:

The bootcamp is hands-on, effort and outcome oriented.
You have to write code.
We will emulate real-life work scenarios.
You will solve problems.
You have to take the initiative.
We'll engage you in frontal instruction, solo assignments, research, working in pairs and teams, and you'll be presenting and explaining concepts and tools to others.
Standups, retros, open tickets, open Pull Requests are bread and butter for us.
You'll prepare for job interviews by answering practical questions and solving very real coding challenges.
Sync and async communication will happen all the time on Slack and Zoom.
You will report bugs with context, describe your attempts to resolve the situation, you will practice asking for help efficiently.
This is not a simulation bubble, this is entropy itself.
Things will break as its their nature.
You might have to reinstall your computer once or twice.
From time to time, you'll curse — and that's totally ok: "This is not a pop album" as Ice-T pointed out oh-so wisely in Warning.

If you are up for the fun, this is what you'll end up with when graduating from Pipeline Academy:

Confidence: you'll realize that the invested amount of curious exploration, the effort put into trying again over and over, your continuous attempt at connecting the dots between what makes sense vs. what you are told, and the work spent collaborating with others correlates directly with your self-confidence for your job interviews and your future career. We provide the stage, but the show is about you.
Context: are you intimidated when checking out data engineer job descriptions, because they are full of seemingly random tools you've never heard of? Assessing and benchmarking them, using them and learning their quirks will enable you to deal with most of the data stacks out there, even the ones that don't exist yet.
Code you own: as a data engineer, your code will most likely be looked at when applying for a job. Proving the right competencies and attitude with the work you've delivered will feel rewarding and will make you stand out.

The Data Engineering Portfolio Project

2021-07-02T00:00:00+00:00

Being the co-founder of the pioneering data engineering bootcamp means having no real blueprint on how to do things. No franchise books, tutorials or blogposts to lean on. Having a decade of experience in data, and half of that on top in the e-learning industry I had to sit down and think again with the head of the hiring manager who had been interviewing data engineers in Berlin for 7 years now, condensing what I have seen in about a thousand interview sessions.

If you join Pipeline Academy, it's very likely that you come to the bootcamp with the goal of leveling up in your career, meaning that you are going to have go through applications and job interviews after the course. In this process, you'll have to prove that you can execute the tasks listed in the job description, and that you are the right Data Engineer/ML engineer/Data Product Owner/... for supporting the goals of your future employer.

The products and projects you've built constitute a massive part of making a positive impression as an engineer, so let's take a look at what we consider fundamental expectations.

How should a professional data engineering portfolio project (aka capstone project) look like?

Peter and I start every month with a virtual open house session where we introduce Pipeline Academy to all the interested folks who join in. We keep repeating like a mantra that our aim for our graduates is that they should leave the course with three things in their pockets:

Context: i.e. understanding the ecosystem and the driving forces of data engineering,
Confidence: for having the right attitude for solving unstructured problems,
Code they own: everything they've produced during the course.

Don't forget - and I am stating the obvious here - our expectations are broadly speaking not different from what startups and tech companies are looking for.

Functionality

Your portfolio project should have the functionality of a typical real world data stack. It does not have to be complex or complicated, straightforward is better. Matching your tooling decisions with the business circumstances shows you listen, think holistically and care about the why.

Functionality that you should cover is:

acquisition of data from a database, an API, a queue/tracker, a scraper,
an automated ETL process, using a Prefect Cloud or a dbt cloud setup is a good choice - a cronjob and a Makefile is always a big plus for me,
loading data to a datawarehouse - hints: SQLite, DuckDB can do wonders, and having data quality measures in place would not hurt either,
if you do plain vanilla machine learning, you're already considered for a machine learning engineer role nowadays,
it is deployed easily (Makefile, Dockerfile, something executable), maybe in the cloud even,
your codebase has some CD/CI on it -- Github Actions will do, so you are dataops now,
it serves data to humans (web interface) or machines (API).

Subtle hint: you can see that our curriculum is pretty much aimed at making sure you can do all of this by the end of the bootcamp course. If you have a data engineering portfolio project that fits this description I would happy to review it and have a chat about it with you.

Source: @svpino’s twitter feed

Skills

These are the skills I'd like to see demonstrated in a portfolio project:

you can explain your/the project's goal in simple terms (you have a one-pager README.md or something similar — Readme Driven Development can help here: describe in plain English what you're trying to achieve, describe what you don't know yet and still have to research),
you have an architectural doodle/blueprint so one can grasp the components and their relationships,
your code is okay enough to be actually read,
you demonstrate structured thinking, communication skills, understanding of the fundamental concepts, and collaboration capabilities (e.g. "I've learned this from that blogpost and I opened an issue on the repo of this tool because I hit a wall").

It does not have to be complicated -- it's better if it's not --, it does not have to solve all the problems of mankind. It just should deliver what you say it should.

It's about showing how you were thinking about a problem and how you've delivered an imperfect solution that is suitable for your constraints. It will have tradeoffs and that's totally fine, and I'm happy to read about why and how you've ended up making certain decisions.

Interest and Attitude

If you can deliver on the above, that's already a decent start in my book. However, you should not forget that the closer you get to the final stages of a job interview process, the more your future coworkers are going to scrutinise your soft skills and the so-called "fit". In general, this is what a lot of people are looking for in a future coworker in engineering when it comes to general attitude:

you are confident in approaching and dealing with problems,
you understand K.I.S.S. as a principle,
you are determined to build a resilient system so you are not on call on the weekends for fixing bugs,
you don't build things just because you can on the job — that is called a hobby,
you solve a problem without introducing a new one,
tools are tools and not means,
you frequently ask questions, you listen to the answer and you think about them,
you don't start a Spark cluster to load a CSV.

Do not underestimate the power of showing that you are somebody who others would like to sit next to. It makes a world of difference.

Examples

Here are a couple of examples delivered by the students of Pipeline Academy. It should be noted that these products and capstone projects are being created under special circumstances (think time pressure, certain individual goals ingrained, focusing on solving one specific problem etc.).

Tomek Florek: Feature enrichment API for data science and analytics
Amy Raygada: Menu generator based on nutritional values
Michele Tassoni: Data aggregation and data analysis app to determine the value of football players
Michail Koskinas: A hip-hop playlist recommender based on the trackID of a classical music piece from Spotify
Sujit Badle: Crypto data ETL backtest

EDIT: click here to see even more portfolio projects that our graduates have built during 2021.

Don't forget

Tailor your capstone project according to your career goals: you can put an emphasis on code, architecture, communication, processes, quality etc. depending on what kind of role you are going for.
Make sure that your project is consistently presented and explained the right way, even for people who are not familiar with its context.
Practice explaining the why and the how: expect questions that uncover your train of thought.
Ask for feedback: wherever you have a chance, ask the hiring manager or whoever is reviewing your portfolio for feedback.

Pipeline Academy on the Data Engineering Podcast Vol.2

2021-06-29T00:00:00+00:00

Daniel and I had the opportunity to sit down and have a chat again with Tobias Macey from the Data Engineering Podcast. His show is a mainstay in the data engineering community, and it was a pleasure to revisit the events of the last 8 months together with him. Listen to the episode right here!

While Daniel’s first appearance on the show from September 2020 was more about our very specific approach to solving engineering problems and turning this into a curriculum and a proper learning concept, and other future plans, this time we mostly speak in past tense.

We’ve talked about what launching Pipeline Academy felt like and we’ve shared some insights about the experiences of our founding cohort. Highlights, proud moments, challenges, lessons learned, takeaways for the future — you name it, it’s all in there. This episode is especially recommended for people who are considering joining a bootcamp and/or learning data engineering, and for the ones amongst you who are asking themselves why context, code and confidence are so important in order to get started in the universe of data and software engineering.

About the host and the show (taken from the website of the podcast):

“Tobias Macey is a dedicated engineer with experience spanning many years and even more domains. He currently manages and leads the Technical Operations team at MIT Open Learning where he designs and builds cloud infrastructure to power online access to education for the global MIT community. He also owns and operates Boundless Notions, LLC where he offers design, review, and implementation advice on data infrastructure and cloud automation.”

“The Data Engineering Podcast tackles a new approach to data management every week. Each new episode provides useful and informative insights into the projects, platforms, and practices that data engineers, team leaders, and data scientists need to know about to learn and grow in their career.”

Like and subscribe on Apple Podcasts and give the show a follow on twitter!

The Data Janitor Letters - May 2021

2021-06-24T00:00:00+00:00

Data engineering salon. News and interesting reads about the world of data.

Analytics Engineering Everywhere
Jason Ganz, Cofounder, ChattyKathi

Or why in five years every organization will have an Analytics Engineering team.

Common data model mistakes made by startups
Metabase

It’s important to note that the anti-patterns we’ll discuss below are specific to startups.

16 fundamental principles for transforming data in a warehouse
Rahul Jain, Head of BI and Data Engineering, Beat

The current discourse on data can get a little tiring because of its over focus on tooling.

Using PostgreSQL as a Data Warehouse
Cedric Dussud, Cofounder, Narrator.ai

With some tweaking Postgres can be a great data warehouse. Here's how to configure it.

Why Spark is NOT the right tool for ETL work
Kevin Bair, Director Sales Engineering, Snowflake

My larger point here is when you have a hammer (spark) everything looks like nail.

Using Apache Airflow DockerOperator with Docker Compose
Flávio Clésio, Staff Engineer Data/Machine Learning, Artsy

I personally believe that Airflow + Docker it’s a good combination for flexible, scalable, and hassle-free environments for ELT/ETL tasks.

The Metagame of Applying Machine Learning
Eugene Yan, Applied Scientist, Amazon

When designing systems, less is more.

PostGIS at 20, The Beginning
Paul Ramsey, Executive Geospatial Engineer, Crunchy Data

All the development was done on the trusty Sun Ultra 10 I had taken out a $10,000 loan to purchase when starting up the company.

Drunk Post: Things I've learned as a Sr Engineer

SQL is king. Airflow is shit, yes.

Data and sustainability: Interview with Zsofia Bognar From Ecosia

2021-06-08T00:00:00+00:00

Pipeline Academy works with a number of selected organisations and supports them with in-house data engineering trainings and with guidance related to their data infrastructure. One of these companies is Ecosia, the search engine that plants trees. The (growing) data team of the Berlin-based startup is led by Zsofia Bognar, and we’ve had the chance to interview her about the challenges at the intersection of data and sustainability.

Zsofia Bognar, BI & Data Team Lead at Ecosia (LinkedIn profile)

Peter: What is your professional background and what drove you towards a career in data?

Zsofia: I’ve been working in the data field for the last 12 years so it’s hard for me to imagine life WITHOUT data :) . But jokes aside, I think what drew me to work with data most is how by analysing data we can get closer to understanding the world around us. I studied Economic Sciences, a part of which was statistics. For a lot of people statistics is a really dry discipline, but I found it fascinating how with the help of statistical methods we can gain fresh insights in a concise way for just about anything.

P: You are currently in charge of the data team at Ecosia. What kind of value is generated from user data for the organisation and what kind of team is dedicated to data initiatives?

Zs: At Ecosia we are on a mission to build a digital green companion that plants billions of trees. We want to help our users make sustainable choices, and be a role model in the transition to a sustainable society.

As a data team we contribute to this mission by delivering data products in three main areas:

1) We focus on helping product teams make data-informed choices in line with their KPIs and their own team mission. We help them monitor and interpret their results. Advising and educating teams plays a big role in ensuring team success.

2) We create reports to evaluate different marketing efforts to understand what the most informative way to reach our users is. We not only want to attract new users but make actionable information available about the climate emergency in an approachable way and we as data team help the marketing team to identify these areas.

3) We build capabilities for the tree team to evaluate their tree planting projects, which in turn also delivers value to our tree planting partners since our tree planting officers are able to communicate transparently and quickly the impact and possible areas of improvement for each project. Our goal is to know about the state of every single tree we help to plant.

Currently we are a four person team: Arnaud, our tech lead, is a true data pipeline plumber who constantly improves our setup and finds better solutions. Nikki, our Product/Marketing Analyst, is the jack-of-all-trades in our team in its best meaning; knowing the company and business model inside out, identifying problems and working towards solving them. And we have our latest joiner, Elise, who is a really great data storyteller. She understands the needs of the stakeholders and how information can be meaningfully visualised for them

P: What are your teams biggest technical and organisational challenges?

Zs: We are a small team with a lot of stakeholders: currently we have 8 stakeholder teams. This means in the past year we had to completely change our mindset of what a data team is. Previously we operated as a service team where we had ad-hoc requests from the teams. With the resource constraints our backlog grew, the team became frustrated and our stakeholders became under-served. Balancing our legacy infrastructure and soaring stakeholder needs was a real struggle.

We understood this was not sustainable so we changed to a data product team mindset. Now we identify the product needs with user stories, and build our products around this. We introduced agile methods and work in sprints. We also educate our users on how best to use these products and integrate their feedback on the data UX design. We have seen that this has increased the velocity of the team but also data has integrated much better into the day-to-day of Ecosia.

Our main technical challenge is handling truly big data sets. There are almost no BI or ETL products or solutions that we can use out of the box. This means we are regularly dealing with challenges to optimise solutions for our needs along the entire data pipeline. For this reason, it’s hard for us to find partners who can usefully consult us on these challenges, which is why it was great for us to work with Pipeline Academy.

Ecosia.org

P: Ecosia acts very transparently when it comes to financial information and respects the privacy of users. How is this translated into the day-to-day operations of a data team?

Zs: We are being transparent both inside the team and towards our stakeholders as well. We have a data-as-code infrastructure so everything we produce is transparent and documented.

We are also showing things honestly as they are. We don’t have user groups with different levels of data access, everyone in Ecosia can see everyone else’s dashboards. We find this very effective as well as there is always some cross-over of which data is needed for the different teams and sharing them openly increases trust in the data.

P: If you would start out in data today, what kind of skills would you focus on in the first place?

Zs: I think understanding where one’s own strengths lie and how they fit into the data products that teams deliver. There are so many aspects of data products and all of them are crucial in producing great output. From understanding stakeholder needs to planning, execution, building the pipeline, designing the UX for users and actually understanding the trends. To master them all is almost impossible, so understanding your own growth path and fitting it together with the team needs is essential. Having good skills in SQL and python helps a lot though, and understanding tools that became industry standards like dbt or Airflow gives one a good advantage.

P: You've hired Pipeline Academy to support your team's move towards analytics engineering, a hot topic in the world of BI. What kind of outcomes do you expect from the upskilling training?

Zs: For us as a small and high velocity team it has been essential that we make the right tooling decisions and avoid some of the pitfalls those tools provide. So for us, trial and error is rarely a path we can take. We started to work together with Pipeline Academy because we knew Daniel and Peter have extremely in-depth knowledge of the tooling landscape and are also really embedded in the data community, so they can really well advise us on datatools across the entire pipeline. I also like how they didn’t offer us a solution as the Holy Grail but gave us an evaluation framework by which we can make these decisions ourselves.

P: I understand your team is growing, and you are looking for data engineering talent. What kind of hard and soft skills do you think makes a newcomer in data stand out?

Zs: We need someone who understands the principles of data infrastructure as code and can help us to be even more reliable towards our users. We’d like to have someone with the mindset who sees data products in the holistic way we do in the team and doesn’t just work on engineering tickets. We would like to have someone who is excited about learning but also can mentor others in the team. It would be great to have someone who has experience with Airflow, dbt and with different cloud providers.

The Data Janitor Letters - April 2021

2021-05-31T00:00:00+00:00

Data engineering salon. News and interesting reads about the world of data.

Diving Deep on S3 Consistency
Dr. Werner Vogels, CTO, Amazon.com

We built S3 on the design principals that we called out when we launched the service in 2006, and every time we review a design for a new feature or microservice in S3, we go back to these same principles.

We were promised Strong AI, but instead we got metadata analysis
Cal Paterson, contract software engineer

How simple structured data trumps clever machine learning.

A Comprehensive Framework for Data Quality Management
Chau Vinh Loi, Data Scientist, ANZ Australia

How to monitor and maintain Data Quality to make sure the data meets certain standards for specific business use-cases

Layering Your Data Warehouse
Mitchell Silverman, Analytics Engineer, Spotify

I never thought I would be comparing my work in data engineering to the great Mike Myers but Ogres and Data Warehouses have a lot in common. Both are misunderstood by most and both can save the day when called upon.

The missing piece of the modern data stack
Benn Stancil, Chief Analytics Officer + Founder, Mode

The core problem is that there’s no central repository for defining a metric.

Benchmarking SQL engines for Data Serving: PrestoDb, Trino, and Redshift
Anton Peniaziev, data and machine learning engineer, Explorium

Data serving is a special business case, which demands real-time low-latency on small queries, the ability to scale and withstand abrupt peak loads, and high concurrency.

Introducing a Dataflow Management System Backed Up by Prefect, AWS, and Github Actions
Maikel Penz, Senior Data Engineer, Spidertracks

Software infrastructure 2.0: a wishlist
Erik Bernhardsson, Ex-CTO, Better

I mean, as a user, I can set up a static website in AWS, but it takes 45 steps in the console and 12 of them are highly confusing if you never did it before.

How Litestream Eliminated My Database Server for $0.03/month
Michael Lynch, Builder of @TinyPilotKVM

Data persistence for people who hate database servers.

It is time to fulfill the promise of CI/CD
Charity Majors, Cofounder/CTO, @honeycombio

Why your software should be auto-deployed within 15 minutes after you merge it, with no manual gates. This is the key to high performing teams and high-quality software.

Top 5 Specialisations for Data Engineers

2021-05-05T00:00:00+00:00

When mankind embarks on a journey of exploring a new subject matter, the first part of the ride is usually bumpy and full of ideas and initiatives that turn out to be erroneous in hindsight. It is very true that failing is essential for learning - just ask the people who call themselves "serial entrepreneur" on LinkedIn. When working on figuring out how to approach a problem we tend to systematically reinvent and restructure hypotheses, tools, methodologies and processes over and over again in order to maximise the likelihood of a desired outcome, and later on we try our best to optimise all this for efficiency. This is an established form of discovery that aims at generating value, all in the name of capitali... prosperity.

Just take the evolution of the automobile or the continuous development of modern medicine as an example. I would argue, that we as a society are just getting started with finding out how software development and maintenance should be sustainably institutionalised, and the growing pains of today can be vividly felt when talking to people who have spent some time working in tech. Setting up organisational structures, distributing tasks, managing workloads, streamlining communication, exploring new methods of collaboration are all popular subjects for bestsellers for a reason. As a consequence of the quickly evolving consumer expectation towards the products and services built upon software, we keep iterating and we remain on the lookout for somebody with answers - or at least with a totemic acronym we can put on pink post-it notes.

But within this circus, there are smaller stages with plays of comparable conflicts: the stage of data being one of them. The quickly changing nature of the tools and processes applied to generate value out of the growing amount of available data requires constant reiteration of the skillset of professionals working with it, and a continuous realignment of organisational roles and responsibilities. You can multiply this uncertainty-soaked complexity with demographic, cultural, geographic and organisational factors that all have an undeniable influence on the daily business of a team working with technology.

This environment and the constant change is what makes it so difficult to create blueprints for long-term career pathways within data, therefore everyone who is planning on staying en vogue should care to remain ahead of the curve... somehow.

how to Level up

The established, basic trifecta of data roles in 2021 consists of the data analyst, the data scientist and the data engineer - yet a lot is happening when you look at further career development. The most straightforward way of increasing the value one generates is to take on more decision making responsibility (financial responsibility, personnel responsibility) - think Senior Data Engineer, VP Data Engineering, Head of Data etc. But there are more and more specialised roles popping up that mix the fundamental competencies of a data engineer with different group of expertise to create a powerful combination. These are some of the roles that I see are highly sought after on the job market, these skill-combos are carrying loads of value to businesses who are ultimately looking for casting the right actors to star the show.

Interestingly, we can observe the participants of the first cohort of Pipeline Academy already getting a feel for their own personal paths and how they are starting working towards that goal. We are putting a lot of emphasis on the fundamentals of data engineering at the bootcamp, yet ultimately it's the combination with the unique professional backgrounds and personal ambitions of our students that make them really stand out on the job market.

Let's take a look at five selected roles that are worth a deeper dive.

1. Analytics Engineer or BI Engineer

The role of the Analytics Engineer or Business Intelligence Engineer started popping up in 2018 in more and more blogs and articles mainly thanks to this thing called dbt.

“Today, if you’re a “modern data team” your first data hire will be someone who ends up owning the entire data stack.”

This title describes basically a pissed off data analyst who is by default able to generate insights out of data, and with some added engineering skills they become an end-to-end powerhouse for setting up solid BI pipelines without the help of an additional engineer or software developer. This idea is part of a broader trend of enabling data consumers to access what they want without intermediaries. Analytics engineers are highly valued especially in the days where hiring data engineers is fairly difficult due to a lack of available supply. Some are even going as far as saying that "80% of analytics is effectively data engineering"... Definitely one of the most promising roles you can enter in 2021. There are a handful of forward-thingking scale-ups who are already hiring for this role in Berlin.

Competencies: analytical domain knowledge, SQL, Python and ETL/ELT knowledge, dbt, data visualisation and storytelling skills.

2. Machine Learning Engineer or AI Engineer

ML engineers are data scientists who can productionize their models in order to solve business challenges - this might sound like an oversimplified definition, yet that's pretty much it. Does this really get you into one of the best paid positions within data? Well, kind of. Currently I see two directions to arrive at this role: either you come from a data science angle and learn how to set up architectures and integrate sophisticated models into the equation in a robust manner (notice the DS —> DE pivot), or you come from the computer science/software engineer/data engineer realm and you level up your understanding of the mathematical/computational models you are supposed to build or paste into your pipelines. The key expectations include:

"Translating the work of data scientists from environments such as Python/R notebooks analytics applications.
Creation of web services/APIs for serving ML/AI model results and enabling access to customers or internal teams.
Automating model training and evaluation processes.
Automating feature engineering, ensuring data for model training is cleaned and readily available and facilitation of the flow of data between ML/AI models and an organization’s data systems."

And the list goes on. You see, putting something 'into production' is what does the trick, this is where the business value is generated. Some critics even go as far as saying:

"Classic algorithms + domain knowledge + niche datasets are going to solve most real problems, not deep neural nets. Most of us aren’t working on self-driving cars."

... and there is certainly a lot of truth to that.

Competencies: experience with the fundamentals of machine learning models, computer science or software engineering background combined with the latest data engineering expertise, SQL, Python and the command line, concepts like Continuous Delivery and Continuous Integration, understanding of Docker and maybe Kubernetes.

3. DataOps and Machine Learning Ops

Data architectures need maintenance, sometimes significantly more than traditional or legacy software stacks to perform and keep the output quality at a high level at all stages of the lifecycle. The reason for this are the constantly changing tools and business requirements that demand attention, and of course the pipelines that need fixing as a result. Shouldn't this be done by the data engineers, you ask? Well, not really, but you'll definitely hear about small-scale data teams that don't cultivate ops in general. Mature (or bold) data organisations who have identified proper machine learning usecases with a positive ROI are the ones who first have to start thinking about maintaining the systems their ML engineers have put in place as well. Running the operations-side of data is often considered the least glorious part of the job, yet it's a very sought-after and rewarding role.

Competencies: SQL, Python, command line, understanding algorithmic complexity, operations experience, knowledge of concepts like CD/CI, logging, monitoring and profiling, understanding of Docker and Kubernetes.

4. Data Product Manager or Data Product Owner

In order to successfully lead a cross-functional team one requires not only a feel for prioritising and managing multiple tasks in parallel, but also excellent communication and social skills. But as we dive deeper and deeper into the dark seas of technical complexity when designing and deploying software and data products, navigating becomes exponentially tougher. If companies would like to meet consumer needs and deliver the right features in an efficient and agile manner, the people in charge of the product roadmap need to be aware of the high-level technical requirements and consequences of their decisions (think data infrastructure costs, maintenance effort, data governance and quality across the org, stakeholder expectation management etc.). It is becoming clear that while the field of building websites or mobile apps is experiencing more and more commoditisation and is therefore turning simpler and more accessible, dealing with the whole lifecycle of data products is much more complicated and requires and increased awareness of the latest tooling landscape, the methodologies and the lingo in data.

Data engineers who would like to be in charge of a product are well suited to fulfil this role, at the same time I am witnessing how experienced product people are leveling up their game to become data/AI POs by learning the essentials of data engineering at the bootcamp. Data leaders of the future, now is the time to do a deep dive into data infra; and in case ML comes into play in the consumer facing product, there are plenty of accessible online courses available for getting the knowledge you need. As Adnan Boz (founder of the AI Product Management Institute) puts it:

“Since the development lifecycle of AI projects is based on “searching” rather than “planning”, companies need professionals who are trained to look at products as optimization problems rather than a programming problems.”

Competencies: organisational skills, communication skills, understanding of the architecture and deployment process of data products and their consequences for the business and the user, stakeholder management.

5. Data Governance Manager or Data Czar

Talking to data professionals who are tired of hearing platitudes about a company being "data-driven", a shared pattern seems to surface: data governance is not a nice-to-have anymore, but essential for utilising your data and your data staff. Now, what they are talking about is not the superficial "oh, sure, we more or less understand GDPR and we do have a data warehouse"-type of governance, but rules that are transparently applied across the whole organisation. The tendency of giving this role to one dedicated person or team who oversees and manages all potential conflicts and helps resolve them within the organisation is becoming more and more popular, especially as the legal responsibility of dealing with data is growing (and so are potential fines for companies). Data czars manage data access for users, deal with data lineage, data quality, data dictionaries, support feature development with guidelines about collecting, storing, leveraging and sharing data in and outside of the company. However they are not to be confused with the data security manager (the Germans have a word for it: Datenschutzbeuftragte).

This role will become a mainstay: take the increasing complexity of data management in general, mix it with the continuously evolving legal environment and add the growing demand of generating more and more value for shareholders through using data. It's a very fine line between becoming an overprotective naysayer versus being a thoughtful enabler of data teams.

Competencies: understanding data privacy and security best practices and legal requirements, stakeholder management skills, database management, encryption and decryption techniques, data quality measurement, documentation and advocacy.

+1

...and something I did not mention above, although it is a real point of differentiation from the employer perspective: industry-specific expertise aka domain knowledge is a major asset. Understanding the mechanisms of a vertical or the language used in a certain sector will always bring you bonus points when applying to a job. If you leverage that the right way it can secure you a head start in the battle for a position.

The only constant in life is change

The lines between these roles are blurry at best, and they will undergo significant changes in the coming years. Engineering skills within data remain predominant however, they are kind of a superpower that can be combined with a surprising amount of unrelated competencies.

I guess the main takeaway should be:

“Most employers expect you to have overlapping skillsets. I feel like in the end it’s not about who gets wiped out, it’s about who is versatile enough to constantly adapt to the ever changing industry.”

If you are working on the frontlines of technology, you have to face the fact that this simply is the nature of this very quickly evolving environment, and you have to accept that being successful requires you to be curious and determined. Indeed adaptation is the magic word and since we’re far from being done exploring technology as a method, we can expect more change to come.

At the same, this challenge is what makes data and tech such an exciting and rewarding place to be in.

PS: In the next episode of Career Specialisations for Data Engineers: Sustainable Data Architect, Data Quality Engineer etc.

PPS: If you think I've missed something, or you have suggestions for the above - feel free to shoot me an email.

The Data Janitor Letters - March 2021

2021-04-28T00:00:00+00:00

Data engineering salon. News and interesting reads about the world of data.

Star Schema Benchmark: ClickHouse Nails Cost-Efficiency Challenge Against Druid & Rockset
Alexander Zaitsev, Co-Founder & CTO, Altinity

When we use the same schema approaches as Druid and Rockset, ClickHouse significantly outperformed both whilst using the cheaper AWS setup.

Speeding up SQL queries by orders of magnitude using UNION
Ben Levy and Christian Charukiewicz, Partners and Principal Software Engineers, Foxhound Systems

SQL’s UNION operation is not usually thought of as a means to boost performance.

Building the world's fastest website analytics
Jack Ellis, Co-founder, Fathom

Performing a migration is such a high adrenaline, stressful task.

SQLite is not a toy database
Anton Zhiyanov

Whether you are a developer, data analyst, QA engineer, DevOps person, or product manager - SQLite is a perfect tool for you.

Taming the Dependency Hell with dbt
Rafael Barbosa, Data Engineering Team Lead, WeTransfer

We’re using dbt to simplify the management of the views and build more trust in the data we store in our data warehouse.

Storage size and generation time in popular file formats
Barthelemy Ngom, Solution Architect & Data Engineer, Adaltas

For archiving it is preferable to choose column based format and ORC.

Why We Don’t Use Docker (We Don’t Need It)
Nicky Rees, MeeZeeCo

We get a single binary.

Data Engineering Confessions #15: Adam Vincze

2021-04-14T00:00:00+00:00

In this interview series we ask data engineers how they ended up choosing this career and how they see the future for it.

You can subscribe to the show on Youtube, Spotify and Apple Podcasts to get notified about new episodes.

Data Engineering Confessions #14: Alex Angelini

2021-03-31T00:00:00+00:00

In this interview series we ask data engineers how they ended up choosing this career and how they see the future for it.

You can subscribe to the show on Youtube, Spotify and Apple Podcasts to get notified about new episodes.

The Data Janitor Letters - February 2021

2021-03-30T00:00:00+00:00

Data engineering salon. News and interesting reads about the world of data.

Why I Built Litestream
Ben Johnson, Builder, Litestream

Despite an exponential increase in computing power, our applications require more machines than ever because of architectural decisions made 25 years ago. You can eliminate much of your complexity and cost by using SQLite & Litestream for your production applications.

Google admits Kubernetes container tech is so complex, it's had to roll out an Autopilot feature to do it all for you
Tim Anderson, Senior Reporter, The Register

More expensive, less flexible, but easier and safer to use.

Serverless Data Pipelines Made Easy with Prefect and AWS ECS Fargate
Anna Anisienia, Python Engineer, TrailStone Group

The easiest way to orchestrate your Python data pipelines.

Apache Superset Tutorial
Joseph Machado, Senior Data Engineer, Narrativ

In this post we go over the architecture of Apache Superset, connect to a warehouse and learn how to build charts and dashboards.

Make Boring Plans
Camille Fournier, Head of Platform Engineering, Two Sigma

You must do the work to go beyond vision, create concrete actions, and make boring plans.

Hacker hacks into startup, rewrites complex backend with PostgreSQL and cron, forces CEO to fire 40% of devs and devops
Matthew Solenya, Editor in Chief

Alarmed, other startups in the industry are hiring more security specialists in order to reduce the risk of so-called "green-hat hacker" attacks.

The Kilobyte’s Gambit ♟️💾 1k chess game
Matt Round, Óscar Toledo G., Pinot W. Ichwandardi

Data Engineering Confessions #13: András Dötsch

2021-03-17T00:00:00+00:00

In this interview series we ask data engineers how they ended up choosing this career and how they see the future for it.

You can subscribe to the show on Youtube, Spotify and Apple Podcasts to get notified about new episodes.

The Campus

2021-03-10T00:00:00+00:00

So many of you have been asking us about our new campus... nah, I'm kidding, nobody is really thinking about classrooms or offices nowadays, because we tend to assume that every form of interpersonal interaction has been banned into the online space (and to our homes at the same time). Yet, as a firm believer in the superior value of in-person education I am very excited to walk you through our new location and tell you about what you should expect when joining Pipeline Academy.

Writing about the place we’ve picked for our bootcamp feels somehow like a strange episode of MTV Cribs: the camera is too shaky, the tour feels like an exercise in humblebragging, and ultimately I'm just showing you stuff I don't own. Nevertheless…

Let me give you the tour: The Kiez

Let's start with the area, which is favourite to a lot of people in Berlin. The campus is located on the western end of Kreuzberg at Gneisenaustraße 27, close to Mehringdamm. This neighbourhood is defined by the variety of cozy parks, friendly streets and various international restaurants, which turns going out for lunch from an ordeal to a real highlight every single day. Historic spots like Gräfekiez, Viktoriapark, Tempelhofer Feld are just a few walking minutes away, and the ever-changing Bergmannstraße with the Marheineke Markthalle is even closer offering more options for grabbing groceries and food. (If you'd like to get a vivid feel for how this Kiez has been evolving, go check out Gerd Nowakowski's lovely piece in the Tagesspiegel about it - in German.)

Getting there should not be a problem for anyone, there are plenty of designated bike lanes in the surrounding area and public transport options are also plentiful. The closest stop right across the street is U7 Gneisenaustraße, but you can also take the U6 to Mehringdamm or the buses 140 and M19. This means that wherever you are commuting from within the city, it should be easy and hassle-free.

come on in: The campus

The area has a very rich history, and most of the surrounding buildings that survived the war are under heritage preservation. This is what went down behind these walls in the past 120 years:

"Der Wohn- und Fabrikkomplex in der Gneisenaustraße 27 wurde im Jahr 1900 errichtet. In früheren Jahren befanden sich hier unter anderem kleinere und größere Fabriken wie eine Messinglinienfabrik und Schriftgießerei, die Armaturen-Apparate-Fabrik Preschona von A. Meyer oder eine Lampenfabrik (alle 1943). Von 1949 bis 2013 diente das fünfgeschossige Gebäude als Produktionsstätte der Firma ROKA Robert Karst GmbH & Co. KG, die Steckverbindungen für die Autoindustrie fertigte, die danach einen neuen Standort bezog."

The building with the old facade is still visible on Google Street View.

The campus has been established 2015 when the generous renovation of the magnificent old industrial building has been completed. Our partner and host, the CIEE Global Institute has transformed the building into a "full-fledged “vertical campus” where living, academic classes, dining and socializing co-exist, creating a state-of-the-art interdisciplinary space. Functioning as a community incubator, the integration of the campus encourages learning both inside and outside of the classroom and facilitates connections that are vital to open exchange and debate."

Here is the part where I gladly refer you to Chris Mosier's photographs on ArchDaily, the photos above are their courtesy. I think they give you a better idea about how it feels to hang out on campus than I could ever do it justice.

The generous dorm on the site is available for our future students, so in case you don't have a place to stay in Berlin and would like to avoid the potential pitfalls of finding an apartment for the three months of the bootcamp, you can get an affordable all-inclusive student apartment at a really cool location. Just hit us up for more info.

Where the magic happens: The classroom

Our classroom offers a versatile and flexible learning environment for our students, it allows us to adjust the setting according to the different requirements defined by the changing didactic methods we apply throughout the 12-weeks of the data engineering bootcamp program. You can find us behind the bright yellow doors right next to the green Ampelmann.

Although the classroom is being used as a streaming studio at the moment for delivering a professional online experience to our students (I am writing this March 10th, 2021), we have the option to host in-person 1-on-1 sessions with our participants who ask for extra support. Both Daniel and I are eagerly waiting to have the whole class around and to finally working together in a non-virtual setting, but health and safety come first.

Exciting times are ahead of us, and we’re glad to have found a place that supports the launch of our small yet ambitious endeavour of building up Pipeline Academy. This is the part when the tour ends, but instead of saying goodbye, I’d like to take the opportunity to invite you to say hi if you are around!

Don't forget, if you'd like to get serious about a newly evolved profession that is likely to become an essential mainstay of the digital/data/innovation economy, an enabler of sustainable digital product development, you can always join our data engineering training with a Bildungsgutschein from the AfA basically free of charge.

Data Engineering Confessions #12: Mikalai Syty

2021-03-03T00:00:00+00:00

In this interview series we ask data engineers how they ended up choosing this career and how they see the future for it.

You can subscribe to the show on Youtube, Spotify and Apple Podcasts to get notified about new episodes.

The Data Janitor Letters - January 2021

2021-02-23T00:00:00+00:00

Data engineering salon. News and interesting reads about the world of data.

We Don't Need Data Scientists, We Need Data Engineers
Mihail Eric, Machine Learning Scientist, Amazon Alexa AI

There are 70% more open roles at companies in data engineering as compared to data science.

What can we learn from SQL's 50 year reign? A story of 2 Turing Awards
Felix Schildorfer, Chief Data Scientist, First Retail Inc.

The relational data model was introduced in 1970 and has dominated for 50 years. What led to its success? Building on first principles and Bushnell's law.

Automating my job by using GPT-3 to generate database-ready SQL to answer business questions
Brian Kane, Data Engineer, SeekWell

Now, I've got a GPT-3 instance that takes a plain English question and translates it to SQL that really works on my database.

How Shopify Is Building Their Production Data Warehouse Using DBT - Episode 171
Michelle Ark + Zeeshan Qureshi, Senior Data Engineer + Tech Lead/Engineering Manager, Shopify

Structure the project to allow for multiple teams to collaborate in a scalable manner, have the additional tooling to address the edge cases, and the optimize the continuous integration process to provide fast feedback and reduce costs.

Introduction to Databases for Data Engineers
Oleg Agapov, Data analyst/BI, GOG.com

Data engineers need to have a broad knowledge about ways of storing and processing data. Most of this knowledge will come with practice. But is still important to understand general ideas behind all concepts I've explained here.

Kafka As A Database? Yes Or No – A Summary Of Both Sides
David Xiang, Engineering Team Manager, Squarespace

I personally have never used a Kafka log as the source-of-truth for my data. Software development is hard enough as it is, even when trying to go “by the book.”

An unlikely database migration
Brad Fitzpatrick + David Crawshaw, Late Stage Co-Founder + CTO, Tailscale

The goal is to keep development speed as close to the early days of JSONMutexDB, when you could recompile and run locally in a fraction of a second and deploy ten times a day.

Achieving 11M IOPS & 66 GB/s IO on a Single ThreadRipper Workstation
Tanel Põder, Co-founder, Gluent Inc.

Modern disks are so fast that system performance bottleneck shifts to RAM access and CPU.

Data Engineering Confessions #11: Micha Kunze

2021-02-17T00:00:00+00:00

In this interview series we ask data engineers how they ended up choosing this career and how they see the future for it.

You can subscribe to the show on Youtube, Spotify and Apple Podcasts to get notified about new episodes.

Learn from the Pros: Guest speakers of our first cohort

2021-02-15T00:00:00+00:00

While following the core 12-week curriculum of the data engineering bootcamp, (almost) every Thursday a selected guest speaker will share their experience on the latest domain that’s being covered, and they will be available for AMA for our students: team structures, tech stacks, hiring processes and the future of the trade - no stone will remain unturned.

Data engineering feels like a black box for a lot of people even after spending some time with the subject matter, and I have to admit, it can be difficult to imagine the everyday life of a data professional. That is why we’ve invited some experienced practitioners to our (currently virtual) campus: our participants will have an opportunity to learn how they can put their knowledge to use, and see how others have built their careers on competences in the intersection of data and software engineering.

We’ll have freelancers who are enjoying working remotely, industry veterans who are on the lookout for new talent for their teams, subject matter experts who have designed our built significant parts of the digital products all of us use on a daily basis… and so on. It’s a very diverse list of perspectives.

Without any further ado, please check out the list of guest speakers for our very first cohort (in order of proposed appearance):

Balázs Krich
Lisbon

Balázs is a data professional with exposure to product, data science and engineering. Having worked in media, banking, real estate, supply chain and e-commerce he also contributed to non-profit projects and data journalism as an Aaron Swartz Fellow at the Open Society Archives. A recent project of his covered collecting and standardizing all transactions in the European Strategic and Investment Funds (ESIF) between 2014 and 2020.

LinkedIn, GitHub

Bence Faludi, Singapore

Bence is a Data Engineer at Facebook. He made major contributions to several open source data munging tools like mETL, hamustro and night-shift. A recent conference talk of his is Data Architecture 101 for Your Business. He was featured in the legendary #idataengineer podcast.

LinkedIn, GitHub

Dr. Martin Loetzsch,
Berlin

Martin is the Chief Data Officer at Project A Ventures, the operational VC. He built several data teams, and made major contributions to the open source ETL tool, mara. Two highlights of his recent conference talks are Data Warehousing with Python and ETL Patterns with Postgres. He was also featured in our #idataengineer podcast.

LinkedIn, Website

Stefan Urbanek, Taipei

Stefan is a Data Infrastructure Architect who contributed to the infra of Facebook, Squarespace and Orange. His open source projects include the OLAP tool Cubes and ETL tool Bubbles. He talks about Cubes at Data Warehouse and Conceptual Modeling with Cubes 1.0.

LinkedIn, Website

Miklós Koren, Budapest

Miklós teaches reproducible coding practices to economists to help them maximize their scientific impact. He believes in the command line, plain text, and that every problem can be solved with the right combination of Stata, Python, Julia, git, and make. He is Professor of Economics at Central European University and the Data Editor of the Review of Economic Studies, a leading scientific journal.

LinkedIn, Research, Training, Twitter

Andreas Dewes, Berlin

Andreas is a serial entrepreneur whose current company, KIProtect makes data security and data privacy easy for companies and organizations.

LinkedIn, GitHub

Rahul Jain,
Berlin

Rahul is a Principal Engineering Manager of Business Intelligence at Omio with decades of experience both as a manager and engineer.

LinkedIn, Blog, Mentoring Club Profile, Twitter

Daniel Oliveira Filho,
Berlin

Daniel is a Lead Production Engineer at Shopify. He is a DevOps expert and practitioner who is also interested in the scaling questions of the online gaming industry.

LinkedIn, GitHub

Flávio Clésio, Berlin

Flávio is a Staff Data Engineer at Artsy. A machine learning and data engineer with teaching experience in data warehousing. A recent conference talk of his is Preventing Revenue Leakage and Monitoring Distributed Systems.

LinkedIn, Website

Data Engineering Confessions #10: Dr. Martin Loetzsch

2021-02-03T00:00:00+00:00

In this interview series we ask data engineers how they ended up choosing this career and how they see the future for it.

You can subscribe to the show on Youtube, Spotify and Apple Podcasts to get notified about new episodes.

2020 - A Year To Remember

2021-01-28T00:00:00+00:00

Some of you might think we are writing this blog just for you, our dearest readers, but the truth is that it mostly serves documentation purposes so in 10 or 20 years we can look back, reminisce and wonder how young and naive we were about starting a coding school in 2020. This post is basically a collection of events of the first 8 months of Pipeline Academy that Daniel and I consider significant in some way, so if you are new to our journey this might be a good starting point.

We started working on Pipeline Academy in March 2020. One could say our timing was not optimal, and I would wholeheartedly agree. As for many, the last year was about figuring out ways to make our business work despite of all the mess that was happening in the realm of the global economy, and our private lives. But it's all about attitude... they say.

Right at the very start we've organised a mini online-bootcamp for data engineering for a lovely bunch of people, and we called it the Summer Camp. Here's part #1 and part #2 about how it all happened what we've learned. (Btw: stay tuned, the dates for summer camp 2021 are already marked in our calendars…)

There are plenty of unspectacular things that kept us busy, however they were necessary: developing and testing the curriculum, dealing with Berlin's very own bureaucratic Kraken when founding a company, setting up corporate partnerships, updating Zoom every single day... just to name a few. The launch of our bootcamp was originally planned for the second half of 2020, but we've decided to postpone as a result of the volatile outlook caused by corona. But spreading the word about what we're about to launch was never put on hold.

We've started a monthly Open House meetup session (virtually for now), so we can talk directly with folks who are interested in joining our training. It's happening every first Tuesday of the month. It helped us bridge the long cold Berlin winter without the beloved corona-compatible Data Stacks Picnics in the Park, which turned out to be a huge hit.

The need for clear guidance for navigating the world of the data engineering profession seems endless, so we put a lot of effort into sharing our POV. Daniel's monthly curated reading recommendations (aka The Data Janitor Letters) received a lot of positive feedback from the data engineering community. Also, if you are trying to get access to the competences of a data engineer, check out his highly successful guide about getting started with data engineering on a budget.

Yours truly has penned some words about learning paths for data engineers, about the state of bootcamps in general and about data engineer salaries in Germany to increase transparency within the field.

Not enough, you say? We got you: here are some of Daniel's favourite keynotes and you can see him on the stage of various conferences right here. Plus the chat with Tobias Macey from the Data Engineering Podcast about Pipeline Academy's pragmatic and commonsensical approach to data engineering challenges and the bootcamp itself.

Our merchandise was very 2020 as well:

Oh, I've almost forgot: we've launched a short podcast series called #idataengineer to share the perspectives of other data engineers, and for Christmas we’ve opened the 24 tiny virtual windows of our data engineering advent calendar with hints and drops of engineering wisdom for everyone.

Looking at my calendar, we've actually met a lot of tech and data professionals both in-person and via video calls. To those of them reading this: your encouragement and input has been essential for our progress, so shout out to you, the humble and decent people of the tech/data/startup scene of Berlin!

We've been even featured on the popular and trashy @bestofkleinanzeigen instagram account. Everything happened so fast...

We've started supporting companies with their upskilling programs and consult data teams on data engineering topics like infrastructure design and architecture. It's an exciting journey and we're very happy to have smart and engaged startups and scale-ups in our growing clientele.

But our drive to explore and to learn goes even further: currently we're deeply involved in a transatlantic coaching-mentoring program, but this is something for the 2021 recap I guess...

At the very end of 2020 we've been greenlit as an official partner of the Agentur für Arbeit, which means that we can support people who have been made redundant during the pandemic or have to work in Kurzarbeit. Bildungsgutschein is the name of the game.

Kicking off 2021 on a high note: incorporating sustainability into our methodology and data engineering curriculum is something we’re extremely proud of, and we truly believe that it will deliver tangible benefits to our future bootcamp participants.

And there is so much more to come in 2021.

Stay healthy, folks.

Data Engineering Confessions #9: Ayan Putatunda

2021-01-26T00:00:00+00:00

In this interview series we ask data engineers how they ended up choosing this career and how they see the future for it.

You can subscribe to the show on Youtube, Spotify and Apple Podcasts to get notified about new episodes.

The Data Janitor Letters - December 2020

2021-01-14T00:00:00+00:00

Data engineering salon. News and interesting reads about the world of data.

An analytics engineer is really just a pissed off data analyst
Seth Rosen, Co-founder, Hashpath

... who has the tools and motivation to make things better for everyone else.

Avro Schema Evolution Strategies on Kafka
Tomasz Kaszuba, Java Big Data Engineer, Swiss Re

I think the only place that the schema registry makes sense is when controlling 3rd party connections or in simple Kafka architectures.

The Big Little Guide to Message Queues
Sudhir Jonathan, Technical Architect, Qube Cinema

Fundamental concepts that underlie them, and how they apply to popular queueing systems available today.

Planning joins to make use of indexes
Zach Musgrave, Software Engineer, DoltHub

Dolt is Git for Data. It's a SQL database that you can clone, fork, branch, and merge.

Back to the '70s with Serverless
Cees de Groot, Principal Software Engineer, Canary Monitoring, Inc.

History will repeat itself, of course.

Going from 5K to 3M Messages/sec with 2ms Latency
Market Pulse

The evolution, failures and design decisions behind one of the world’s largest real-time, high-frequency and low-latency streaming systems.

We Burnt $72K testing Firebase + Cloud Run and almost went Bankrupt
Sudeep Chauhan, Founder, Milkie Way Inc.

This is the story of how close we came to shutting down before even launching our first product, how we survived, and the lessons we learnt.

Introducing: Sustainable Data Engineering

2021-01-10T00:00:00+00:00

Starting 2021 we're enhancing the educational focus of our data engineering bootcamp by integrating sustainability as a core value in addition to transparency, pragmatism and collaborative way of doing things.

Pipeline Academy graduates will become the first data professionals equipped with practical data engineering concepts and best practices that support the environmental, economic and social sustainability of the products they build and the companies they join.

We do this to empower our participants to use their engineering skills consciously taking all stakeholders of the ecosystem into consideration, but also to push data engineering culture in general to the next level. We're more than certain that this expertise will translate into a significant competitive advantage on the job market for our graduates.

But what does sustainability mean in the world of data engineering?

1) Environmental sustainability in data engineering

The architectures designed by data engineers leave a footprint on our planet: the most obvious but often ignored factor when setting up a data stack is energy consumption. Understanding the metrics and core drivers of the ecologic impact is essential for every environmentally conscious engineer. Just imagine, what if architects would ignore the quality and quantity of construction materials and processes they use for their buildings? What about managing data waste (hardware and software)? Have you ever compared how green the major cloud providers are compared to each other? Applying this know-how responsibly results in more efficient and sustainable data systems.

2) Economic sustainability in data engineering

Building and scaling infrastructure can quickly become a serious cost driver for organisations collecting and leveraging data actively. The blurry and deliberately incomparable pricing structures of competing data management tools make decision-makers without a clear guiding framework face a tricky situation with high financial risks. Applying the right KPIs, anticipating business needs and considering technical constraints can lead to meaningful cost reductions for tooling, and the contribution of the responsible experts will support the long-term success of their businesses. Our advisor, Dr. Martin Loetzsch is the Chief Data Officer at Project A Ventures, and their Sustainability Playbooks are among the useful guiding lights for us.

3) Social sustainability in data engineering

For the last decade, data has been treated as the new oil for economic growth. But just like with fossil fuel, there are certain social consequences of managing and using data for business purposes. Even in the current regulatory environment (that is expected to become more and more strict in the near future), data engineers have one of the most central roles when it comes to informing stakeholders and executing engineering work according to guidelines like GDPR (General Data Protection Regulation) and CCPA (California Consumer Privacy Act). Addressing privacy and security concerns proactively, understanding the potential for abuse when working with user data, having a data management/governance in place are just a few of the concerns data professionals have to deal with on a daily basis. But it does not stop there: creating a healthy working environment for data teams by managing roles and responsibilities, increasing the transparency for their work to manage expectations are just as important for sustainable careers and successful work. To name an example, Maciej Cegłowski of Pinboard talked about this back in 2015.

Why are we doing this?

Your opportunity

The primary purpose of Pipeline Academy is to help individuals learn data engineering fundamentals so they can improve on their career opportunities. Daniel and I focus a lot on ideas that increase the long-term effectiveness of the content and methodologies we use. When observing the tectonic shift in consumer behaviour towards a more environmentally and socially conscious economy, we can see that it presents a great opportunity for every professional today. Learning how to create products or manage a business so they do little to no harm to others is becoming a huge competitive advantage on the job market, and will eventually be essential for tomorrow's purpose-driven leaders. In simple terms: we give our bootcamp participants not only the technical know-how about building data infrastructure, but also context about their responsibilities and options for managing its impact.

Our shared responsibility

Data engineers set up infrastructure that impact our social and ecological environment, and it is their duty to design systems that consider the ecosystem of stakeholders it is interacting with. Designing software architectures and optimising operational efforts will become much more pronounced mid term: employers selling i.e. eco-friendly products are going to hire data professionals who can execute on this specific value proposition. As educators, we have the power to equip the future architects and builders of data platforms with the knowledge that enables them to create a better working and living environment. They will build the systems that power the technological reforms in mobility, healthcare and education in the upcoming years, and they will educate future generations of data experts - and this multiplier effect makes us even more aware of our influence. Check how your company or a company you're planning to join does in these terms.

Rooted in our resourcefulness

In the last 15 years hyper-growth has become the standard expectation towards tech companies, which resulted in an unsustainable business- and working culture that does not support the long-term coexistence of social and environmental stakeholders. With Pipeline Academy we're building a bootstrapped educational institution that we'd like to establish long-term, and we won't sacrifice our lean DIY-attitude and minimalistic approach for growth. Conscious environmental management of our own organisation, symbiotical coexistence with our partners, and actively fostering a sustainable data and learning culture with our educational concept are proof of that. Compared to some other programs we put the emphasis on long-lasting value in handmade quality with a focus on the individual, and avoid ephemeral and trend-driven promises addressing the mass market.

Sustainable data engineering in practice

The sustainability concept is integrated into our curriculum in the form of weekly lectures, discussions, case studies, reading materials, but also very hands-on exercises. Some of the topics and concepts we discuss in the bootcamp:

Designing lean data architectures - explain it to your grandma,
Unix philosophy and the Zen of Python,
KISS, figure out what to worry about, learn what not to do,
'Take the long view' (Prof. Galloway), invest in durable knowledge,
Strive for fast and good enough (Pareto and YAGNI),
Code longevity, reuse and recycle code,
System optimisation: remove, retire and delete,
Measuring tooling efficiency: speed, data storage and energy consumption,
Just because you can it does not mean you should - you are not Google, you don't need their stack,
Don't feed the FAANG beast with your data.

This is your chance to get ahead of the curve and learn data engineering enhanced with principles that contribute to a more sustainable software development and data management culture.

PS: We too are learning and exploring along the way, so if you have suggestions for us on how to improve on the subject matter, or would like to cooperate - do let us know! We're always open for meaningful collaborations.

Bildungsgutschein-ready

2021-01-10T00:00:00+00:00

Pipeline Academy has received the coveted AZAV-certification, so from now on you can join our bootcamp with a Bildungsgutschein issued by the Agentur für Arbeit (AfA)! This means basically that in case you are between jobs, have been put on Kurzarbeit or are at risk of losing your current employment, you are very likely to be eligible for a voucher (the Bildungsgutschein) from the German AfA, so you can join our bootcamp free of charge.

Since the launch of Pipeline Academy, our mantra was making data engineering more accessible. This is indeed a huge step both for us towards that direction, just as for the people who would make a great fit for our course but could not afford taking part until now. This is our guide to help you leverage this life-changing opportunity and putting yourself on a sustainable career path.

The Bildungsgutschein can cover your full tuition fee if:

You are unemployed receiving ALG I.
You are on Kurzarbeit
You are employed and are eligible for QCG (Qualifizierungschancengesetz)

Getting a Bildungsgutschein when unemployed

The Bildungsgutschein is a form of direct financial support provided by the German federal state for those individuals who seek support for acquiring new skills at certified educational institution. The goal of this instrument is to help people avoid unemployment or to enable individuals to (re-)enter to the job market successfully and in a sustainable way. It's used either for broadening an existing skillset or for switching careers and starting a journey in a new profession.

The process of receiving a Bildungsgutschein is fairly straightforward: check if you are eligible, select a training program matching your goals, present your case and enroll with your voucher. This is a so called "Soll-Leistung", which translates into an optional offering: based on an evaluation of your individual situation by your designated clerk at the AfA, it will be determined whether you can receive a Bildungsgutschein for the course of your choice. However, the current economic situation forces a lot of people to find a new occupation and the Jobcenter recognizes that by supporting them accordingly.

The process looks something like this:

Schedule a call with us to discuss your plan of joining Pipeline Academy, we're happy to give you advice on how to obtain a Bildungsgutschein.
Register with the Agentur für Arbeit or Jobcenter (if you haven't done so yet) and schedule a consulting session with your contact person to discuss your eligibility and your plan.
Prepare your case for choosing data engineering as your desired future career path and Pipeline Academy, and be ready to answer questions like:
- What does a data engineer do?
- What convinced you to become a data engineer?
- Why is Pipeline Academy the best school to learn data engineering?
- What kind of positions are you going to apply for after the bootcamp (e.g. data engineer, BI engineer, machine learning engineer, data platform manager)?
The market demand for data engineering skills and the number of open positions are very convincing arguments for AfA advisors, as they want to see their clients succeed by getting a great a job.
Once you've received the green light from the AfA, let us know so we can secure your spot in one of our upcoming cohorts and we'll also take care of the necessary paperwork.
Clear your schedule for the 12 weeks of the bootcamp and prepare for learning something that is definitely going to expand your professional horizons!

A couple of relevant points to keep in mind

Do your research: take the time to understand the role of a data engineer, look at the profiles of companies hiring and the trends behind the growing demand for data engineering competences. If you have other training options you are considering, make sure to talk to them to have a better overview and get a feeling for how it feels to be enrolled in their classes.
Prepare for the interview with the AfA: the goal of the AfA is to secure your employment for the upcoming years. If you have figured what path fits to you and what you would like to do in the future professionally, make sure to be able to explain your plan in a simple and clear manner.
Commit: the opportunity that is presented to you via a free Bildungsgutschein is something that you should not take lightheartedly. We're here to do everything in our power to help you get started in data engineering, but we expect you to show up and put in the work: this is how you can set yourself up for a new career.

If you are on Kurzarbeit

During the COVID-19 lockdown a lot of businesses were forced to make difficult decisions in order to secure their existence. One of the measures that has proven very effective was the introduction of Kurzarbeit: employers who suffered a hit during the economic crisis have the opportunity to secure financial compensation for their employees from the federal state in exchange for limiting their working hours. It means practically that employees get to keep their jobs, work less hours, but get reimbursed similarly to as if they would work full-time - this is how massive short-term layoffs are avoided. If you find yourself in such a situation, you might have the time and the chance to learn something new with the support of the Agentur für Arbeit in form of a Bildungsgutschein. Make sure to check-in with your employer's HR, they will support you in the process, but the steps to get there are pretty much the same as outlined above.

QCG aka Qualifizierungschancengesetz

This measure creates a unique win-win-win situation. It's aimed to help employees at risk of losing their jobs: in cooperation with their employers they can participate in a bootcamp that makes it possible for them to avoid unemployment on the long run. On top of funding the training for the employee, the employer receives financial support as well to cover the salary of the employee and social security for during the training. For this, we encourage employees to approach the responsible departments in their organisations to start the application process with the AfA. We also support employers looking for meaningful upskilling for their staff.

Make sure to look into all of the available financing options we provide so you can make an educated decision about this realm. Depending on your eligibility, your current financial situation and future plans you should pick the financing method that fits best to your needs. If you'd like to learn more or have a chat about the pros and cons, feel free to contact us or schedule a call.

Data Engineering Confessions #8: Oleg Soroka

2021-01-07T00:00:00+00:00

In this interview series we ask data engineers how they ended up choosing this career and how they see the future for it.

You can subscribe to the show on Youtube, Spotify and Apple Podcasts to get notified about new episodes.

Data Engineering Confessions #7: Adrian Brudaru

2020-12-30T00:00:00+00:00

In this interview series we ask data engineers how they ended up choosing this career and how they see the future for it.

You can subscribe to the show on Youtube, Spotify and Apple Podcasts to get notified about new episodes.

Data Engineering Advent Calendar 2020

2020-12-25T00:00:00+00:00

Throughout December 2020 we’ve shared a daily dose of semi-esoteric data engineering wisdom on our social media channels (instagram and LinkedIn). This post shall serve as a commemorative monolith you can always turn to when the data engineering gods are not picking up your call.

#1: Don’t write code, solve the problem.

#2: Python is always at hand to pretty print a JSON:

$ python3 -m json.tool some.json

#3: EXPLAIN is your friend.

#4: "Choose boring technology." Dan McKinley @mcfunley

#5: Complicated is better than complex.

#6: What do you do on your CLI?

$ < ~/.bash_history | sort | uniq -c | sort -n

#7: SQLite (2000) has one trillion (1e12) active installs. It's a file with SQL API and window functions.

#8: "Premature optimization is the root of all evil." Tony Hoare (Quicksort)

#9: Keep It Simple (&) Stupid and remember separation of concerns.

#10: Log in to a recently launched container (via @basmatitree)

$ docker exec -it $(docker ps -q | tail -1) /bin/bash

#11: Get details of last failed Redshift load.

SELECT * FROM stl_load_errors ORDER BY starttime DESC LIMIT 1;

#12: "When in doubt, use brute force." Ken Thompson (Go, UTF-8, Unix)

#13: Maintainable is debuggable and testable, and has version control.

#14: Remove all local git branches other than master and the currently used one (via @advincze)

$ git branch --no-color | grep -v 'master' | grep -v '*' | xargs git branch -D

#15: Order of execution in SQL:

FROM WHERE GROUP BY HAVING SELECT [DISTINCT] UNION ORDER BY

#16: "Don't reinvent the flat tire." Alan Kay (Squeak, Smalltalk, OOP, GUI)

#17: Code is dependency. Others' code is dependency squared. Delete, remove, retire.

#18: You can run SQL directly on your CLI on CSV or TSV files with http://harelba.github.io/q/

#19: Queries on MPPs? Use WITH/CTEs, filter with WHERE, SELECT explicitly, avoid JOIN, SORTKEYS are your friends. It's all about scanning less.

#20: “Bad programmers worry about the code. Good programmers worry about data structures and their relationships.” Linus Torvalds (Git, Linux)

#21: The longer a technology lives, the longer it can be expected to live (Lindy effect)

#22: Delete files recursively:

$ find . -name "*.pdf" -print0 | xargs -0 rm

#23: Why PostgreSQL (1996)? It is the open source RDBMS with columnar (cstore), geo (PostGIS), timeseries (TimescaleDB) and REST API (PostgREST).

#24: "Use simple algorithms as well as simple data structures." Rob Pike (Go, UTF-8, Unix)

The Data Janitor Letters - November 2020

2020-12-24T00:00:00+00:00

Data engineering salon. News and interesting reads about the world of data.

Marketers are Addicted to Bad Data
Jacques Corby-Tuech, Marketing Operations Manager, CyberSmart

36% percent of people in the UK use an adblocker.

Personal Data Warehouses: Reclaiming Your Data
Simon Willison, Datasette

AWS S3 — Disaster recovery using versioning and objects metadata
Jacek Małyszko, Data Engineer, Fandom

Accidental removal of data on S3 is something that no Data Engineer on AWS wants to be involved in.

ClickHouse, Redshift and 2.5 Billion Rows of Time Series Data
Brandon Harris, Cloud + Analytics, Discover Financial

In this post I show you how to synthesize billions of rows of true time series data with an autoregressive component, and then explore it with ClickHouse, a big data scale OLAP RDBMS, all on AWS.

Python and Parquet performance optimization using Pandas, PySpark, PyArrow, Dask, fastparquet and AWS S3
Russell Jurney, Principal Consultant, Data Syndrome

This post outlines how to use all common Python libraries to read and write Parquet format while taking advantage of columnar storage, columnar compression and data partitioning.

ClickHouse Capacity Estimation Framework
Oxana Kharitonova, SRE, Cloudflare

Our current insertion rate is about 90M rows per second.

How to Build a Production Grade Workflow with SQL Modelling
Michelle Ark, Senior Data Engineer, Shopify

Currently, we have a warehouse consisting of over 100 models, and this validation step takes about two minutes.

The State of Open-Source Data Integration and ETL
John Lafleur, Co-Founder, Airbyte

Is an open-source (OSS) approach is more relevant than a commercial software approach in addressing the data integration problem.

Data Engineering Confessions #6: Bence Faludi

2020-12-23T00:00:00+00:00

In this interview series we ask data engineers how they ended up choosing this career and how they see the future for it.

You can subscribe to the show on Youtube, Spotify and Apple Podcasts to get notified about new episodes.

Data Engineering Confessions #5: Leandro Loi

2020-12-16T00:00:00+00:00

In this interview series we ask data engineers how they ended up choosing this career and how they see the future for it.

You can subscribe to the show on Youtube, Spotify and Apple Podcasts to get notified about new episodes.

Become a Data Engineer on a Shoestring (aka The Best Free Courses and Learning Resources)

2020-12-15T00:00:00+00:00

I was tinkering with the idea of finding the right way to help others to identify the resources that give you bang for the buck when it comes to upskilling yourself in data engineering… and this is what I came up with. So this is how to spend the remainder of your learning budget before the year ends.

Alan Kay (Squeak, Smalltalk, OOP, GUI) suggested that programming is pop culture, because it spreads much, much faster than mentorship, education or formal study does. I think the best example of this are the talks of James Mickens, they can help shine a light on why people think that blockchain or machine learning are the solution.

Treat this as a restaurant menu. No one should consume it end-to-end. Scan it and pick something that looks good. Try it. Rinse and repeat! Wishing you a great 2021!

tl;dr

A) I want to pass the interview for ...
Learn Spark or Kafka or whatever people do these days at a Coursera Specialization or a Udacity Nanodegree — I did them, I did not like them. Expect to burn about a hundred bucks.

B) I want a band aid aka branded lock-in trainings!
Cloud providers, and OS-as-a-marketing-tool companies like Cloudera or Databricks are there to help you! Either for free or for a thousand dollars.

C) Gamify my learning!
For browser-based self-paced bytesize Sudoku, try learning Python and SQL in a command line at DataQuest or at DataCamp — the latter one is also available on mobile.

D) I have to be able to bluff my way into mastering data engineering tools by tomorrow!
Udemy is the 'you get what you pay for' eBay kleinanzeigen for your needs. 10 bucks will cover most courses when they are on sale == that's basically always.

Learn SQL

It is back.

The single most important technology that anybody who has 'data' in her title must master.

Select Star SQL (free, 3 hours)
An interactive book for learning SQL with SQLite running in the browser - no setup required, the best intro for a beginner.

The SQL Murder Mystery (free)
A good way to practice SQL skills after the tutorial, works in the browser.

SQL Police Department ($20)
If you got a taste for story-based learning, continue here.

Learn Programming

The uncrowned coding intro is definitely the Harvard edX CS50 (free or €167 for a Verified Certificate) or you can learn at Dr. Chuck's Python for Everybody at Coursera (you can do it in a focused month for €41).

If you have previous programming exposure you could spend 4 days on honing your Python skills with Practical Python Programming by David Beazley. I totally recommend his presentations/workshops Generator Tricks For Systems Programmers and A Curious Course on Coroutines and Concurrency. I was about to embed his Discovering Python keynote, but I'd better just point to that it's one of my favourite keynotes ever.

Watch screencasts

I didn’t understand the format until Torsten showed me Destroy All Software. It's not for everyone, it's not for everything, but now I understand its place in the universe. And yes, you can even learn computer science with it. Computer science is actually fun, too bad you don't get to do it much in the everyday crunch — although it pays well if you know its most crucial concepts.

Destroy All Software ($29/month)
Classic is mostly Ruby, computation is Python. Can't miss the Unix and Git ones.

execute program (16 free lessons then $19/month)
Hit the SQL and the Regex.

Computer Science Essentials, (free samples, $79)
You need Database Normalization, Make, Reading Shell Scripts, Shell Script Basics. Has a book version, see below at The Imposter's Handbook.

The Missing Semester of Your CS Education (free)
Check Shell Tools and Scripting, Data Wrangling, Command-line Environment, Version Control (Git), Debugging and Profiling.

Top 3 books I recommend

Jeroen Janssens: Data Science at the Command Line, 2e (O’Reilly)
Not just data science at the command line. The new one with make will take you very, very far.

Bill Karwin: SQL Antipatterns - Avoiding the Pitfalls of Database Programming (Pragmatic Bookshelf)
When I got to the third chapter describing situations that I encountered personally in my career I was 100% sold. Also for all who think database programming is not a thing. Hold my beer for a sec.

Greg Wilson: Data Crunching, 2005 (Pragmatic Bookshelf)
My first book on data. No frills, talks about file formats, has code examples, hits regex. Every day you will do something that is covered in this book.

More books for the data engineer

The Imposter's Handbook (each book $30)
You need the first book: Database Normalization, Make, Reading Shell Scripts, Shell Script Basics. The second one is a good-to-have. See Computer Science Essentials above for the video version.

A Curious Moon ($30)
Beautiful ebook with SQL exercises, also intro to Postgres with real NASA data.

Markus Winand: SQL Performance Explained (book for money, free web version)
A multiformat, practical handbook on advanced SQL and running databases in general.

A free Allen B. Downey: Think Python 2nd Edition and a $6.99 Ian Miell: Learn Bash the Hard Way ebook wouldn't hurt to have either.

Books for the connoisseur

Julia Evans: programming zines
Important topics are covered, and it will show how you can talk about concepts without fuzz. I like them all. Check the black and white free ones.

Philipp K. Janert: Data Analysis with Open Source Tools (O’Reilly)
High-end analysis practice, more useful than most ML/AI. Very well written. It raised the bar for me in technical writing.

Angus Croll: If Hemingway Wrote JavaScript
Only if you're into languages and the thinking structures they impose, but then you will start gifting it.

Books for your Zoom background

Martin Kleppmann: Designing Data-Intensive Applications (O’Reilly)
I wish this would be our problem, not pop culture. Great and detailed, limited relevance from a practical point of view. Very academic, you can watch ‘em videos.

Ralph Kimball and Margy Ross: The Data Warehouse Toolkit 3rd Edition
I mean it's too long, it's too slow, it belongs to a previous era, where computing has been immensely less performant, still if you want to build mental models of how data describes different industries, a good old printed SAP.

Python Cookbook 3rd Edition
One has to have a reference book on Python. You know, no Internet, no Stack Overflow.

Thanks to Torsten Becker, Péter Fábián, Balázs Dénes and Steffen Kiedel for discussions over lunches in Berlin.

Support gastronomy during lockdown, order food from your valued vendors.

One more thing:
Check out the second edition of this series with even more recommended learning experiences for aspiring data engineers!

Data Engineering Confessions #4: Zoltán C. Tóth

2020-12-09T00:00:00+00:00

In this interview series we ask data engineers how they ended up choosing this career and how they see the future for it.

You can subscribe to the show on Youtube, Spotify and Apple Podcasts to get notified about new episodes.

Data Engineering Confessions #3: Sejal Vaidya

2020-12-03T00:00:00+00:00

In this interview series we ask data engineers how they ended up choosing this career and how they see the future for it.

You can subscribe to the show on Youtube, Spotify and Apple Podcasts to get notified about new episodes.

A data prediction for 2021

2020-11-26T00:00:00+00:00

Yes, it's that time of the year again: in the coming weeks ultra-scientific research institutes, cutting-edge management and innovation consultancies and basically everyone with a podcast are going to share their predictions about the most important technology trends for 2021. In a parallel universe called the real world, businesses have to make their own predictions for next year as well (for revenues, staffing, organizational changes etc.), they just prefer to call it planning.

Thinking about scenarios for what's ahead carries tremendous value, this has been known for long in the military even before Dwight D. Eisenhower famously said:

“Plans are worthless, but planning is everything”

It's about preparing for what is likely to happen and somehow for the unknowns that might impact our vague idea of how the future will unfold. The value I've mentioned is in part generated by the fact that anticipating certain future events today and acting in a controlled way so that these events result in positive (or not-negative) outcomes make a business more likely to survive the stormy seas of an economy shaken by the pandemic. But there is also value in the process itself, as it's an exercise in preparation.

So while these lists are being released in the upcoming weeks, the name of the game for the readers is finding patterns that help uncover the challenges on the radar and navigating through the various agendas inherently built into them. By doing this, one might arrive at something that's fairly likely to end up a real trend in the near future, not just an empty buzzword for the bin.

My obsession with predictions in general is a result of trying to identify the right sources to listen to. I am sure it's not just me constantly on the lookout for trustworthy and reliable content that helps me put my perception of the future into context. And I am also fairly confident that a lot of you, dear readers, are thinking about finding the right direction not just for your businesses, but for yourselves professionally and figure out where your career path is supposed to take you.

The Only Top 10 Game-Changing Sustainable Tech Future Innovation Agile Disruption Strategy Trend Report Workbook Executive Summary CEO Guide 2021 You Need - Download PDF here

Gartner is one of the first firms releasing their Top Strategic Technology Trends for 2021. The list of topics they've considered meaningful enough to make it to the list doesn't include blockchain! Wow, we're off to a great start. Sometimes I try to imagine how the season just before Christmas must be really difficult for blockchain enthusiasts and crypto fanboys. Every year they have to sit down and come up with new excuses why the past year was not the breakthrough year for their beloved technology and how the next one is going to be all different with mass-adoption, with unicorn valuations and maybe even with a valid usecase... but I digress.

Back to Gartner: "This year’s trends fall under three themes: People centricity, location independence and resilient delivery." - they say. They point out the following nine trends:

Internet of Behaviors
Total experience
Privacy-enhancing computation
Distributed cloud
Anywhere operations
Cybersecurity mesh
Intelligent composable business
AI engineering
Hyperautomation

For a CEO or COO who would like to enable their workforce to act upon these proclaimed tendencies, this would translate into: process digitalisation, improving access to data, data management processes and preparing for continued organisational readjustments. This is a pretty clear confirmation of the notion that COVID is merely accelerating existing momentums like remote working, data-led decision making and automation in business. What struck me when reading through the report was that data is very much in the center of all emerging trends coming our way, even more dominantly than at first glance. Duh.

But there is an observable shift toward data infrastructure-related investments that enable positive outcomes for the business as a whole instead of just doing machine learning l’art pour l’art to fuel a lighthouse project for AI or to "build a data science muscle".

This means that the nature of innovation is changing by moving away from the shiny and sexy, therefore the allocation of capital has to shift in parallel as a result. So here is a very real question that a lot of engineering managers and data team leaders ask themselves: what kind of competences does my team need in order to be able to deliver on the strategic expectations laid out the business plans for 2021?

Skilling up for 2021

"Knowing how to invest today means knowing what talent is needed."

Source: JF Gagne (CEO and founder of Element AI)

What I am seeing is one more pattern that corroborates the growing demand for data engineering talent, and confirms that there is no clear end in sight for it to drop. If the trends listed above turn into tangible initiatives within your company, and you have to start laying out a staffing plan for 2021, there is a very good chance you'll find the need for someone who understands tech infrastructure, the special nature of data and can deal with the increased amount of alignment meetings.

Maslow's argument remains true when it comes to my own biased POV,

"... it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail."

But as I always say, don't believe me, believe the numbers: below just two extracts of the latest researches showing the enormous growth of demand for data engineering skills:

Source: Dice Tech Job Report Q1 2020

Source: opendatascience.com

I expect more trend reports to come out in the next weeks underlining the need for data professionals in the context of the new normal (WFH, experiences and events going digital etc.) either directly or indirectly. It's a great sign for everyone who has already embarked on the journey of transitioning their careers into the world of data, and one more reason for rookies to focus more on engineering skills.

The data suggests that data engineering is here to stay, and having observed how data science has evolved as a profession, and especially how first-movers gained an almost unfair advantage compared to whoever tries to get into the game today... it is safe to say that the right time to start data engineering is yesterday. That's my humble prediction.

Data Engineering Confessions #2: Tamás Németh

2020-11-25T00:00:00+00:00

In this interview series we ask data engineers how they ended up choosing this career and how they see the future for it.

You can subscribe to the show on Youtube, Spotify and Apple Podcasts to get notified about new episodes.

Data Engineering Confessions #1: Ramzi Al-Aruri

2020-11-18T00:00:00+00:00

In this interview series we ask data engineers how they ended up choosing this career and how they see the future for it.

You can subscribe to the show on Youtube, Spotify and Apple Podcasts to get notified about new episodes.

The Data Janitor Letters - October 2020

2020-11-06T00:00:00+00:00

Data engineering salon. News and interesting reads about the world of data.

Testing SQL for BigQuery
Barbara Scherlein, Scala Backend and Data Engineer, Soundcloud

When I finally deleted the old Spark code, it was a net delete of almost 1,700 lines of code; the resulting two SQL queries have, respectively, 155 and 81 lines of SQL code; and the new tests have about 1,231 lines of Python code.

Why Percona wants your database to be open source, and not everyone is happy about it
Matt Asay, Head of Open Source Strategy and Marketing, AWS

Percona runs open source databases as managed services, which makes the company popular with customers but less so with competitors.

ETL Batch Processing With Kafka?
Tomasz Kaszuba, Java Big Data Engineer, Swiss Re

For small batch loads using traditional ETL tools is less complicated and much simpler to implement. But if the ETL pipeline needs to handle large amounts of data and scale Kafka wins hands down.

Meet whale! 🐳 The stupidly simple data discovery tool.
Robert Yi, Co-founder & Chief Data Officer, Dataframe

A Python library that scrapes metadata and formats it as markdown. A Rust CLI interface to search over that data.

A Short Story About SQL’s Biggest Rival
Cedric Chin, Content Marketing, Holistics Software

We might have once lived in a world where QUEL and SQL would have continued to duke it out, and where the ‘best’ language might have found its own niches.

Using CTEs to do a binary search of large tables with non-indexed correlated data
David Christensen, Senior Software and Database Engineer, End Point

The initial query went from timing out in the webservice in question to returning results in a fraction of a second with basic binary search.

New BigQuery Integration for GA4 Properties
Charles Farina, Head of Innovation, Adswerve, Inc.

Free BigQuery export from GA to all customers.

How to set up a multi-touch attribution model
Cyprien Marcos, Business Intelligence Manager, Project A Ventures

We show you how you can easily set up a multi-touch attribution model to track website conversions with Google Analytics, Google Tag Manager and a Jupyter notebook.

How to apply to the bootcamp

2020-10-17T00:00:00+00:00

I firmly believe that within the legal and moral framework of our society, education is by far the most potent transformational force that enables socioeconomic mobility. Therefore, we're working hard to make our bootcamp easily accessible for everyone, and to help our participants unlock professional and economic opportunities though education. In other words, we teach you something that has the potential to take you to the next level in your professional career.

In an ideal world, Pipeline Academy would have a suitable offering for everyone interested in data engineering regardless of geography, existing skills, preferred learning method etc., but we're not there yet. But how do you measure access to a school, and how do you communicate that?

Un/Restricted access

If you take a look at the key performance indicators that educational institutions like to use to showcase their superiority, you might stumble upon some organisations priding themselves in how low their acceptance rates are (e.g. "only 4.7% of applicants are accepted to Harvard"). When high demand faces scarcity, without intervention the result will inevitably be an increased price. Approaching it from an economic point of view, it only makes sense for a business to capitalise on the reputation it built for itself through years of impactful outcomes (successful graduates), and turn its offering into a product of exclusivity. But let's just take a step back:

Is our society across the board benefitting from making the highest quality education a luxury good?
Is building fences around knowledge not counterproductive to our shared welfare and reducing our future opportunities for the sake of short-term profits?
Is this trend not solidifying socioeconomic inequality?

My point: you're never going to see us pride ourselves in excluding people. A low acceptance rate at a bootcamp should only be acceptable if there is no way of scaling the number of seats without sacrificing the quality of the service. Otherwise it is a likely indicator of people trying to cash in, instead of trying to help others. Preselection is absolutely necessary, but it seems like we've forgotten why we're doing it, and today it only serves schools instead of students.

How to apply

We've designed our application process with one principle in mind: how can we make sure that we include everyone who is seeking opportunity and willing to put in the work. The steps don't include anything unexpected, it's more about how we deal with our applicants:

We're doing our best to stay easily approachable so the folks who are interested in joining Pipeline Academy can talk to us, ask questions and get a good feeling for what to expect. You can schedule a call right here anytime even without submitting an application.
Communicating the prerequisites of our bootcamp is supposed to help you figure out what you need to do in order to get admitted, not to discourage you. We would like to be very clear on what is expected from you before and during the program, and make sure you get some initial guidance before you get started.
After filling out the application form on our website you'll receive an email as a response (potentially with some answers to your questions you've shared via the form) that includes a link so you can select a slot in our calendar for a call that usually takes about 20-30 minutes.
During the admissions call we like to learn about your motivation and attitude besides your professional experience: our goal is to understand why and how you can make this happen instead of comparing you to other candidates. Usually we ask you about the organisations and the teams you've worked with in the past, the technologies you are familiar with and some projects you were involved in. We don't do any coding tests (yet) though. We get back to you within a couple of days after the call with the decision about your admission.
We will always encourage everyone to learn more, even if this means that you're not going to be spending money with us. We are happy to point you towards online programs or other alternatives to our bootcamp if we can't admit you for some reason. Again, the goal is to help you on your path of becoming a data engineer. Period.
Our school and your peers remain accessible for you even after the bootcamp. You'll get invited to our events if you want to, and we help you with your job search even well after the training.

That's pretty much it. There is no magic, no secret test, no tricky coding challenge to filter you out. Nevertheless, the expectations are high, and we promise that your time at Pipeline Academy won't be boring.

Other actions we proudly take to ensure everybody has a chance to get closer to the craft of data engineering and our coding bootcamp:

Communicating our approach and actions in a very transparent manner,
Design our curriculum to be accessible for applicants with different levels of experience and skills,
Guiding rookies just as experienced professionals who approach us with questions,
Supporting the data engineering community actively with events and professional input in various formats,
Offering multiple financing options and in some cases even scholarships,
…

If you consider applying to Pipeline Academy, we encourage you to do your research (the blog is a great starting point) and check if data engineering is a career path you'd like to embark upon. If you feel motivated but have some doubts, just talk to us to see if there is a way we can support you on your journey.

The Data Janitor Letters - September 2020

2020-10-13T00:00:00+00:00

Data engineering salon. News and interesting reads about the world of data.

Data Cleaning IS Analysis, Not Grunt Work
Randy Au, Quantitative UX Researcher, Google

Also, most data cleaning articles suck.

Importing account statements and building a data warehouse
Kristian Köhntopp, Senior Scalability Engineer, Booking.com

I was experimenting with importing the account statements from my German Sparkasse, which at that time were being made available as a CSV.

Our Online Analytical Processing Journey with ClickHouse on Kubernetes
Sudeep Kumar, Member of Technical Staff - 2, ebay

With our new, cross-region aware OLAP pipeline, we reduced our overall infrastructure footprint by over 90 percent.

Dawn of DataOps: Can We Build a 100% Serverless ETL Following CI/CD Principles?
Luis Velasco, Cloud Data Analytics Specialist, Google

Is it time to enjoy the benefits of DevOps in the informational space?

Publish events, not logs
Kislay Verma, Software Engineer, Cure.Fit

I believe that logging as understood commonly is an ad hoc activity, and cannot handle the unknown-unknowns of a production system. We need to switch to an event perspective to leverage it more effectively for system design and reliability.

Be Vigilant about Time Order in Event-Based Data Processing
Mingwei Li, Senior Software Engineer, Expedia Group Technology

How to handle the timing of events.

Data Engineer Job Description Boilerplates

2020-10-12T00:00:00+00:00

Are you an HR-professional recruiting new members for your organisation’s data team? If you are looking for talent to fill a data engineer role, you have probably faced the challenge of writing a proper job description. Your CTO/Director of Engineering/Head of Data is very busy, and gave you only some high-level points, but nothing that fits into a JD. Don't worry, you are not alone.

Below you can find some boilerplate wording for the 'Tasks & Responsibilities' section of a data engineer job description. Adjust, adapt and complete it the way you see fit, the point is really to make you think less about the wording and more about finding the right person for the job.

One hint: be selective in what you write into the job ad, more does not equal better in this case.

Data engineer job description sample text

Work closely with the product team on the platform
Develop and scale the recommendation engine and comparable algorithms
Interface with the infrastructure team
Scale the architecture to manage increased traffic
Work on the backend structure, the API and partner integrations
Start integrating the recommendation engine
Measure and ensure data quality across the organisation
Use agile software development processes to deliver features
Build and document solid data pipelines
Clean, transform, and aggregate data from different sources
Develop and document processes for data mining, data modeling and data warehousing
Support building complex algorithms that provide business value to the customers
Ingest and aggregate data from both internal and external data sources
Help streamline the data science processes
Plan data models and architecture
Implement customer lifecycle and retention models based on an existing methodology
Employ an array of coding languages and tools to set up the data infrastructure
Interface with the reporting team and support their objectives with infrastructure tweaks
Execute/implement new data product features end-to-end
Engage with the machine learning team and uncover hidden efficiencies in the pipelines
Turn high-volume activity data into a highly accessible resource
Ingest and aggregate data from both internal and external data sources
Transform the available data into meaningful insights working with the analyst team
Develop simple models and integrate them with the visualisation tools
Work closely with the data science and business intelligence teams to develop data models and pipelines for research, reporting, and machine learning
Integrate state-of-the-art data management and software engineering technologies
Tap into new data streams from third-party APIs
Create custom software components for the data platform
Collaborate with the stakeholders in an interdisciplinary team
Research new approaches for making data accessible to customers
Connect legacy and new data systems together
Define basic tooling, metrics and other solutions and maintain quality to the highest standards
Write, extend and debug microservices for planned features
Collaborate with other engineers, data analysts, and product managers
Lead data strategy and inform the product strategy team on how world-class data experiences are built
Take leadership opportunities and shape the data culture within the organisation
Build near real-time and batch data processing pipelines
Design and optimise low latency systems
Build highly reliable but flexible service infrastructure
Maintain tooling and enable algorithms to move into production faster
Relate and match entities from different data streams
Increase the implementation speed of data tools
Creating secure processes to keep the data pipelines safe
Improve data models and foster data-driven decision making
Draw a comprehensive picture of user flows and enable deeper analysis
Specify data requirements and pre-processing routines
Hold companywide data trainings
Develop solutions for automatic labeling of data
Model front end and back end data sources
Design and optimise complex queries and deliver user value
Work closely with data scientists on modeling
Grow the data competency across the company

Presenting and Teaching via Zoom 101

2020-10-03T00:00:00+00:00

Yes, we all miss the full bandwidth of reality, but let's face it, we're left with experiencing the world via videoconferencing tools for now.

You can find dozens of videos answering specific questions related to the setup, the tools and the nitty-gritty, here I summarize what I figured -- standing on the shoulders of giants, as always. (You don't want to know how many different items and setups I tried before settling on the bare minimum.)

Rule #1: no wireless.

Wireless technology is on the brinks on dealing with this device density given the bandwidth available. If you want a reliable setup, use cables. Most importantly use an Ethernet cable to connect your computer to the router. This is the 80%.

Presenting

This is the more focused, defined format so figure the following:

camera distance: you should fit in the screen, almost fill the frame, but being able to use all your gestures,
lens height: get the lens at eye level, your nostrils are not that fun,
light: have a window near with natural light, preferably on the side, for extra fun you can add a 5600K bulb in a lamp, that bounces its light on the wall or coming at your face at an angle,
background: should be something neutral, a bookshelf is great, and Zoom's virtual background with a green screen actually runs on a 5 year old computer,
sound: generic Apple Earpods will just do fine, but getting an entry level lavalier mic would not hurt,
camera: the built-in Mac camera is okay. If you talk to a monitor, don't go further than the Logitech C920,
a presenter — hard to get a wired one.

This is the backstage. In this specific setup I used an old Joby Gorillapod to fix the Neewer LED lamp with 5600K.

This is what I looked like presenting online at Budapest Data Forum 2020.

Teaching

Different styles and methods ask for different setups, still I believe this a minimum viable approach:

real estate: more of it! You will need an extra big monitor to be able to follow the interaction, the chat, the faces, your slides and your terminal,
drawing: an 6th generation iPad with a first generation Apple Pencil as a separate Zoom user will do the trick if you want to draw on your slides or just simply use it as a whiteboard,
non-technical items: I let you figure the ones I found useful for teaching — just by looking at the photo :)

This is what I looked like live teaching via Zoom during the Summer Camp.

Postscript

Choosing the software we have to use is not always on us. Quality-wise Facetime, Bluejeans and Zoom are the descending order in our experience. Depending on your computing power you might want to give a chance to Snap Camera, especially Meeting Gestures by Cameron Hunter, or if you have a power station give a chance to mmhmm.

Data Engineering Podcast With The Data Janitor

2020-09-23T00:00:00+00:00

Tobias Macey, the Joe Rogan of the data engineering scene interviewed Daniel in the Data Engineering Podcast. Give it a listen right here!

Tobias’ summary of the insightful conversation in episode #151:

“Data engineering is a constantly growing and evolving discipline. There are always new tools, systems, and design patterns to learn, which leads to a great deal of confusion for newcomers. Daniel Molnar has dedicated his time to helping data professionals get back to basics through presentations at conferences and meetups, and with his most recent endeavor of building the Pipeline Data Engineering Academy. In this episode he shares advice on how to cut through the noise, which principles are foundational to building a successful career as a data engineer, and his approach to educating the next generation of data practitioners. This was a useful conversation for anyone working with data who has found themselves spending too much time chasing the latest trends and wishes to develop a more focused approach to their work.”

About the host and the show (taken from the website of the podcast):

Like and subscribe on Apple Podcasts and give the show a follow on twitter!

Data engineer salaries in Germany 2020

2020-09-22T00:00:00+00:00

TL;DR: the expected annual gross salary for a junior or mid-level data engineer in Germany is between €45.000-75.000 with a steep increase after a couple of years in the field.

Below a rundown done in August 2020 based on secondary sources from across the internet. This has been verified by our primary research with stakeholders of the Berlin data ecosystem. (Note: if you don’t like the inconsistent structure of the below spreadsheet, you are not alone.)

Data Engineer Salary Germany 2020

Source	Position	Salary
datadrivencompany.de	Junior Data Engineer	€40.000-60.000
	Mid-level Data Engineer	€50.000-90.000
	Senior Data Engineer	€80.000-120.000
orange-quarter.com	Junior Data Engineer	€40.000-50.000
	Mid-level Data Engineer	€50.000-65.000
	Senior Data Engineer	€65.000-90.000
datasolut.com	Junior Data Engineer	€50.000
datasolut.com	Mid-level Data Engineer	€70.000
hays.de	Data Engineer (level of seniority unspecified)	€45.000-70.000
neuvoo.de	Data Engineer (level of seniority unspecified)	€70.750
glassdoor.de	Data Engineer (level of seniority unspecified)	€60.000
stepstone.de	Data Engineer (level of seniority unspecified)	€54.600
gehalt.de	Data Engineer (level of seniority unspecified)	€62.777

Click the links in the Source column to see the the referenced website. Please note, the above information is subject to change.

Things you should know when looking at these figures:

Expected salaries are not advertised in job listings in Germany, and most companies are highly secretive about this question which adds a lot of inefficiencies to the job search.
The listed salaries are before taxes, which are determined by various factors. Here's one of the websites helping you calculate what your net salary expectations should look like.
There are considerable regional differences in cost of living within Germany, make sure to adjust the above figures accordingly. Also, you'll find that there is a major mismatch between what a data engineer makes in Silicon Valley compared to Europe in general: the job markets, tax structures and the cost of living are just so different, that a comparison hardly makes sense.
There is an upwards trend in salary average and the number of open positions for engineering roles. It should also be noted, that salaries increase with a higher level of seniority much faster than for most other positions, and that the starting salaries for more commoditized engineering positions (e.g. frontend developer) are usually significantly lower than for data engineering roles.

Some considerations for your job search:

Experience with some specific tools or programming language, background in a certain industry, prior experience in coding or even with managing people have a strong positive impact on salary expectations. Use that when negotiating.
The scope of the data engineer's role is broad, which is one of the reasons why the position often runs under a different name: machine learning engineer, BI engineer, backend engineer data platforms, data lead etc., furthermore some companies are looking for engineers for using specific tools: Spark engineer, Hadoop engineer etc.
The career path is also not straightforward in terms of naming conventions for more senior positions, so senior data engineer, data architect, big data architect, head of data etc. are also keywords one should check.
Do your research: check out local job platforms, job metasearch engines, glassdoor and kununu for company reviews (these platforms can save you from some employer-induced PTSD), and make sure to also read the latest news about the organisation you are considering joining.
Salary is not everything: when interviewing, ask about team structure and who you are supposed to report to, career paths, stack and learning opportunities. Having a strong mentor can bring you more value on the long run than a few more euros.
Keep on learning, but even more importantly: keep on working. Create a hands-on portfolio that stands out, experiment with tools and contribute to open source projects to show your skills.

Data engineering is an accessible career path that involves competencies that most career advisors and recruiters would consider essential for the upcoming decades. Regardless if you are a generalist or a specialist, working in tech or any other sector, understanding the flow of data and how to manage it makes any employee into an asset.

The Data Janitor Letters - August 2020

2020-09-16T00:00:00+00:00

Data engineering salon. News and interesting reads about the world of data.

The startup data stack starter pack (2020)
Lewis Hemens, CTO, Dataform

I advise a lot of people on how to build out their data stack, from tiny startups to enterprise companies that are moving to the cloud or from legacy solutions. There are many choices out there, and navigating them all can be tricky.

Why is dbt so important?
Stephen Whitworth, Senior Software Engineer, Monzo Bank

If you choose not to use dbt, you’ll probably waste time building a less-fully featured, buggy implementation of it yourself. Give it a serious look.

Guiding principles for a data engineering team
Rahul Jain, Principal Engineering Manager, Data engineering and BI platform, Omio/GoEuro

We prefer boring but battle tested technologies over tech-fetishism.

ClickHouse & Redshift Face Off in NYC Taxi Rides Benchmark
Alexander Zaitsev, Co-founder, Altinity

2020 versions of both ClickHouse and Redshift show much better performance. However, open source ClickHouse continues to outperform Redshift on similarly sized hardware, and the difference increases as the query complexity grows.

Unpopular Opinion - Data Scientists Should Be More End-to-End
Eugene Yan, Applied Scientist, Amazon

Going out of the regular DS & ML job scope helped with delivering more value, faster.

Dear Google Cloud: Your Deprecation Policy is Killing You
Steve Yegge, Head Dude, Ghost Track

Backwards compatibility keeps systems alive and relevant for decades.

Get rid of AI Saviorism
Shreya Shankar, Machine Learning Engineer, Viaduct

Machine learning is a tool, not a panacea. If our tools don’t immediately work for them, it’s not their fault.

The state of coding bootcamps in 2020

2020-09-03T00:00:00+00:00

My role at Pipeline Academy entails a variety of things I really love to do, one of them is setting up a strategy for the company and translating it into actionable operational steps.

It's pretty much like using any map:

Understanding the industry/market that my company is active in by analysing where we came from and where we are right now.
Define a desirable future state or an idea of the value you generate (use your preferred framework).
After that comes the fun part, finding the path that gets us there.

So here is my step #1, let's see how the world of coding bootcamps looks like from different perspectives (industry analyst, bootcamp organiser, participant) based on some of the key resources I've encountered during my never-ending voyage on the internet. If you'd like to get closer to understanding this fairly new educational format and you are looking for a non-comprehensive starting point, you might find the links below useful.

Field guide

One of the most important write-ups about coding bootcamps has been done by Quincy Larson, the teacher who founded freeCodeCamp.org: this very no-nonsense document called The Coding Bootcamp Handbook is a reality check and golden piece of advice for anyone who just recently fell in love with the idea of going to a bootcamp and magically upgrade their lives 'get-rich-quick-scheme'-style. The sobering reality is:

“Going to a bootcamp can be the best decision you ever made. Or it can be an awkward financial setback. Do your research. Save up your money. Learn coding fundamentals first. Bootcamps aren't magic. They aren't going to do the work for you. In the end, the experience is what you make of it. So make the most of it.”

The DOs and DON'Ts of this universe have been shared in a four-part story by Marco Morawec, Co-Founder of the Firehose Project that has meanwhile been acquired by Trilogy education (that has meanwhile been acquired by 2U Inc.). It presents solid evidence-based guidance for the makers of bootcamps and for aspiring participants as well, and makes you cut the bullshit and focus on what brings you forward:

“Quality and depth of the curriculum
Emphasis on the coding fundamentals – algorithms and data structures
A bootcamp team that is technical and 100% focused on student outcomes, not marketing metrics
Everything else a coding bootcamp focuses on is useless”

Here is the rub: whether you are planning to organise a coding bootcamp or you are considering participating in one, make sure you keep it real. While there are way too many buzzwords and promises floating around, what you put in is what you'll get out of it.

The coding bootcamp market

Most of the public information about the state of the market is based on data from the US and Canada. While the formal education system in Europe is different in many aspects, and it seems like there is very little structured and comparative data available on the current size and quality of the industry, the European countries face similar challenges in filling the competence-gap on the highly competitive and fast-paced job market. Therefore, any implications and assumptions need to be treated with caution.

The media loves to reference the latest market analysis by CourseReport.com, since this coding bootcamp review platform offers its readers the most basic, and at the same time the most crucial key performance indicators in a simple format. In the comprehensive Coding Bootcamps in 2020: Your Complete Guide To The World of Bootcamps report you get access to a lot of details about their research and the most important findings, that they've summarised like this (note: the data is self-reported by the participating schools):

“Coding bootcamps are intensive, accelerated learning programs that teach beginners digital skills like Full-Stack Web Development, Data Science, Digital Marketing, UX/UI Design, Cybersecurity, and Technical Sales.
The average bootcamp costs ~$13,500, and graduates report an average starting salary of $67,000. Bootcamps can vary in length from 6 to 28 weeks, although the average bootcamp is ~14 weeks long.
Coding bootcamps teach modern, widely used programming languages and frameworks like Ruby on Rails, Python on Django, JavaScript, and PHP stacks through project-based learning. Students graduate from bootcamps with a portfolio, an online presence, interview skills and more. Most bootcamps help graduates find an internship or match students with an employer network – in fact, in Course Report's most recent research, 83% of bootcamp alumni report being employed in programming jobs.
There are bootcamp campuses in over 85 cities throughout the US/Canada. Coding bootcamps are predicted to graduate 23,000 students and gross $309MM in tuition revenue in 2019.
The average bootcamper has 6 years of work experience, has at least a Bachelor's degree, and has never worked as a programmer. However, the number of students with degrees appears to be declining slightly over time.
A lot has changed since bootcamps launched in 2012: University Bootcamps are now competing with household bootcamp names, payment options like Income Share Agreements and Deferred Tuition have exploded in recent years, and many bootcamps are dipping into the corporate training market.”

CourseReport started releasing its Coding Bootcamp Market Size Study in 2017, and since then they have done it in 2018 and 2019 as well. In addition, a separate Coding Bootcamp Alumni Outcomes & Demographics Report has been issued 2018 and 2019 to share more transparency for (potential) students and manage expectations towards promised outcomes.

An even more academic approach can be found in the RTI Press publication Alternative and Independent: The Universe of Technology Related "Bootcamps" by Caren A. Arbeit, Alexander Bentz, Emily Forrest Cataldi and Herschel Sanders. This 2017 report does a great job in highlighting the reasons behind the critique that feels almost like an overused cliche in 2020:

“While the bootcamp model holds a great deal of promise, it is currently an unregulated, unaccredited sector of higher education with little transparency and no independent outcomes data.”

My favourite and most comprehensive market research has been released on CareerKarma: written by James Gallagher, the State of the Bootcamp Market Report 2020 is a highly professional and extensive analysis. It is an attempt to cover all the aspects that belong in a proper market exploration while trying to redefine some of the standards of measuring the dynamics of this specific market (note: CIRR is one attempt to move towards a clear standard in reporting outcomes). He ends up with a significantly higher gross total market revenue for 2019 than CourseReport, which shows how intransparent the US market is - not to mention the rest of the world. In addition, the report includes a map of the ISA landscape, which will be subject to consolidation in the upcoming years similarly to the bootcamp business.

“If bootcamps continue to innovate and find success equipping people with tech industry-ready skills, the bootcamp sector will likely see significant growth over the next five years — both in the number of bootcamps and the number of students enrolling in them. There also may be a period of consolidation in the space, as existing players become more dominant and look to expand their influence. In addition, the growth of coding bootcamps and the increased diversity in their offerings suggests that the bootcamp model could have wider applications than solely within the context of programming instruction. As courses in digital marketing, technology sales, and other non-coding technical disciplines have become more common, it may be a signal that bootcamps are looking to expand outside of programming disciplines. This would allow the bootcamp model to be applied to a much larger market segment: post-secondary training in non-technical fields. Overall, bootcamps have proven themselves an effective pathway to a career in tech for thousands of people, and all signs indicate that the sector will continue to grow in coming years.”

Jay Wengrow, the CEO of a coding bootcamp called Actualize summarizes the state of the industry (as of 2018) based on his insightful experiences building up his company focusing on the challenges of the past years and the ones to come in the near future.

Organic growth is clearly visible, however not at the pace of software startups. Every time you encounter new data about the market or participants, you have to double-check the source and be cautious when basing decisions on them.

Business model and scaling

The latest analysis called Bootcamps and ISAs: Economics, Challenges, and Opportunities by Christoph Rindlisbacher looks at the coding bootcamp concept and business model from the perspective of a financial investor and comes to a conclusion that won't please VCs, but it ultimately shows a positive outcome, highlighting that the last decade was fairly turbulent:

“The structural obstacles to scaling a bootcamp identified by this report will make it difficult for any bootcamp to achieve venture scale returns. This does not mean that short-term education is not a worthwhile endeavor or a viable business idea. Affordable alternatives to four year degrees are good and needed, and it’s theoretically possible for bootcamps to turn a profit while providing high quality education. However, all the available information suggests it’s difficult for bootcamps to be both profitable and to provide high quality education while growing at a rapid pace. It’s difficult to imagine a scenario where a coding bootcamp is a good venture capital investment.”

One approach for scaling exercised by a lot of coding bootcamps is focusing on diversity and promoting (and often subsidising) female and minority participation in order to boost their ratio in the tech workforce. The events unfolding in 2020 have accelerated this notion radically.

According to StackOverflow's Developer Survey Results 2019, about 15.4% of all respondents have participated in a full-time developer program or training bootcamp. It should be noted, that bootcamps are not necessarily meant to be starting points for careers in tech, they represent an opportunity for reskilling and moving away from commoditised coding languages or unpopular fields.

Y Combinator intends to play a significant part in the future of non-traditional education, and has put their money where their mouth is attempting several promising tweaks to the business model... without any major success yet though. It seems like even their bootcamp called Modern Labor tried experimenting with connecting their educational business model with private sector projects through an ISA, but they had to realise that scaling education while keeping up the quality might be a more difficult task than anticipated. This leads us to our next point.

Seeing Silicon Valley's tendency to abuse their users in exchange for rapid company growth and in order to please their shareholders, the limits of scalability are both a challenge for coding bootcamp organisers and a helpful indicator for people searching for the right school. Reinventing the wheel did not work out for any school yet, but this does not mean that it won't in the future - you just need to make sure you are clear about the risks involved.

2020

I am not sure how this year is going to go down in history, but I am confident that we all will have some anecdotes to tell about it. Observing how the lockdown, the social distancing measures, the pandemic-caused rise in unemployment and the WFH trend are accelerating the overdue seismic shift in higher education, the way we approach education as a means of personal socioeconomic improvement changes significantly.

Sidenote: the popularity of MOOCs shot through the roof during lockdown, but a certain level of sobering is already visible on the horizon as most learners have realised that they are just not motivated and/or disciplined enough to complete the courses they've signed up for - the 2021 numbers will tell the full story.

Critique and scandals

There are several solid articles for everyone looking for some research-backed advice on the upsides and downsides of attending a bootcamp, and various much more personal pieces from former students who suffered serious physical and mental stress during attending an intense school.

While there were several small-to-medium scale scandals and closures around coding bootcamps in the last couple of years, they usually repeat the same mantra:

The bootcamp graduates are not fit to become software engineers at the highest-profile tech companies like Google,
Coding bootcamps use misleading marketing claims and post false outcome stats in terms of placement and salaries,
The quality of education participants receive is subpar and not worth the tuition fee (combine this with critique about ISAs as a financial instrument),
The bootcamps are intense and stressful, some are ill-equipped to keep up with the fast pace.

Now, there are real life examples to all of the above claims (and usually they come in a bundle), but it shall be noted that the same critique applies to for-profit higher ed in general. Universities and colleges (while partially being regulated) don't receive scrutiny for unfulfilled outcomes, but the related financial debt has become one of the key economical issues of todays generation in the US, and therefore a highly politicized topic.

One additional theme that comes back in a cyclical manner is related to VC investment in larger bootcamps that cannot deliver on their inflated market valuations. The latest large scandal being about Y Combinator backed Lambda school, and its founder Austen Allred. An other sizeable scandal that made it into mainstream media was a sexual misconduct story at DataCamp.

The unquestionable gap in the education market filled by bootcamps and its importance in the future has been noticed by Washington as well, and we'll be witnessing even more political involvement in the coming years mostly focusing on regulation of financing and quality control - or even tying the two together by connecting institutional compensation with positive outcomes.

Future outlook

The socioeconomic trends driven by digitalisation outside of Silicon Valley and the "traditional" tech sector will result in an even higher demand for skilled tech workers. This is going to drive more experimentation, but also more consolidation to the (coding) bootcamp market. Business models combining online and offline learning, individual and corporate training and various forms of financing opportunities will to come and go. And politics is going to get involved with the notion to ensure outcomes for participants, regulation and comparisons with formal educations are already on the horizon.

Differentiation and expansion into new non-tech areas should be expected, just as vertical expansion of the value chain turning educational institutions into end-to-end players on the tech (labour) market.

The future of the bootcamp business will be shaped by the general progress in edtech, and the growing, unsustainable socioeconomic bubble of US higher education, where multiple analysts expect major disruption in the upcoming decade. The likely new entrants to the educational sector are Apple, Google, Microsoft and Amazon: all of them have access to the capital and the tech competence to get involved in higher ed in a big way. Offering edtech tools and training, hardware to equip classrooms or enable learning from home, digital learning platforms combined with subscription models that end with a certification… you name it. Partnering with ivy league institutions who have raised their margins in an unprecedented manner in the last five decades will enable a more accessible college experience which however will be trimmed to deliver stakeholder value, not one that is aimed at serving society across the board. As for coding bootcamps, most of them don’t have the brand recognition that would justify an investment from the largest players, but the opportunity to partner up with medium-sized or local tech players is still huge.

Before we close this chapter, let's reflect on what the key learnings of the last decade really are. Here is some more operative advice for coding bootcamps by Jonathan Lau, the founder of SwitchUp.org, one of the two leading review site for coding bootcamps:

“DON’T SCALE UP AT THE COST OF QUALITY
REFLECT THE NEEDS OF LOCAL MARKETS
KEEP YOUR OFFERINGS CURRENT AND RELEVANT
COMPETE ON CREDIBILITY ABOVE ALL ELSE
EMBRACE FEEDBACK FROM THE MARKETPLACE”

Similarly, Jason Moss (President and Founder of the data science bootcamp Metis) has provided his POV on the last years working on turning his school to a success story:

“Embrace continuous transformation. Nothing is “right” for long.
Think digital first. Plan for scale from the get-go.
Fish in large ponds.
Don’t make assumptions lightly. Be clear about big decisions.
There is excellence in focus.
Invest in people and reputation. Good things will follow.”

Closer

To finish off, allow me go back to The Coding Bootcamp Handbook for a minute and point out a few words that are on my mind since the decision about founding Pipeline Academy, the first school for data engineering has been made:

“[...] new coding bootcamps have something to prove. Their teachers and staff will work like crazy to ensure the school succeeds. They'll try their hardest to train you. They'll help you get a good job so they can get a win under their belt and onto their testimonials page.”

Whatever the future vision will look like (and trust me, I already have a not-so-vague idea about it), this will be the guiding light when starting the journey. And this is what you should expect from us. Nothing less.

The Data Janitor Letters - July 2020

2020-08-13T00:00:00+00:00

Data engineering salon. News and interesting reads about the world of data.

Some SQL Tricks of an Application DBA
Haki Benita, Development Team Lead, PCENTRA

Databases are the backbone of most modern systems, so taking some time to understand how they work is a good investment for any developer!

Sessions for analysis, the eternal fiction
Randy Au, Quantitative UX Researcher, Google

There are a vast multitude of hypotheses and potential narratives we could attach to any session.

Our journey to a new data warehouse
Ana Gulevskaia, BI Analyst, Omio

How do the benefits of 3NF and SF fit in here?

1.1 Billion Taxi Rides using OmniSciDB and a MacBook Pro
Mark Litwintschik #BigData Consultant

The Q1 time is the fastest for any workstation benchmark I've done. To get this level of performance on a regular piece of office equipment is a big game changer.

Big Data Small GPU, No Problem
Rodrigo Aramburu, CEO, BlazingSQL

BlazingSQL is no longer limited by available GPU memory for query execution.

Evolution of the Modern Data Warehouse
Paige Roberts, Open Source Relations Manager, Vertica

A data warehouse is essentially a business-driven, enterprise-centric and technology-based solution.

How much can you trust your data?
Ellen König, Senior Data Engineer, ThoughtWorks

Data quality assessments are an effective, but often overlooked way to make your company’s data products more trustworthy.

Let's build a Full-Text Search engine
Artem Krylysov, Senior Software Engineer, Datadog

Despite its simplicity, it can be a solid foundation for more advanced projects.

How to make simple Geolocation service
Max Kostinevich

On Cloudflare Workers I've got almost x10 better performance in comparison to AWS.

Apache Arrow 1.0.0 Release

The 1.0.0 release indicates that the Arrow columnar format is declared stable, with forward and backward compatibility guarantees.

I'm not an expert. I have experience.

2020-08-07T00:00:00+00:00

I don't like to talk. I don't like to talk at conferences. I'm an introvert at heart. Still, sometimes one has to step out of the comfort zone and stand for things that deserve to be stood for. All my talks were born this way, some more technical, some more reflective on the practicalities of data life.

In 2016, at Berlin Buzzwords in Berlin I shared our journey of how we migrated a data stack from AWS to Azure. How we managed to do it with open sourcing tools as roadkills and what we’ve learnt about the barebone necessities ending up within the sole body of a Raspberry Pi may shock the absolute believers of distributed computing. An adventure in bash, make and SQL with a detour in Moore's law and falling memory prices. You can find the slides right here.

The Data Janitor Letters - June 2020

2020-07-20T00:00:00+00:00

Data engineering salon. News and interesting reads about the world of data.

I gave the business what they asked for and they never used it
Kenny Ning, Data Engineer, Better

An addition to the long list of unused ML projects: do the simplest solution first, modeling is never done, failure is okay.

Getting machine learning to production
Vicki Boykis, Machine Learning Engineer, automattiC

Deploying is hard. Deep learning is deceptively easy. Go for prebuilt as much as possible. Understand networking and scale. Iterate quickly.

AI – the no bullshit approach
Filip Piekniewski, Scientist, Accel Robotics

Common sense will emerge only when a connectionist like system will have a chance to develop the internal symbols to represent the relationships in physical world.

What I learned from looking at 200 machine learning tools
Chip Huyen, Engineer, Snorkel AI

If you have to choose between engineering and ML, choose engineering. It’s easier for great engineers to pick up ML knowledge, but it’s a lot harder for ML experts to become great engineers.

Most tech content is bullshit
Aleksandra Sikora, Software Engineer, Hasura

Realize that there's tons of misconception in the world. Adapt solutions to your particular use case. Your solutions are not any worse than the ones on the internet.

How we migrated our data warehouse from Redshift to BigQuery
Rahul Jain, Principal Engineering Manager, Data engineering and BI platform, Omio/GoEuro

A journey step-by-step.

In search of speed — debugging Elasticsearch performance
Martin Iotchev, Fullstack Software Engineer, Prezi

The underlying hardware plays a significant role in the performance of an Elasticsearch cluster. Provisioning larger data nodes will yield better performance as compared to the smaller default nodes currently used in production. Furthermore, a cluster with more shards will perform better on larger data sets.

Lightweight alternatives to Google Analytics
Ben Hoyt, Software Engineer, Compass

For site owners who just need basic traffic numbers, GoatCounter and Plausible both seem like excellent options. Those who like more visual polish and documentation might prefer Plausible; those who value a more developer-friendly tool with easy self-hosting will probably prefer GoatCounter.

This is not a test, this is a summer camp - part #2

2020-07-08T00:00:00+00:00

This is the second part of the story about setting up a live remote coding workshop in the midst of an economic downturn. Part one is about the circumstances that led to all this, you can read it here. Part two is about how we've rocked the workshop itself and about the feedback we've received after.

When I started reading the responses from the eight participants of the Data Engineering Summer Camp to the feedback questionnaire I've shared with them right after the workshop, the feeling this whole situation sparked in me reminded me of the effect named after the 1950 Jidaigeki crime movie Rashomon by Akira Kurosawa. Karl G. Heider used the term "The Rashomon Principle" to refer to the effect of the subjectivity of perception on recollection, by which observers of an event are able to produce substantially different but equally plausible accounts of it. Even though the ten of us (including Daniel and myself) were not external observers of the Summer Camp but active participants of the event, when putting these ten individual recollections next to each other the data becomes a physical manifestation of this particular gestalt.

My point: whoever is reading this should keep in mind that this summary is inherently biased and there is no one correct interpretation of the event in question. What I am looking for are shared patterns in the multitude of witness perspectives involved that indicate a direction for the path to our own improvement.

But then you see a tweet like this from one of our happy students and you instantly forget about the semi-scientific approach you were about to force upon yourself:

Source: Flávio Clésio's twitter (saved version), @soobrosa is Daniel's twitter handle.

First, let's go back to where we left off the last time.

The Data Engineering Summer Camp

Quick recap: between the 18-22nd May (Monday-Friday) we've held the first Summer Camp by Pipeline Academy with eight participants in the virtual classroom. The purpose was supporting people and businesses by teaching them a valuable competence through a hands-on project and learning about our own areas of improvement through the process.

The agenda was designed to cover the most important building blocks of setting up a simple data product: an introduction to data engineering, ETL basics, some SQL and deployment. Since we prefer actual building to just talking about building, the decision to focus on real implementation was quickly made. The below schedule reflects this attempt with 2.5 days of classroom experience aka 'campsite' (frontal teaching, live coding, group discussions, presentations and demos, Q&A) and 2.5 days of individual or team adventures (in duos) for implementing the data pipeline.

The week before the workshop the participants received homework. At first glance, the task was fairly simple and straightforward, but it included some challenges that could be solved in various ways, depending on what tools and methods a student prefers. Imagine you're asked to travel from point A to point B within a city: there are plenty of alternative ways of completing the task picking different routes, means of transportation, and the chosen way serves as a testament to the preferences and skillset of the traveler. We used this dry run to get a better understanding of what our cohort is comfortable with vs. what topics we need to put more emphasis on to achieve the desired outcome for the week.

The first morning was about getting familiar with each other, some chit-chat about our daily routines during corona and our different experiences with working with data. We've moved on to discussing what data engineering is and how the skills that make a data engineer relate to the participants' individual skillset. Here's an example: a data scientist usually has more advanced mathematical know-how than a frontend engineer, but the latter is likely to have seen or written more maintainable code (especially collaboratively in larger teams). The afternoon was spent with an overview of ETL procedures (extract, transform, load).

The first day was very exciting but exhausting as well. I was relieved that we've managed to capture the attention of the participants as they kept coming back the following days, without any measurable churn rate. Some of the students had to skip a couple of hours of classes during the week due to bureaucratic obligations and work emergencies, and sometimes people did not join the class instantly due to issues with their internet provider at home (Berlin, du bist so wunderbar). We've had one single person who pretty much gave up on delivering his solution (albeit staying on board until the end) as a result of unforeseen duties that hijacked their time and attention for the week.

This is not an illustration.

Part of the plan was letting the hands of the students go so they can explore the newly learned methodologies and tools by themselves and combine it with their existing knowledge. Our aim was to enable independent work and push for applied creative problem-solving, while staying available for everyone for questions and support via chat. The student feedback confirmed our assumption: this was a highly productive segment of the week that accelerated the pace of learning rapidly.

It was remarkable how much interest there was in seemingly niche data engineering topics and obscure tools: some people were enthusiastic and some were a bit more skeptical about building a data pipeline with some unfamiliar puzzle pieces they were not accustomed to (i.e. SQLite), but most students were open to exploration and experimentation. Five days passed by in an instant.

As a closing event, the participants who have worked in teams had to present their data products on Friday. The solutions showed two examples of highly successful and productive collaborations, and two teams had hit roadblocks that could be clearly identified and discussed to pave the way for a late delivery. In addition to the course certificate, Daniel and I decided to gift Udemy courses to the students in order to make sure that they continue pushing themselves forward on the endless path of data engineering.

Feedback time

We've asked the students to fill out a survey so we get a better understanding of how they see the Summer Camp, you can read some of their sentiments in italic below. The learnings were many:

General concept: there is a large demand for data engineering skills on the market, and even for more seasoned veterans, finding the right resources to access and learn this know-how in a structured and methodologically proven way is almost impossible. We're on the right track.
The quality of the content and the delivery was rated very high across the board and our students had fun. Oh, you are asking about an NPS score you crazy marketing researcher: it's excellent, baby. Good stuff.
- "Overall, very satisfied with the course, Daniel is a great teacher and this course was very valuable to me."
In-person interaction in a classroom vs. live remote learning via Zoom is something polarising, however the majority of students prefers personal contact. Reasons: higher productivity, more interactions with peers (aka camaraderie) and online education is just not as direct and immersive. At the same time, the remote live experience is the right way to go when it comes to a fallback plan in case COVID-19 is coming back for seconds in the winter.
Teamwork makes the dream work: we're strong believers of the value that lies in working in teams, and this is something we'll keep as integral part of our methodology.
- "The team adventures helped me to see other ways to solve the same problem and I think this should be kept for the next courses for sure.", "They helped consolidate the learnings by rolling up your sleeves and solving challenges practically with fellow peers or by yourself.", "Yes, got unblocked a few times by my partner."
We need to focus more on defining clear expectations for the challenges and tasks: striking a balance between allowing room for creative problem-solving and defining clear outcomes is the name of the game for engineering exercises. This is something we're going to fine tune by the launch of our first cohort in the fall.
- "I was demotivated at times. The instructions were too vague and I got lost in useless details (flask issue on pythonanywhere)."
Assessing the difficulty of the course based on our small sample is tough: depending on the student's level of experience, general mindset, life-circumstances outside of the summer camp and approach to self-assessment, students end up with a variety of sentiments. Fortunately a proper bootcamp experience with 10+ weeks of full-time education allows for more personalised mentoring.

More fluff

"The course delivered a good blend between technical hands-on and practical aspects about the Data Engineering profession and roles. The course has a modern approach to combine the pragmatism of the data activities (e.g. reliability, well thought changes) with modern approaches (e.g. online deployment, orchestration, monitoring, etc)."

Curricula and roadmaps for getting started with data science are plenty, data engineering however requires a different type of didactics and a learning environment that mimics the engineering and collaboration processes at companies leveraging technology.

Adam, long time data science teacher has written up his experiences at the summer camp on his blog focusing on how this week's learnings enabled him to move forward with his own projects. Make sure to check out his post for a more technical POV (full disclosure: I've had the pleasure of working with Adam for a while and consider him a friend):

"As someone who has taught data science for a while, the most impressive thing was the simplicity of the stack. It is very easy when teaching to complicate things for students, or to teach complex tools that confuse more than help.

It can’t be understated the power of leaving after five days with a working product. The value of being able to see and interact with your data is huge, for spotting problems with your data pipeline to showing off to customers (or employers!)."

The Summer Camp was my major highlight of the lockdown: it turned out to be a productive validation for Pipeline Academy, and furthermore we've managed to support people and share valuable knowledge while staying true to our principles - transparency, collaboration, pragmatism and common sense.

I hope to see you at our campus in the fall.

5 + 1 Keynotes to remember

2020-06-21T00:00:00+00:00

Unfortunately most tech content is bullshit, still there are solid cornerstones of the engineering society. They do raise their voices and did keynotes at conferences. These are the keynotes I watched more than once, some are dire, some are hilarious. Use them for inspiration, learning or simply just for fun.

Jonathan Blow: Preventing the Collapse of Civilization at DevGAMM 2019

He is an American video game designer and programmer, who is best known as the creator of the independent video games Braid and The Witness.

James Mickens: Computers are a Sadness, I am the Cure at Monitorama PDX 2014

Caveat: I do maintain the James Mickens Shrine so it was a hard decision. The classic.

David Beazley: Discovering Python at PyCon 2014

What happens when you lock a Python programmer in a secret vault containing 1.5 TBytes of C++ source code and no internet connection? Python as a secret weapon of "discovery" in an epic legal battle.

Gary Bernhardt: The Birth & Death of JavaScript at PyCon 2014

This science fiction / comedy / absurdist / completely serious talk traces the history of JavaScript, and programming in general, from 1995 until 2035

Joe Armstrong: The Mess We're In at Strange Loop 2014

The late Joe Armstrong was one of the inventors of Erlang and a great storyteller.

+1: Chris Ford: African Polyphony & Polyrhythm at Strange Loop 2016

Now this is an odd choice, isn't it? It's about models, reality, data, interpretation, culture. Also in-browser music.

The Data Janitor Letters - May 2020

2020-06-12T00:00:00+00:00

Data engineering salon. News and interesting reads about the world of data.

Why we at $FAMOUS_COMPANY Switched to $HYPED_TECHNOLOGY
Saagar Jha

Today we are making some of the code that we can afford to open source available on our GitHub page. It is useless by itself and is heavily tied to our infrastructure, but you can star it to make us seem more relevant.

Why is Artificial Intelligence So Useless for Business?
Matthew Eric Bassett

We face two more immediate problems: a lack of data and a lack of awareness.

No cookie consent walls — and no, scrolling isn’t consent, says EU data protection body
Natasha Lomas, TechCrunch

There’s increasing pressure on regulators to actually enforce the rules — with GDPR’s two year anniversary fast approaching.

Don’t Become a Data Scientist
Chris, Towards Data Science

The advice I give when someone asks me how to get into data science. Become a software engineer instead.

Teradata switches CEOs mid-flight while being eaten alive in the cloud, but it's not game over yet for data warehouser
Lindsay Clark, The Register

Last year, Gartner noted that just two vendors – AWS with Redshift and Microsoft with Azure SQL Data Warehouse – accounted for 70 per cent of the revenue growth in the overall DBMS market for 2017 and 2018.

TimescaleDB
Ajay Kulkarni, CEO and co-founder, Timescale

A multi-node, elastic, petabyte scale, time-series database on Postgres for free.

Carnegie Mellon University Database Group - Introducing ClickHouse (VIDEO)
Robert Hodges, CEO, Altinity

The Fastest Data Warehouse You've Never Heard Of

What every software engineer should know about Apache Kafka | Hacker News

"Do you happen to know what sort of thinking leads a team down this path? It seems a fundamental mistake. Resume driven development?"

This is not a test, this is a summer camp - part #1

2020-05-26T00:00:00+00:00

This is the first part of the story about setting up a live remote coding workshop in the midst an economic downturn. Part one is about the circumstances that led to all this and the way we're going with the flow.

Let's try to do this differently

I knew the road to starting a company is rocky, but dealing with the sudden surge of a pandemic was not on my radar. Once the risks of COVID-19 became clearer and the measures to stop the spread of the virus started to take shape back in March, their impact on our social lives and the projected consequences for our economy forced all of us to start coping with the idea of a future that is way more uncertain than just a couple of weeks before. If this mental state reminds you of what grief must feel like, you're not wrong.

I guess Daniel and I have arrived at the final stage of the grieving process relatively quickly: acceptance for us meant, let's move ahead and change our initial plan of opening our own coding bootcamp for now and figure out how we can contribute in a meaningful way to help people. Unlike with other natural disasters, there is no blueprint for what you should do in order to support your community (beyond the social distancing and hygiene measures), so we had to get creative. That's how we ended up with the concept of a free live remote data engineering course for individuals and businesses who (we think) can leverage the value of this type of expertise to a) reduce the risk of unemployment by diversifying their competencies and to b) enable organisations to become more efficient and therefore more resilient to an economic slump. Teaching data engineering might not sound as compassionate as organising the umpty-umpth hackathon against corona where the only comprehensible output is the participants’ vehement virtue signalling under a new hashtag, but hey, that's what we think we're good at. It was reassuring to see a lot of companies launching their makeshift initiatives to actually help others in need parallel to us in the following weeks, I think Tom Hayton put it very well in his new book comparing this process to the movie classic, Groundhog Day:

[...] Phil eventually realises that he can turn the situation to his advantage, mastering new skills and trying to win over a love interest. Only when he focuses his energy on serving others, though, does he achieve true fulfilment and happiness... [...]

Before you accuse us of being not just utterly handsome and smart, but extremely generous and altruistic as well, allow me to set the record straight: we have told all interested participants that ex post facto we would like to have one thing in exchange: their feedback.

Preparation for validation

So the overarching concept of the Pipeline Data Engineering Summer Camp was born, but the details had to be worked out:

What is a good intersection of useful, valuable data engineering expertise and practical content that we can transfer to our participants in a five day workshop?
What are the technical and methodological constraints of using live videoconferencing instead of a real classroom experience? How do we need to change our approach?
What can we expect from our participants? What should they expect from us?
What platforms and tools should we use for the virtual classroom and communication in general?
How do we make it motivating and fun?
How do we stay on top of our personal obligations (babysitting, homeschooling, watching Tiger King etc.) and manage WFH in parallel?

Don't get me wrong, it's not that there is no literature on these topics or we haven't thought about them before, but our approach based on our gut feeling has to be verified, adjusted and tested before launch.

This workshop is supposed to resemble a miniature version of our original plan, a full-blown coding bootcamp for data engineering. Our vision for creating a broader platform for the data workers of the future remains untouched, we've only adjusted the next step on the roadmap. This is important because in order to understand our rapidly growing niche even better we have to hear what works and what doesn't, especially now, when consumer sentiment and attitude are on the brink of a major realignment. Needless to say, we have a very good idea about what we should be teaching and who our ideal corporate partners would be, yet the question remains: what do our customers think about our ideas? We have to validate our assumptions and gain direct feedback from our target group(s), and this is what the Summer Camp can be used as a vehicle for.

Ultimately, we're trying to create a win-win situation here: first and foremost our participants should benefit from the workshop and based on the gained experience and feedback we should get better at what we do as well. Even though our initial motive came from a selfless place, in the end our intention has forked: helping others while helping ourselves.

Less talk, more rock

Carving out the details included sticking to our guiding principles when doing business: creating value for every stakeholder involved through an enjoyable high-quality experience. Achieving this required some ground rules, so Daniel and I agreed to the following:

No more than eight participants so we have sufficient time to focus on each person's individual challenges,
As little frontal teaching as possible, individual problem solving and working in pairs is going to be encouraged,
The goal for everyone is to deliver an end-to-end data product in five days, so the workshop will be extremely hands-on,
The scope and the way we interact with the participants should resemble our fairly unique approach to the subject matter ("the little black dress vs. fast fashion"),
The party is going to be invite only, it's meant for the network of our networks in the first place to make sure that we have participants who benefit from the camp.

After we've set the date to end of May, we started reaching out to our respective networks in order to fill the virtual room with the right people. Our assumption in April was that with the lockdown still going strong and with online courses experiencing a renaissance (especially the ones revolving around data), with the job market being in a major turmoil and with organisations still trying to get accustomed to the new normal, the demand for a free workshop about a red-hot topic like data engineering will be through the roof.

Well, this turned out only to be half-right: we've offered a spot to only 15-18 data leaders from Berlin (CTOs, CEOs, engineering leads, head of product etc.) asking them to nominate somebody from their respective data/tech teams for the workshop if they think they have a person who's a good fit (has the basics, is eager to learn and would benefit from the camp) and in case they can release them from work for a full week. In addition, we had two spots reserved for people who we knew directly and expressed interest in taking part. After two weeks with some back-and-forth about availability, we've had eight people on the list.

In the meantime, both of us were busy setting up the tools required for class communication (this involved some cursing), benchmarking competing services for live remote education (a lot more cursing), doing actual live rehearsals to get accustomed to the online format and making sure that the agenda and the schedule work fine. We've picked Zoom for videoconferencing not because it is good in any way, but it's the best out there. Using it was just like you would expect it: like sitting on a plane that is being assembled during takeoff while the flight attendant explains the nuanced meaning of the word 'operational' and their unique concept of security.

Ready for liftoff

Just a couple of days before the workshop one participant had to drop off due to an unforeseen conflict in their schedule. By sheer coincidence, the same day a person reached out to Daniel via LinkedIn: he was looking for some guidance to get his foot in the door in the world of data engineering and prepare for a new chapter of his career. He was not aware of the summer camp as we kept it under the radar (no public promotion at all), but after he learned about the scope, he was in. We've ended up with a data scientist, data engineers with different levels of experience, two data analysts and a software engineer with background in mobile and backend development on the roster. It was the perfect mix of varying skill levels and competencies, the fact that we had people from prestigious startups and scale-ups from the Berlin scene was just icing on the cake. We were ready to go.

Part #2 is about how the summer camp went and the feedback we've received. Follow us on social media and sign up to our newsletter to learn more about our adventure!

Own Your Data - like fo' real.

2020-05-17T00:00:00+00:00

Have you ever been wondering why Facebook has your memories and why you don't have them?

I had this urge last Friday CET 4:40 PM to try to get my own data from different digital services that handle it. As an EU citizen, this is my right according to the Art. 20 GDPR "Right to data portability".

Why is this important:

Startups, but even unicorns tend to go bankrupt from time to time and the users don’t get to choose the way they say goodbye to their data, meaning that whatever you have aggregated during the months and years of using a service, might disappear as quickly as a MySpace backup.
The legal and technical challenge of data portability is widely unresolved, as the idea of the user leveraging their own data on different competing platforms does not seem to be a main concern for legislators who are definitely not influenced by underpaid FAANG lobbyists.
The commoditization of the data ecosystem increases the possibility of creating data products, but it requires a governance for accessing accessing user data at least on a private and personal level.

I'm summarizing my experience as it might be useful to you also these days when some companies just fold like blam — think travel, mobility or any fluffed up unicorn. It’s about getting back what is legally yours, not necessarily about what you’re going to do with it.

Big Tech:

Amazon a pretty new development, you have to confirm it in an e-mail, took 8 days. They provided 61 separate ZIP files one has to download one by one.
Apple
Google
Microsoft

Social:

Facebook All of my data - JSON - High - Create File
LinkedIn Download larger data archive - Request archive
Twitter Twitter data download (No other download request is possible for 30 days. Like what.)

Media:

Deezer One has to file a ticket.
Mixcloud You can't.
Netflix You have to confirm in an e-mail.
SoundCloud One has to file a ticket.
Spotify Download your data You have to confirm in an e-mail.

Location:

Foursquare Export My Data
Yelp Download a copy of your Yelp data

Code:

Github Start export

If you don't find a product or service in the list below I kindly ask you to reach out to me and share your findings and experiences. Typically, the download mechanism can be found under settings and privacy. In 100% of cases, it must be triggered manually. On average you will get feedback within some workdays. In 100% of the cases, it is only a manually downloadable, expiring compressed file.

It's time to build data pipelines

2020-05-04T00:00:00+00:00

Desperate times call for uplifting speeches from venture capitalists, apparently. Some six weeks into the COVID-19 induced economic crisis Marc Andreessen has blessed us with the glorious essay "IT’S TIME TO BUILD" (sic).

For those of you who have not read it yet, here's my brief rundown:

The corona virus made us realise there are certain scalable products and services our society desperately needs but simply does not have enough of (surgical masks etc.), and the reason for that is not a lack of technical know-how or missing financial means for production, but that we consciously have chosen in the past not to build more of them. Andreessen is segueing from healthcare to housing, education and transportation to name some examples for sectors that we should literally expand both in scale and accessibility so we're better off collectively. After a spirited request to the political decision-makers on the right and the left he argues, although building is difficult, it will lead to the next wave of economic-idealogical prosperity in the US. Here's the finishing punchline:

Our nation and our civilization were built on production, on building. Our forefathers and foremothers built roads and trains, farms and factories, then the computer, the microchip, the smartphone, and uncounted thousands of other things that we now take for granted, that are all around us, that define our lives and provide for our well-being. There is only one way to honor their legacy and to create the future we want for our own children and grandchildren, and that’s to build.

First and foremost, I feel the confusion of Vicki Boykis, the queen of Normcore Tech (who also happens to be a master at curating artwork for her newsletter):

What are we, the people, supposed to make of an essay that is asking, nay, demanding that we build while at the same time ignoring what that the author has previously built - or, rather, invested in, has created as many, and maybe more, societal problems as it’s solved? The situation online as a result of many of these startups is so bad even the inventor of the internet doesn’t like it anymore.

Let's put that very valid argument aside for a second and detach the message from the messenger. Furthermore, let's also ignore our current economic realities, the ideological landscape of the Western hemisphere and the systemic structure of our society. Let's ignore the cheap argument of going borderline Sheryl Sandberg ("we need to do better") before the closing HATERS-WILL-SAY-IT'S-PHOTOSHOP!-style pre-emptive defence at the end of the original post ("I expect this essay to be the target of criticism.").

Ben Thompson comes to a much more positive conclusion after putting the piece into a broader context:

I do believe that It’s Time to Build stands alone: the point is not the details, or the author, but the sentiment. The changes that are necessary in America must go beyond one venture capitalist, or even the entire tech industry. The idea that too much regulation has made tech the only place where innovation is possible is one that must be grappled with, and fixed. And yet, Andreessen himself said that we need to demand more from one another. We need to figure out how to fix Wisconsin, not flee from it. We need to figure out how to build real businesses that build real things, not virtualize everything. And we need to start fighting for not just infinite upside, but the sort of minute changes in cities, states, and nations that will make it possible to build the future.

Moving from individual action to cooperation to arrive at collaboration is a tremendous part of what made humankind prosper in the first place. Building is a manifestation of an agreement about what we as a species need for the future and how we should invest our collective resources and yes, it presumes a shared desire and a grand vision. Building as a process carries an inherent value that is able to give purpose and hopefully even meaning to our lives. In this context building becomes the enabler of prosperity.

What does this have to do with data engineering, you ask?

A couple of weeks ago COVID-19 has derailed our timeline for starting our new school for data engineering. While we were trying to get used to the new rules of lockdown-life and attempting to reorganise our daily routines, we've decided that there is no way that we can sit back and watch how this story unfolds without us doing anything about it. We are firm believers of the transformational power of education and that sharing our professional and personal experience in the realm of data is something that can help individuals and businesses improve their competences, which as a result will make them more resistant to the economic impact of the crisis and possibly more attractive on the job market on the long run. So we came up with an idea that we're working on right now. I'll share more on this in a couple of weeks (sign up for the newsletter and follow us on instagram and LinkedIn).

And this is why the below paragraph of Marc Andreessen's essay has resonated with me so strongly, it comes from the same place as our intention. I am doing my best to avoid sounding overly pathetic and I won't compare our pseudo-altruistic relief effort to the ones essential workers are coping with on a daily basis. Still, I am excited about the next steps of this journey and encourage everyone to think about this for a minute:

Every step of the way, to everyone around us, we should be asking the question, what are you building? What are you building directly, or helping other people to build, or teaching other people to build, or taking care of people who are building? If the work you’re doing isn’t either leading to something being built or taking care of people directly, we’ve failed you, and we need to get you into a position, an occupation, a career where you can contribute to building.

The right time to build

It might sound controversial to some and a bit of a cliche to others, but I am actively trying to convince myself that starting a company during a recession is a good idea. There are plenty of well-known companies that made it and there are honest founders to tell their heroic stories of success. On a separate but related note I'd also like to introduce you to my old friend, survivorship bias... but on the other hand:

Source: Facebook

In any case… back to building.

The Data Janitor Letters - April 2020

2020-05-03T00:00:00+00:00

Data engineering salon. News and interesting reads about the world of data.

Why you should stop using Google Analytics on your website
Marko Saric, Plausible Analytics

Good reasons, not even mentioning the real important ones.

Data Science: Reality Doesn't Meet Expectations
Dan Friedman

Data & infrastructure have serious quality problems.

DeflAition
Filip Piekniewski, Scientist, Accel Robotics

The realization that deep learning is not going to cut it with respect to self driving cars and many other applications is now an open secret.

Python Web Scraping with Virtual Private Networks
Mark Litwintschik, #BigData Consultant

I'll explore two solutions, the first using WireGuard and the second, using an OpenSSH SOCKS5 proxy.

10 Things I Hate About PostgreSQL
Rick Branson, Engineering Leader, Segment

In general I’d recommend starting with PostgreSQL and then trying to figure out why it won’t work for your use case.

Things I Wished More Developers Knew About Databases
Jaana Dogan, Engineer, Google

You are lucky if 99.999% of the time network is not a problem.

Server-side tracking: surprisingly easy
Martin Loetzsch, Chief Data Officer, Project A Ventures

Pixel-based tracking is dead.

Optimize PostgreSQL Server Performance Through Configuration
Tom Swartz, Software Engineer, Crunchy Data

If a query performs heavy joins or other expensive aggregate operations, or if a query is performing a full table scan where an index could be used, it will nearly always perform poorly, no matter how well the database settings are tuned.

How to become a data engineer?

2020-04-21T00:00:00+00:00

Ok, now that we've spent some time with understanding the current state of affairs when it comes to data engineering as a function or career choice, let's take a brief look at how to actually get there.

First off, not one of today's data engineers grew up as a kid imagining becoming a data professional once they grow up. Period. There is always a weird list of circumstances that led to a point when it became necessary to update their title on LinkedIn saying "Data Engineer" or something similar. For the last 10-15 years, the more prevalent the systematic extraction of value from data has become on a macroeconomic level, the more software engineers and data analysts have been moving/pushed towards setting up and maintaining relevant infrastructures that enable data products on the micro level. The undisputed (yet more and more often criticised) hype around data science for the last 8 years has been accelerating the demand for a broader understanding of dealing with data within organisations and showed that a progressive approach is welcomed by shareholders as it indicates the intention and commitment for market dominance through innovation (just think how many times you've heard about products with "AI" inside). When it comes to talent, tooling and organisational structure, the last years were all about the rapid professionalisation of the data ecosystem and as in any healthy market starting taking shape, many different competing offerings have emerged.

For the last five years, there has been an ongoing discourse between industry stakeholders and the job market to try to codify the role and responsibility of a data engineer. With the expectations converging about what this position shall include on various levels, the educational market has started to take note and to try to address this need. However, just as quickly as the focus of data teams and the corresponding data tools change, the involved disciplines and their know-how have to adapt really fast as well. The adjustment of the supply side (available people on the job market with the right skillset) takes time though as it is influenced by so many other general economic and individual factors, and this seemingly endless yet rapid shapeshifting on both sides makes the supply-demand gap for data engineering as a function so difficult to define and grasp. As a benchmark, data science seems to be 5-6 years ahead. At Pipeline Data Engineering Academy we have a distinct point of view about the approach, general attitude and skillset a data engineer has to show in order to succeed, but for the sake of this post, allow me to address this in detail in a separate one.

Below you can find a non-comprehensive overview of the educational sector's sometimes outstanding and sometimes half-assed attempt of aiming at a moving target. Be warned: you'll have to navigate between well-written, didactically thoughtful learning tools and more or less empty and pointless yet overpriced courses taking advantage of the overhyped expectations around a career in data.

So you've made the decision to become a data engineer

As mentioned above, there are various formal options promising to help you get there, so let's start with some self-assessment in order to understand which path would be the right one for you.

Motivation: is data engineering a hobby you'd like to explore, a pre-requisite for your sidegig, or the focus of your future career; is your motivation is intrinsic or extrinsic; would you like to improve your knowledge or to receive a certificate you can show off with etc.
Experience: how new are you to the world of data and software
Time: think about the long-term commitment and the (quality) time available on a daily basis you can dedicate to learning
Costs: don't just consider the tuition or the subscription fees for courses, but the additional costs for housing, opportunity costs etc.
Learning methodology: do you like to learn on you own pace by yourself, are you looking for a classroom experience with a lot of social interaction, would you like to focus on industry best practices or on a more academic approach etc.

Keep in mind: there is no one-size-fits-all solution, but having an overview helps you to exclude the options that most likely won't work for you. If you have the chance to become an intern or an apprentice next to an experienced data engineer, you should probably take it - learning on the job offers the fastest learning curve from all the below options. Think strategically, invest your time and your money into the path that projects the highest ROI in your particular situation.

Online courses

Let's start with the online courses that offer an easy and affordable entry and self-paced learning paths for data engineers. Popular MOOCs like Coursera and Udemy offer learning tracks puzzled together mostly from an abundance of data science, data analysis, cloud engineering, Python and SQL programming courses, while Udacity is offering a nanodegree you can receive investing about 4-5 months of your time. This is a result of the above mentioned vague definition/understanding of what data engineering is and what it is supposed to be (at least that's what haters will say), and shows that thinking about data infrastructure and the non-fancy plumbing work one is confronted with later on is more like an afterthought rather than a real focus. With that being said, one has to show love for the quality of the courses, teachers and the materials: they can be engaging and rewarding, but by definition they will be far from what the industry considers state of the art. Easy to use websites, mobile apps with simple exercises (especially if you are looking into learning Python) all help to immerse yourself in the world of data related roles, but every time you hit the enroll button just ask yourself: are you in for the know-how or the certificate?

Datacamp and Dataquest are both specialised platforms focusing on the three career paths we consider traditional nowadays: data analyst, data scientist and data engineer. Heavily discounted subscription fees and extremely well made missions help you get started and build up the confidence to solve coding challenges you are faced with during a regular job interview process. Whether you prefer text-based or video-based learning, Python or R, whether you like to engage in community discussions or rather work by yourself should determine which one to pick. As a result of being much more domain-specific than generic MOOCs they offer more depth and cater for the future data professional significantly better (e.g. integration of Jupyter). For $200-$500 per annum you can get access to all their career paths so you can switch from engineering to analyst in case you deem the first one too challenging along the way.

We can't ignore branded trainings organised by the key players in the data ecosystem: hardware manufacturers and SaaS providers offer extensive trainings so their clients can actually deal with their products (Google Professional Data Engineer, Microsoft Azure Data Engineer, Oracle Cloud, Cloudera Certified Professional Data Engineer etc.). These courses are usually meant for engineers with a decent level of experience, and offer very specific know-how as opposed to a more general approach towards making things work. Pluralsight offers a wide range of courses mainly focusing on B2B clients and their workforce, but offers a self-assessment tool to support you in picking the right courses matching your skill level and your goals.

All fully online trainings lack the social component that defines traditional education: a multi-layered experience. The lack of direct in-person interaction with teachers, instructors, mentors and your peers turns the learning experience into a very efficient transfer of value, but it won't create a community in the traditional sense and it won't make you experience the social-emotional landscape of a school. Online trainings most likely won't enable you to become part of a (professional) network or an alumni group you can leverage later in life, and it's difficult for these platforms to provide proper career coaching as a result of the cultural and economic differences between regions their students come from (think job market in Silicon Valley vs. job market in Brazil).

Recommended: for newcomers trying to get a feel for what data professionals do and how disciplines compare, for software professionals without exposure to data products or statistics yet, BI managers and analysts broadening their competences and leveling up etc.

Higher education

Oh, the never-ending discussion about the countercyclical relationship between economic cycles and the offering and enrollment at higher ed institutions... Instead of going into detail about the reasons for disparities between geographic regions and socioeconomic structures in the sector of higher education, let's agree on a couple of fairly provocative statements: most universities are slow to adapt and tend to focus more on certification than the actual transfer of know-how. Historically, anticipating significant changes on the demand side of the job market and providing an appropriate response in form of an adjusted, market relevant curriculum has not been their strong suit. Hence we can observe that on the European continent - taking it as an example as it is the most relevant for Pipeline Academy - there are only a handful of universities offering Data Engineering in form of a course or a masters degree (MSc): Technische Universität Berlin (GER), Technische Universität München (GER), Hasso Plattner Institute (GER), Jacobs University (GER), Data ScienceTech Institute (FRA), Universität Potsdam (GER) and Charles University (CZ) to name the few who already integrated data engineering into their program.

The pros and cons for enrolling at a university (time commitment, financial investment, degree as a signal for the job market etc.) is something I won't cover here, yet I encourage you to take a close look at the curriculum, the teachers and the alumni to get a sense of the quality of their education. Is an MOOC an alternative to the university experience? Well, it is not. But they can complement each other very well. Also, keep in mind that the unique landscape of colleges and universities (especially the top tier) in the US will sooner than later become a playground of Silicon Valley, so expect a significant change in what we consider today a "university experience".

Recommended: computer science and software engineer graduates with BSc., business and economics students looking for a specialisation that enables them to start their career at a tech company etc.

Coding bootcamps and live trainings

Software engineering is a trade without any strict professional standards (as opposed to medical doctors, architects, lawyers etc.). Up until now, the demand for software/data professionals on the job market has been constantly higher than the supply coming from formal education and this trend does not seem to change anytime soon. These two circumstances led to the emergence of coding bootcamps taking on the role of making certain popular professions in tech accessible for anybody (web developers, UX/UI designers etc.). First and foremost, they promise focus and speed for a fraction of the costs of a traditional education (depending on the duration, classroom size, reputation etc. this will fall somewhere between $5.000-20.000 in the US, in Europe it’s around €6.000-16.000). They also offer a classroom experience with approachable instructors, a project portfolio that can be leveraged when looking for a new job, and of course career coaching that will make you nail job interviews.

Data science and it's little sister, the more approachable data analysis has been on the menu of bootcamps for a couple of years now, and it's popularity is indisputable. Data engineering however has been untouched for several reasons:

The handful of experienced data engineers are currently hotter than ever on the job market scoring the highest salaries among data professionals.
Not all data engineers have the willingness and/or the capability to teach students.
Setting up a proper data engineering curriculum and figuring out the right admission process, approaches, tools, methodologies is an investment most bootcamps don't have the capacity to make.

This is where Pipeline Data Engineering Academy comes in filling the gap. Our mission is to teach you how to build and maintain data products, machine learning systems and business intelligence tools through a 12-week intensive course. It is our belief that with this knowledge is going to set you up for a sustainable and rewarding long-term career.

But I digress...

Before you sign up to our next training, I urge you to read Quincy Larson's The Coding Bootcamp Handbook: it will help you prioritise your selection criteria and become more informed before you make a decision about joining any coding bootcamp. Approaching this journey with a certain level of scepticism is the healthy way forward. You probably want to take a look at student feedback on CourseReport and SwitchUp, the two leading coding bootcamp review sites: just make sure you consider survivorship bias while reading through the comments. For the folks in the US, Thinkful has created a smart and simple bootcamp comparison tool.

A note on COVID-19: most coding bootcamps have been forced to move their in-person courses online. There is a good reason why these institutions had classrooms in the first place, and this is something everyone expects to come back one way or the other after the corona restrictions are lifted. However, in case you choose to have a 100% remote coding bootcamp experience, you should ask yourself if the value for money ratio still holds up when compared to a recorded online training.

Recommended: for data analysts and jr. data scientists looking for expertise in building and maintaining data products, for engineers from different areas (frontend, devops etc.) transitioning into a new field etc.

Other resources

Whatever path you choose, there are a handful of resources that you should know about in order to complement your learning experience and to help you navigate your own ship. In case you find something you think should be on this list, please reach out to me and let me know!

Notable articles and blog posts:

What is a Data Engineer? (dataquest.io)
A Beginner’s Guide to Data Engineering — Part I, Part II, Part III (medium.com)
Data Engineering — Complete Reference Guide From A-Z [2019] (towardsdatascience.com)
Data engineers vs. data scientists (oreilly.com)
Newsletter: Data Eng Weekly (https://dataengweekly.com/)
Online mentoring platforms: SharpestMinds, MentorCruise

I encourage you to explore your options and start your transitioning into the world of data engineering.

The Data Janitor Letters - March 2020

2020-04-15T00:00:00+00:00

Data engineering salon. News and interesting reads about the world of data.

The End of Starsky Robotics
Stefan Seltz-Axmacher, CEO, Starsky Robotics

The biggest open secret is that supervised machine learning does not live up to the hype.

We Made SQL Visual - Why and How
Dave Fowler, CEO, Chartio

We don’t pretend to have all the answers, and we’re not done innovating yet—eight in ten is not ten in ten—but it’s worth reflecting on what has worked so far.

Intel x86 Root of Trust: loss of trust
Mark Ermolov, Positive Technologies

Nobody should worry about security anymore if they are virtualizing on shared hardware.

PostgreSQL is the worlds' best database
Kirk Roybal, 2ndQuadrant, PostgreSQL Trainer and DBA

The title is not clickbait or hyperbole.

The role of data engineering post-COVID-19

2020-04-12T00:00:00+00:00

The corona virus has changed consumer behaviour more significantly than any other event in this century and as a result, the job market has been turned upside down. The demand for certain products and services has vanished from one day to the other, but at the same time a handful of sectors are growing faster than ever before. While in the middle of setting up Pipeline Data Engineering Academy, we've been caught up in this unexpected turmoil as well, so I thought to myself, the only right path forward is making sense of what's next to ensure that the value we aim to provide stays relevant.

Tech is here to stay, but it's going to adapt

First, let's turn to Anil Dash of Glitch for some clarity about what technology really is and how we would expect it to behave during a crisis:

Technology isn’t an industry, it’s a method of transforming the culture and economics of existing systems and institutions.

It is a misconception that tech is an industry by itself, therefore we can't just make predictions or assessments about the companies we consider part of the Silicon Valley ecosystem. All verticals are going to be impacted differently as consumption is quickly adjusting to the new rules of living under lockdown (i.e. travel vs. streaming media), while within those verticals winners will emerge on top of the losers corpses. Technology as a method, the main driver of growth in the last decade is forced to change, there is no way around it: if existing systems and institutions are adapting to the (post-)corona world, the approach of transforming their culture and economics is going to have to find it's new ways as well.

"Nobody ever got fired for buying IBM." Part #2

During the time of economic growth and general positive attitude towards the near to mid term future, corporates and individuals are willing to take more risks when it comes to investments. The pre-corona world at the stock market made a large bet on growth driven by emerging technologies which require a significant financial investment and long-term commitment from organisations, but this type of capital also presumes consumer demand (in the form of disposable income of households) and a general positive trajectory of the markets. This notion resulted in a crazy number of startup unicorns and what felt like the next industrial revolution: things were good working in tech, job security was not a topic on peoples minds. Some signals were there pointing towards an economic bubble waiting to burst, but being bitchslapped by a pandemic was not a scenario to be considered when discussing the upcoming fiscal year in the board room.

In times of financial distress and unpredictability, people start looking at what they have and how this measures up when hardship is knocking on the door. Decision makers are going to have one thing on their minds: avoiding unnecessary risks. In Google-speak:

By dialing back our plans in other areas, we can ensure Google emerges from this year at a more appropriate size and scale than we would otherwise. That means we need to carefully prioritize hiring employees who will address our greatest user and business needs.

A significant part of the innovation economy is high-risk high-reward, and all of us are witnessing the process of sobering/falling out of love with unpredictable results - also referred to as 'market correction'. Spending corporate money without demonstrating hard ROI will be nearly impossible, and this goes for hiring in particular.

Data scientists and data engineers have built-in job security relative to other positions as businesses transition their operations to rely more heavily on data... [ ]

Makes sense, right? Basically, your tech job will be only as good as the value of your measurable output, and data related roles carry significant returns for consumers (external) and decision makers (internal) alike.

The case for data and engineering

One of the building blocks of technology itself that enabled/drove digital transformation with exponentially increasing pace is data, and the players in various industries making more and more use of it (whether the economic value provided through leveraging technology and data is adequately measured by the stock market and company capitalisation shall remain a separate conversation). Taking a closer look at the various disciplines within the realms of a cross-functional digital product team (from agile coaches, design thinking facilitators, backend developers, UX designers, devops engineers etc.), it's likely that non-essential roles will have trouble flourishing as a consequence of the above when the main company concern is providing a cost-efficient and technologically robust core service, ideally based on data-driven decision-making. We are going to observe a general structural shift towards the maintenance and operations of existing tech products in exchange for developing new prototypes and "failing quickly".

It's back to the basics, and the mindset of an experienced data engineer is all about that. But don't take it from me, take it from McKinsey Digital recommending CDOs to focus on staying operational and ensuring business continuity as a first instance.

Data engineering is one of the most underrated but essential roles in a digital product team that is going to stay on the hiring list of companies working with tech. Before COVID-19, we've seen a 50% yoy growth in 2019 in the demand for engineers able to work with big data and a corresponding rapid increase of salaries, and although the growth is somewhat slower now, the crisis verified that the demand is robust.

This too shall pass.

If there is something the dotcom bubble and the 2008 economic crisis have taught us it’s that the economy is able to bounce back, it always has in the past. Whether it's a 'V'-shaped return or more like an elongated 'U', we'll learn soon enough. Regardless, when it comes to the years between the high-times, you better stick to the basics... and it seems like data engineering remains one of the essential disciplines companies working with data desperately need.

The Data Janitor Letters - February 2020

2020-03-08T00:00:00+00:00

Data engineering salon. News and interesting reads about the world of data.

Agile as Trauma
Dorian Taylor

The Agile Manifesto is an immune response on the part of programmers to bad management.

Andreessen-Horowitz craps on “AI” startups from a great height
Scott Locklin

Machine learning will be most productive inside large organizations that have data and process inefficiencies.

Five Interesting Data Engineering Projects
Dmitriy Ryaboy

DBT, Prefect, Dask, DVC, Great Expectations.

Distributed Data for Microservices — Event Sourcing vs. Change Data Capture
Eric Murphy, Debezium

Great sales speech, good tidbits, make up your own mind.

Automating MySQL schema migrations with GitHub Actions and more
Shlomi Noach, Principal Software Engineer, GitHub

Your mileage may vary, a reasonable case study.

Are ad blockers doomed or have we already won? A history lesson
Vasily Bagirov, AdGuard

Our guess is that to stay competitive, ad blockers will have to move to a network-level approach. It means that an ad blocker of the future will have to monitor traffic of the entire network.

The Data Janitor Letters - January 2020

2020-02-16T00:00:00+00:00

Data engineering salon. News and interesting reads about the world of data.

Code reading is the single largest expense in software development, and very few even talk about it.

@girba

Stop Hiring Data Scientists.
Luke Posey, CEO, Spawner

Your ROI is suffering from an inability to hire properly.

Flow chart: Do we need to deploy a new technology/system?
Stefan Urbanek, advisor, Data Engineering Academy.

You bet.

GPT-2 and the Nature of Intelligence
Gary Marcus, CEO, Robust.AI

In essence, GPT-2 has been a monumental experiment in Locke's hypothesis, and so far it has failed.

Our Neophobic, Conservative AI Overlords Want Everything to Stay the Same
Cory Doctorow

Ultimately, machine learning is about finding things that are similar to things the machine learning system can already model.

Are you using #postgres via #docker for mac? Have you ever noticed EXPLAIN ANALYZE slowing down your queries by like 60x? The important takeaway is that our modern stacks are incredibly complex and fragile.

@felixge

ClickHouse Cost-Efficiency in Action: Analyzing 500 Billion Rows on an Intel NUC
Alexander Zaitsev, Co-founder, Altinity

A single ClickHouse server can be used to collect and monitor temperature data from 1,000,000 homes, find temperature anomalies, provide data for real-time visualisation and much more. Since it is a single server, setting it up, loading 500B rows and running sample queries is very easy.

Security in Machine Learning Engineering: A white-box attack and simple countermeasures
Flávio Clésio, Senior Machine Learning Engineer, MyHammer AG

After running a simple script based in using Scikit-Learn, I noticed there’s some latent vulnerabilities not only in terms of objects but also in regarding to have a proper security mindset when we’re developing ML models.

Fast IPv4 to Host Lookups
Mark Litwintschik, #BigData Consultant

My interest here is in seeing the performance differences between using PostgreSQL with a B-Tree index versus ClickHouse and its MergeTree engine for this use case. The performance gap in the hourly lookup rate favouring ClickHouse is off by an order of magnitude.

Pipeline Data Engineering Academy

Pipeline Academy on Hiatus

Health and family

Not a “Goodbye", but hopefully a “See you soon!”

Meet The Graduates: Guoda Paulikaite

Pipeline Academy Setting Trends at the EdTech Awards

I am absolutely delighted to announce that our work has been recognised in two categories this year:

The Data Janitor Letters - February 2022

The Unbundling of Airflow Gorkem Yurtseven, Co-Founder, Features and Labels

Rebundling the Data Platform Nick Schrock, Founder, Elementl

Fundraising by data companies in 2021 Bence Arató, Managing Director, BI Consulting

Analytics Stacks for Startups Jan Katins, Senior IT Consultant/Data Engineer, kreuzwerker GmbH

Make Your Workflows Better With This Classic Unix Tool Adam Green, Data Engineer, Gridcognition

GDPR enforcer rules that IAB Europe's consent popups are unlawful Irish Council for Civil Liberties

CNIL decides EU-US data transfer to Google Analytics illegal NOYB – European Center for Digital Rights

Here's What's Wrong With GDPR Shoshana Wodinsky, Data Reporter, Gizmodo

Unbiggen AIAndrew Ng, Founder and CEO, Landing AI

The Data Janitor Letters - January 2022

We’ve only scratched the surface of the full potential for the data warehouse Mikkel Dengsøe, Head of Data Science, Operations & Financial Crime, Monzo Bank

Git, SQL, CLI Vicki Boykis, Machine Learning Engineer, Automattic

One Year of dbt Adam Boscarino, Director of Data Engineering, Devoted Health

Why OpenMetadata is the Right Choice for you Suresh Srinivas, Co-Founder, Collate

The future history of Data Engineering Matt Arderne, Product Engineering, Data & Analytics, focal

Airflow Alternatives: A Look at Prefect and Dagster Pedram Navid, Head of Data, Hightouch

Modern Data Stack: Which Place for Spark? Furcy Pin, Lead Data Engineer, Younited Credit

Why I Quit Data Science Nirant Kasliwal, Analytics Engineer, Sundial

Become A Better Data Engineer On A Shoestring (More Free Resources)

General

Python

SQL

CLI

DevOps

What Did You Build at Pipeline Academy? This.

Recommended READ:The data engineering Portfolio project

Portfolio projects

The Data Janitor Letters - December 2021

Databases in 2021: A Year in Review Dr. Andy Pavlo, Co-founder, OtterTune

Building an End-to-End Open-Source Modern Data Platform Mahdi Karabiben, Senior Data Engineer, Zendesk

Postgres, Kafka, and the Mysterious 100 GB Matt Ritter, Software Engineer, Google

Consider SQLite Wesley Aptekar-Cassels

Five Tips For a Healthier Postgres Database in the New Year Craig Kerstiens, Product, Crunchy Data

The Death of Feature Engineering is Greatly Exaggerated Pete Warden, Staff Research Engineer, Google

Don’t start with microservices in production – monoliths are your friend Arnold Gálovics, Delivery Manager, EPAM Systems

"Open Source" is Broken Christine Dodrill, Software Designer, Tailscale

Our 2021 in a Nutshell

The Data Janitor Letters - November 2021

Tracking-industry body IAB Europe told that it has infringed the GDPRIrish Council for Civil Liberties

Why the Data Analyst role has never been harderPetr Janda, CTO, Pleo

Spreadsheets: The Duct Tape of the Modern Data StackMikkel Dengsøe, Head of Data Science, Operations & Financial Crime, Monzo Bank

Serverless dbt on Google Cloud PlatformRobert Sahlin, Senior Data Engineer, MatHem.se

Orchestrating ELT with Prefect and dbt — a Flow of Flows (Part 1) Anna Geller, Solutions Engineer, Prefect

Modelling Type 1 + 2 Slowly Changing Dimensions with dbt Weng-Kin Lee, Associate Consultant, Servian

Mapping our data journey with column lineage Borja Vázquez Barreiros, Senior Analytics Engineer, Monzo

Lesser Known PostgreSQL Features Haki Benita, Development Team Lead, PCENTRA

DuckDB-Wasm: Efficient Analytical SQL in the Browser André Kohn and Dominik Moritz, DuckDB

From Data Engineer to SysAdminJonathon Belotti, Team Lead Data Platform, Canva

Will Nix Overtake Docker? Connor Brewster, Software Engineer, Replit

Storm in the stratosphere: how the cloud will be reshuffled Erik Bernhardsson

The Pipeline Academy Awards 2021

THE MODERN DATA STACK.

Separating the wheat from the chaff

The 2021 Pipies go to…

Data Acquisition: Airbyte

Telemetry: Quix

ETL/ELT: Prefect

Data Modeling: dbt

Data Warehousing: ClickHouse

Data Quality: Great Expectations

Clouds: Pulumi

Deployment: Fly

Serving Data: Streamlit

Honorable mentions:

Computation: Saturn Cloud

CD/CI: GitHub Actions

Serving Data: Datasette

The Pipies 2021

Skills Gap in Data Engineering

READ THE FULL REPORT: Developing skills and establishing new pathways into jobs (.pdf)

10 Reasons Why Aspiring Data Engineers Choose Pipeline Academy

1. Online courses don’t work for most

The Unbundling of Airflow
Gorkem Yurtseven, Co-Founder, Features and Labels

Rebundling the Data Platform
Nick Schrock, Founder, Elementl

Fundraising by data companies in 2021
Bence Arató, Managing Director, BI Consulting

Analytics Stacks for Startups
Jan Katins, Senior IT Consultant/Data Engineer, kreuzwerker GmbH

Make Your Workflows Better With This Classic Unix Tool
Adam Green, Data Engineer, Gridcognition

GDPR enforcer rules that IAB Europe's consent popups are unlawful
Irish Council for Civil Liberties

CNIL decides EU-US data transfer to Google Analytics illegal
NOYB – European Center for Digital Rights

Here's What's Wrong With GDPR
Shoshana Wodinsky, Data Reporter, Gizmodo

Unbiggen AI
Andrew Ng, Founder and CEO, Landing AI

We’ve only scratched the surface of the full potential for the data warehouse
Mikkel Dengsøe, Head of Data Science, Operations & Financial Crime, Monzo Bank

Git, SQL, CLI
Vicki Boykis, Machine Learning Engineer, Automattic

One Year of dbt
Adam Boscarino, Director of Data Engineering, Devoted Health

Why OpenMetadata is the Right Choice for you
Suresh Srinivas, Co-Founder, Collate

The future history of Data Engineering
Matt Arderne, Product Engineering, Data & Analytics, focal

Airflow Alternatives: A Look at Prefect and Dagster
Pedram Navid, Head of Data, Hightouch

Modern Data Stack: Which Place for Spark?
Furcy Pin, Lead Data Engineer, Younited Credit

Why I Quit Data Science
Nirant Kasliwal, Analytics Engineer, Sundial

Recommended READ:
The data engineering Portfolio project

Databases in 2021: A Year in Review
Dr. Andy Pavlo, Co-founder, OtterTune

Building an End-to-End Open-Source Modern Data Platform
Mahdi Karabiben, Senior Data Engineer, Zendesk

Postgres, Kafka, and the Mysterious 100 GB
Matt Ritter, Software Engineer, Google

Consider SQLite
Wesley Aptekar-Cassels

Five Tips For a Healthier Postgres Database in the New Year
Craig Kerstiens, Product, Crunchy Data

The Death of Feature Engineering is Greatly Exaggerated
Pete Warden, Staff Research Engineer, Google

Don’t start with microservices in production – monoliths are your friend
Arnold Gálovics, Delivery Manager, EPAM Systems

"Open Source" is Broken
Christine Dodrill, Software Designer, Tailscale

Tracking-industry body IAB Europe told that it has infringed the GDPR
Irish Council for Civil Liberties

Why the Data Analyst role has never been harder
Petr Janda, CTO, Pleo

Spreadsheets: The Duct Tape of the Modern Data Stack
Mikkel Dengsøe, Head of Data Science, Operations & Financial Crime, Monzo Bank

Serverless dbt on Google Cloud Platform
Robert Sahlin, Senior Data Engineer, MatHem.se

Orchestrating ELT with Prefect and dbt — a Flow of Flows (Part 1)
Anna Geller, Solutions Engineer, Prefect

Modelling Type 1 + 2 Slowly Changing Dimensions with dbt
Weng-Kin Lee, Associate Consultant, Servian

Mapping our data journey with column lineage
Borja Vázquez Barreiros, Senior Analytics Engineer, Monzo

Lesser Known PostgreSQL Features
Haki Benita, Development Team Lead, PCENTRA

DuckDB-Wasm: Efficient Analytical SQL in the Browser
André Kohn and Dominik Moritz, DuckDB

From Data Engineer to SysAdmin
Jonathon Belotti, Team Lead Data Platform, Canva

Will Nix Overtake Docker?
Connor Brewster, Software Engineer, Replit

Storm in the stratosphere: how the cloud will be reshuffled
Erik Bernhardsson

Eating the Cloud from Outside In
Shawn Wang, Developer Experience, Temporal.io

Why Lightspeed invested in ClickHouse: a database built for speed
Gaurav Gupta, VC, Lightspeed Venture Partners

Day In The Life Of A Data Engineer — What Do Data Engineers Do?
SeattleDataGuy

ROAPI: An API Server for Static Datasets
Mark Litwintschik, #bigdata Consultant

Announcing Streamlit 1.0! 🎈
Adrien Treuille, Co-Founder and CEO, Streamlit

Function pipelines
David Kohn, Developer, Timescale

Implement a slowly changing dimension in Amazon Redshift
Milind Oke and Bhanu Pittampally, Amazon

DBT at HomeToGo. Creating a scalable framework
Gijs de Kruif, HomeToGo

How to Execute Pandas Workloads in a Distributed Manner With Apache Spark
Hyukjin Kwon and Xinrong Meng, Databricks

Bash functions are better than I thought
Gary Verhaegen, Senior Software Engineer, Digital Asset

Cloudflare’s Disruption
Ben Thompson, Stratechery

Operations is not Developer IT
Mathew Duggan, DevOps Manager, GAN Integrity

How Big Tech Runs Tech Projects and the Curious Absence of Scrum
Gergely Orosz

Who are analysts, technically?
Bobby Pinero, CEO and Co-Founder, Equals

Re-evaluating Kafka: issues and alternatives for real-time
Olivia Iannone Technical Writer at Estuary

A very detailed comparison of Python stream processing libraries
Mike Rosam, Cofounder and CEO, Quix

The First Rule of Machine Learning: Start without Machine Learning
Eugene Yan, Applied Scientist, Amazon

Reaching MLE (machine learning enlightenment)
Vicki Boykis, Machine Learning Engineer, Automattic

From Data Driven to Driving Data — The dysfunctions of Data Engineering
MrTrustworthy

What's an OLAP cube? 🎲
Claire Carroll, Analytics Engineer, analyticsengineers.club

Expecting Great Quality with OpenLineage Facets
Michael Collado, Staff Software Engineer, Datakin

A future for SQL on the web
James Long, Software Engineer, Stripe

MinIO: A Bare Metal Drop-In for AWS S3
Mark Litwintschik, Big Data Consultant

No, we don’t use Kubernetes
Maik Zumstrull, Site Reliability Engineer, Ably

Evolution of search engines architecture
Julien Lemoine, Co-founder & CTO, Algolia

Building a data team at a mid-stage startup: a short story
Erik Bernhardsson, Working on something, "Bernco"

Analytics is at a crossroads
Benn Stancil, Chief Analytics Officer + Founder, Mode

Against SQL
Jamie Brandon, independent researcher

For SQL
Pedram Navid, Head of Data, Hightouch

The Untold Story of SQLite With Richard Hipp
Adam Gordon Bell, CoRecursive Podcast

The Analytics Engineering Guide
dbt Labs

Welcome to Snowpark: New Data Programmability for the Data Cloud
Isaac Kunen, Senior Product Manager, Snowflake

Accidentally exponential behavior in Spark
Ivan Vergiliev, Tech Lead, Leanplum

The ritual of the deploy
Vicki Boykis, Machine Learning Engineer, Tumblr

Gently Down the Stream
Mitch Seymour, Illustrator, Author, Founder, Round Robin Publishing

What's Kafka and what does Confluent do?
Justin Gage, Technically

Cloudera to go private as KKR & CD&R grab it for $5.3B
Ron Miller, TechCrunch