Pipeline Data Engineering Academy home blog pages letters

The Data Janitor Letters - December 2021

Data engineering salon. News and interesting reads about the world of data.

Databases in 2021: A Year in Review
Dr. Andy Pavlo, Co-founder, OtterTune

It was a wild year for the database industry, with newcomers overtaking the old guard, vendors fighting over benchmark numbers, and eye-popping funding rounds. We also had to say goodbye to some of our database friends through acquisitions, bankruptcies, or retractions.

Building an End-to-End Open-Source Modern Data Platform
Mahdi Karabiben, Senior Data Engineer, Zendesk

A detailed guide to help you navigate the modern data stack and build your own platform using open-source technologies.

Postgres, Kafka, and the Mysterious 100 GB
Matt Ritter, Software Engineer, Google

I think the experience shows that PoCs using “production infrastructure” can expose pitfalls that might appear in a real implementation.

Consider SQLite
Wesley Aptekar-Cassels

As long as you don't expect to need tens of thousands of small writes per second, thousands of large writes, or long-lived write transactions, it's highly likely that SQLite will support your usecase.

Five Tips For a Healthier Postgres Database in the New Year
Craig Kerstiens, Product, Crunchy Data

While onboarding customer after customer this year I've noted a few key things everyone should put in place right away - to either improve the health of your database or to save yourself from a bad day.

The Death of Feature Engineering is Greatly Exaggerated
Pete Warden, Staff Research Engineer, Google

I don’t want to minimize deep learning’s achievements in reducing the toil involved in building feature pipelines, I’m still constantly amazed at how effective they are. I would like to see more emphasis put on feature engineering in research and teaching though.

Don’t start with microservices in production – monoliths are your friend
Arnold Gálovics, Delivery Manager, EPAM Systems

I’m gonna hit you with hard truth my friend.

"Open Source" is Broken
Christine Dodrill, Software Designer, Tailscale

Why I don't write useful software unless you pay me.