Pipeline Data Engineering Academy home blog pages letters podcast

Data Engineering Advent Calendar 2020

Throughout December 2020 we’ve shared a daily dose of semi-esoteric data engineering wisdom on our social media channels (instagram and LinkedIn). This post shall serve as a commemorative monolith you can always turn to when the data engineering gods are not picking up your call.

#1: Don’t write code, solve the problem.

#2: Python is always at hand to pretty print a JSON:

$ python3 -m json.tool some.json

#3: EXPLAIN is your friend.

#4: "Choose boring technology." Dan McKinley @mcfunley

#5: Complicated is better than complex.

#6: What do you do on your CLI?

$ < ~/.bash_history | sort | uniq -c | sort -n

#7: SQLite (2000) has one trillion (1e12) active installs. It's a file with SQL API and window functions.

#8: "Premature optimization is the root of all evil." Tony Hoare (Quicksort)

#9: Keep It Simple (&) Stupid and remember separation of concerns.

Data engineering advent calendar, day 10

#10: Log in to a recently launched container (via @basmatitree)

$ docker exec -it $(docker ps -q | tail -1) /bin/bash

Data engineering advent calendar, day 11

#11: Get details of last failed Redshift load.

SELECT * FROM stl_load_errors ORDER BY starttime DESC LIMIT 1;

Data engineering advent calendar, day 12

#12: "When in doubt, use brute force." Ken Thompson (Go, UTF-8, Unix)

Data engineering advent calendar, day 13

#13: Maintainable is debuggable and testable, and has version control.

Data engineering advent calendar, day 14

#14: Remove all local git branches other than master and the currently used one (via @advincze)

$ git branch --no-color | grep -v 'master' | grep -v '*' | xargs git branch -D

Data engineering advent calendar, day 15

#15: Order of execution in SQL:

FROM WHERE GROUP BY HAVING SELECT [DISTINCT] UNION ORDER BY

Data engineering advent calendar, day 16

#16: "Don't reinvent the flat tire." Alan Kay (Squeak, Smalltalk, OOP, GUI)

Data engineering advent calendar, day 17

#17: Code is dependency. Others' code is dependency squared. Delete, remove, retire.

Data engineering advent calendar, day 18

#18: You can run SQL directly on your CLI on CSV or TSV files with http://harelba.github.io/q/

Data engineering advent calendar, day 19

#19: Queries on MPPs? Use WITH/CTEs, filter with WHERE, SELECT explicitly, avoid JOIN, SORTKEYS are your friends. It's all about scanning less.

Data engineering advent calendar, day 20

#20: “Bad programmers worry about the code. Good programmers worry about data structures and their relationships.” Linus Torvalds (Git, Linux)

Data engineering advent calendar, day 21

#21: The longer a technology lives, the longer it can be expected to live (Lindy effect)

Data engineering advent calendar, day 22

#22: Delete files recursively:

$ find . -name "*.pdf" -print0 | xargs -0 rm

Data engineering advent calendar, day 23

#23: Why PostgreSQL (1996)? It is the open source RDBMS with columnar (cstore), geo (PostGIS), timeseries (TimescaleDB) and REST API (PostgREST).

Data engineering advent calendar, day 24

#24: "Use simple algorithms as well as simple data structures." Rob Pike (Go, UTF-8, Unix)