Cast an array of items using lambdas in Python.

python

Let’s cast an array of items using lambdas in Python. It’ll look cleaner than its forloop counterpart. They’re a great way to clean up super verbose code and helpful when doing array manipulation. Before we begin, if you don’t know what a Python lambda is please check out this article we recently wrote. My most … Read moreCast an array of items using lambdas in Python.

Run Apache Airflow Locally in Docker

We’ll walk you through how to run Apache Airflow locally in Docker. The first chunk of this post will cover how to get Airflow standing, the second will go into some nuance and will answer several whys. My Setup Getting Airflow Standing Docker Airflow Looking for more detailed info? Check out Airflow’s official docs. Happy … Read moreRun Apache Airflow Locally in Docker

Execute Bash Commands and Return Results in Python

bash

I’ve had numerous cases where I’ve needed to execute bash commands and return the results in Python for some additional manipulation. My most recent example involved me working with BigQuery schemas. Getting schemas from the CLI was easy peasy but I’m not the best at BASH programming so I naturally turned to something more familiar … Read moreExecute Bash Commands and Return Results in Python

What Happens When Your Code Does Not Follow The Python Rules?

python

So what happens if your code does not follow the Python rules? This article will be fairly short. Most of this will not depend on your version of Python. Tabbing/Spacing So what happens if you get your tab or spacing incorrect? Let’s give it a go. Often times the best way to figure something like … Read moreWhat Happens When Your Code Does Not Follow The Python Rules?

Apache Airflow DAG is Failing Silently

python

So your Apache Airflow DAG is failing silently. Are you running an ETL on a huge dataset? This is a symptom of an Airflow instance without sufficient memory. Dig into your instances logs and you’ll probably see an evicted worker if your running your instance’s workers on Kubernetes. You’ll see similar logs wherever you run … Read moreApache Airflow DAG is Failing Silently

Functional Annotations in Python 3.x

python

Have you used Functional Annotations in Python 3.x? Maybe you’ve heard them mentioned? Regardless, let’s explore what they are and how they help us. Because if they don’t help us, then we probably shouldn’t care. The Problem You’re programming and don’t know what thisRandomFunction should return. Maybe it’s a bool maybe it’s a string who … Read moreFunctional Annotations in Python 3.x

How To Get Started With Apache Airflow?

When Airbnb was scaling rapidly, they faced the problem of organizing complex data pipelines. To combat this and become a data-driven organization, Airbnb launched Apache Airflow in 2015, their custom-made open-source platform to manage complex workflows. In simple words, Apache Airflow is a platform where you can create, schedule, and monitor complex workflows using simple … Read moreHow To Get Started With Apache Airflow?