Uncategorized

So, I was digging into some DAGs the other day and kept thinking about how Airflow was literally born at Airbnb. I thought I’d write this up because a lot of folks treat Airflow like a magic black box, but understanding that Airbnb DNA actually helps you write better pipelines. Here’s what I’ve learned from … Read more

So, you’re dumping data into BigQuery and suddenly your storage bill is looking a little spicy. You’ve got partitions, but they’re hanging around forever. This happens. It’s growing pains. I’ve been there. Let’s go over how to set partition expiration so you can automate the cleanup and keep costs down. Why Bother with Expiration? You … Read more

So, I’ve been orchestrating data pipelines with Airflow for a while now, and while the Airbnb origins are cool, the real value is in how you actually use it day-to-day. I thought I’d share a few practical tips and code snippets that saved me from pulling my hair out. Keep Your DAGs Simple Airflow loves … Read more