Building, just because…
In this blog post we’re going to cover how to make your SQL queries blazing fast. So, BigQuery is an incredible tool for wrangling massive datasets and running SQL queries at scale. But let’s be honest—just because it can handle huge queries doesn’t mean you should throw inefficiency at it. The faster your queries run, … Read more
This past weekend I decided to spin up a quick Airflow deployment for some personal scripts I wanted executed on a schedule. I didn’t have scale in mind and I didn’t have robustness in mind. I had speed. I’ll review how I’m deploying my version-controlled DAG code to my EC2. I want to highlight, this … Read more
Efficiently managing partitioned tables in Google BigQuery is crucial for performance and cost optimization. A common task is determining partition existence in BigQuery within a table. Instead of scanning entire tables, which can be time-consuming and expensive, BigQuery offers a more efficient approach using the INFORMATION_SCHEMA.PARTITIONS view. Using INFORMATION_SCHEMA to Check Partition Existence BigQuery’s INFORMATION_SCHEMA … Read more
Comms as a Data Engineer can be tough. Should you email a group of people? Should you dump a message in a public Slack channel they frequent? Should you follow up daily, weekly, etc? It’s a lot of manual labor. I don’t like manual work. Also, I hate email. This seems to be a common … Read more
What in the the world are Apache Airflow DAG Factories and why should you use them? Let’s go into what they are, why they’re used, and how they could make your life easier. We’ll also go into the nitty gritty of how to design and build one. Also, before I jump into this post, shout … Read more
So, a loooong time ago I wrote this post on how to clone a Git repo in Python3. I used subprocess that first time around to run git commands. I was essentially trying to run git commands in python explicitly. But, there’s a better way to do this. It’s prettier, it’s easier to read. There’s … Read more
This post will show you how to pull a domain from a full website path in BigQuery. So let’s set the stage for a hypothetical. You own a URL shortener company. You want to partner with a website for whatever reason. You decide that you want to do analysis over the data you’ve streamed or … Read more
Background Info This blog post will discuss AWS CloudFormation, PHP, and WordPress issues. So, a couple of years back I decided to leverage a CloudFormation template to scaffold a WordPress blog. Do note that this link is close to what I used, but it isn’t exactly what I used. It spun up a load balancer, … Read more
Hello folks. My name is Carlos Cabán and I’m a Data Engineer at Reddit as of October of 2023. I got into Data Engineering without a software degree. I thought I’d go over how I went from being an analyst out of college to doing larger scale data work with companies like Reddit and Netflix … Read more
Users have expectations when they’re using your tools. Maybe you have a message app? Your enter button should probably send your message. Tab might make you switch windows or menus. We’ll cover how to run a function when your user hits the enter button in a NextJS application. This will also work for any generic … Read more