Feral Cat - Software Development
Building, just because…
Are you managing a BigQuery environment at scale and dealing with slow BigQuery job execution? Does your company have hundreds or thousands of jobs running simultaneously and sometimes experience wildly slow query execution? This happens. It’s growing pains. We’ll go over a couple of methods of bringing your query speeds back up which might eventually … Read more
In this blog post we’re going to cover how to make your SQL queries blazing fast. So, BigQuery is an incredible tool for wrangling massive datasets and running SQL queries at scale. But let’s be honest—just because it can handle huge queries doesn’t mean you should throw inefficiency at it. The faster your queries run, … Read more
Efficiently managing partitioned tables in Google BigQuery is crucial for performance and cost optimization. A common task is determining partition existence in BigQuery within a table. Instead of scanning entire tables, which can be time-consuming and expensive, BigQuery offers a more efficient approach using the INFORMATION_SCHEMA.PARTITIONS view. Using INFORMATION_SCHEMA to Check Partition Existence BigQuery’s INFORMATION_SCHEMA … Read more
This post will show you how to pull a domain from a full website path in BigQuery. So let’s set the stage for a hypothetical. You own a URL shortener company. You want to partner with a website for whatever reason. You decide that you want to do analysis over the data you’ve streamed or … Read more
In what situation would you want to use BigQuery’s having clause outside of an interview? We’ll go over a couple of use cases and how I use it as a Data Engineer for Reddit. My Setup What Is BigQuery? BigQuery is a data warehouse as a service. Google handles your compute, your storage, and does … Read more
Why Might I need A Partition Expiration? Partition expirations can be leveraged for several use cases. Some countries have laws pertaining to how long PII can be held. Storage is very expensive, maybe your company wants to save some money? The list can and will go on. But it gets a little harder when the … Read more