Building, just because…
We’ll walk you through how to run Apache Airflow locally in Docker. The first chunk of this post will cover how to get Airflow standing, the second will go into some nuance and will answer several whys. My Setup MacOS Ventura M1 Chip Getting Airflow Standing Docker 1. Download Docker Desktop from this link. Make …
I’ve had numerous cases where I’ve needed to execute bash commands and return the results in Python for some additional manipulation. My most recent example involved me working with BigQuery schemas. Getting schemas from the CLI was easy peasy but I’m not the best at BASH programming so I naturally turned to something more familiar …
So what happens if your code does not follow the Python rules? This article will be fairly short. Most of this will not depend on your version of Python. Tabbing/Spacing So what happens if you get your tab or spacing incorrect? Let’s give it a go. Often times the best way to figure something like …
So your Apache Airflow DAG is failing silently. Are you running an ETL on a huge dataset? This is a symptom of an Airflow instance without sufficient memory. Dig into your instances logs and you’ll probably see an evicted worker if your running your instance’s workers on Kubernetes. You’ll see similar logs wherever you run …
Have you used Functional Annotations in Python 3.x? Maybe you’ve heard them mentioned? Regardless, let’s explore what they are and how they help us. Because if they don’t help us, then we probably shouldn’t care. The Problem You’re programming and don’t know what thisRandomFunction should return. Maybe it’s a bool maybe it’s a string who …
When Airbnb was scaling rapidly, they faced the problem of organizing complex data pipelines. To combat this and become a data-driven organization, Airbnb launched Apache Airflow in 2015, their custom-made open-source platform to manage complex workflows. In simple words, Apache Airflow is a platform where you can create, schedule, and monitor complex workflows using simple …
Let’s work on manipulating a dataframe. Let’s work on editing all elements of a DataFrame according to a condition. You have a horde of data you just imported from a CSV or an Excel doc. You’ve managed to get the data into a Pandas DataFrame using one of the built in import methods like read_csv …
So you’re trying to upload a Pandas DataFrame to DynamoDB using Python? Let’s take a step back first. Why are we using DynamoDB? What is DynamoDB? What is DyanmoDB DyanmoDB is a non-relational fully managed database product offered by Amazon’s cloud computing arm AWS. So why would you go the DynamoDB route vs MySQL, Postgres, …
Hey everybody! I thought I’d do a quick intro on an app I started working on. Drum roll please… Moomosa! It’s a React-Native application that uses Firebase as a database at the moment (more on this later). That database is managed with a React application I developed to add, delete, and approve new content. I …
Soooo you got the IndexError: only integers, slices (`:`), ellipsis (`…`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices error. Let’s get that annoying message fixed. My Computer Setup Python3 macOS For those Short On Time You’re trying to access an element of a Pandas index like df.columns where df is a DataFrame …