Chapter 1: A New Year in Data Engineering

January often symbolizes both a fresh start and a culmination of efforts for many. This month tends to see a surge in self-improvement content, as the wave of New Year's resolutions serves as the self-help industry's peak season. Yet, amidst the chaos of deadlines and personal commitments, one truth stands out: January is a hectic month.

In the realm of technology and data, this busyness is no different. While many view January as a time for new beginnings, in corporate environments, it marks the onset of Q1. This period is filled with brainstorming sessions, individual goal-setting meetings, and, for those who plan well, a roadmap of initiatives designed to be tackled throughout the year.

As organizations adapt their strategies, they also look to implement new technical solutions. Notably, by the close of January, Google is set to phase out cloud functions that utilize Python 3.8.

Photo by Mika Baumeister on Unsplash

In mid-2023, a significant update from Pandas caught many users off guard as it deprecated a widely-used method. The change raises questions about how to adapt to new features and functions.

Chapter 2: Embracing Changes in Pandas

One of my go-to functions in Pandas has always been the .append() method. I recognize it may sound peculiar to have a favorite function, but for someone who frequently works with APIs that return data in various formats, .append() has been an invaluable tool. It provides a straightforward way to create a DataFrame from a list of dictionaries found in JSON payloads, a common scenario in API-driven data sources.

Due to the frequency with which I use this method, it has become second nature to me. Thus, I was taken aback when a routine upgrade of a cloud function revealed that my trusted method was no longer reliable.

The error related to the DataFrame is just one of many challenges I (and likely many of you) faced over the past year. Instead of following the trend of “Best of (year),” I prefer to reflect on my most obscure and sometimes embarrassing errors.

In 2023, I encountered challenges ranging from Python to SQL to Airflow. I documented some of the most notable errors, shared how I resolved them, and provided insights to help others avoid similar pitfalls.

Chapter 3: Learning from Mistakes

Mistakes in the workplace should not be feared. In a recent discussion with an executive, we concluded that errors are a natural part of the process; however, recurring mistakes warrant scrutiny.

Junior team members often face the steepest learning curve, and those in senior roles, like myself, bear the responsibility for guiding them. A supportive senior colleague fosters an environment where juniors can thrive, yet we also expect some level of initiative and responsibility.

A couple of years ago, I shared advice on how to avoid frustrating senior colleagues from the perspective of a junior developer. As I revisit this topic in January 2024, I aim to offer insights that might benefit both junior and senior engineers alike.

Chapter 4: Insights for Junior Developers

In the tech world, a senior role might not carry the same implications as in other fields. It often represents a step up in the hierarchy rather than a dramatic increase in authority. However, my experiences working with stakeholders, technical partners, and junior team members have equipped me with valuable insights.

This is not merely a space for airing grievances; instead, I intend to share proactive strategies that can help junior developers become assets to their teams more quickly.

For these insights and more than 200 other resources related to the data industry, follow Pipeline.