Large enterprises never have just one or two data sources. It’s always tens or hundreds of places they need to pull data from, if not thousands. What may have started out as a couple of shell scripts in the early days of a company, may have turned into a crontab of tasks, which morphs into a spider-web of cron-dependencies, and eventually you’ve got program management sniffing down your department’s neck saying “Why does this ETL process keep failing, and costing us money?”. …
I’ve been working in software development since 2015, and have worked with lots of teams, and all of them have gone through phases of what they called “agile” at one point. But the experiences of this process have varied wildly, and I’ve rarely seen teams that would consider themselves hyper-productive, as some agile teams describe. I recently read Learning Agile, and got a whole different worldview from what I had experienced. Here’s how I was introduced, and why a lot of it wasn’t done right.
- First experience with agile, scrum, kanban
- key takeaways from the book “Learning Agile”
- a lot…
- Why
- Estimates, Targets, and Commitments.
- We’re all bad at estimating
- Agile business, rigid estimates
- An estimate comes with uncertainty
I’ve been on a few projects that came in on time, and far more that didn’t. When I reflect critically on this, I’ve definitely made my fair share of mistakes, but somehow I never felt at any given time that I was making a mistake while I was doing it. I wanted to dive deeper to understand the reasonings behind my past mistakes, so I recently read Steve McConnell’s Software Estimation: Demystifying the Black Art. …
I recently did a deep dive on computer networking and the internet, and I found some of these concepts to be really interesting. If you’ve been a developer for a while, but never really looked into the networking stack, I’d really encourage you to spend a little time hoking around to see how things work. Here are some of the more interesting concepts I really enjoyed:
- Internet Architecture
- Addresses — CIDR
- What’s IPv6?
- DNS
Internet Architecture
How does the internet actually work? Admittedly, I was totally ignorant to how information moves around, to me it was just “well… it’s all connected, and…
- Motivations & Goals
- Progress thus far
- What I found to be useful techniques for progress
- Future plans
Motivations and Goals
I’m getting older. Fast approaching the 30 milestone — 30 years old, and about 30 kg heavier than I was at the start of my twenties 😂. Last year was quite successful for my professional development, I led a huge project and mentored junior developers, gained 3 GCP certifications, and switched jobs.
I normally start to wind down my personal efforts for the year around 10th December, just before my birthday and Christmas break, and this normally gives me some time to reflect…
- Why split tables
- date sharded tables
- partitioned tables
- when to use which
- what about clustering?
When database tables become very large, querying them eventually leads to one of 2 problems — it becomes quite difficult, as your database has a lot of data to scan and aggregate, or if you have a very scalable database, it becomes very expensive!
In order to combat this, databases offer an array of useful features —typically, an RDBMS will allow indexes to quickly find data, partitions to physically split data into multiple tables internally, and multi-dimensional clustering to allow sorting the data in a…
- Some backstory of my dev experience
- Patterns I noticed in my software
- Steps to improve
- Robert Martin’s Clean Code
- Martin Fowler’s Refactoring
I entered software engineering in a less common fashion, I started working in the industry as a data analyst writing SQL. After 4 months on the job, I realised I could have the most impact by filling in a gap that few others seemed to be filling around me, by developing my technical skills with SQL. …
- What’s the cloud confusion?
- What’s cloud-native?
- Which definition should your company aim towards?
A few years ago, I was getting some help from a mentor who worked in one of my company’s infrastructure teams, around some work we were doing on our on-premise private cloud. At one point we had to stop and ask each other the question, “What do you think the cloud is?”
He said “it’s elastic computing infrastructure”. He was a lot more experienced than me, so I just shut up and looked funny, until he asked me what I thought the cloud meant. I said “it’s…
If you have a product as popular as LiveScore, with 50M monthly users, you soon accumulate a LOT of data about those users. BigQuery is practically limitless in terms of what you can do with the serverless warehouse — but it’s not a charity. Here are some missteps we made along the way, and how we eventually optimised things
As a huge GCP user, it will come as no surprise that I’ve spent my fair share of time working with BigQuery and dealing with large datasets. But when livescore began recording so much data every month, our costs started to…
During a recent migration, I seized an opportunity to have permissions for our workspace closer to the team, to reduce bottlenecks and keep the team moving quickly.
We moving from an on-premise github instance to github’s managed offering, there will almost always be someone from your IT department tasked with enterprise security and Active Directory integrations and whatnot. This is a fairly tedious and likely thankless job.
A typical setup might be as follows:
- someone does some analysis to work out what AD groups ought to exist, and creates them. This is often painful, as the person provisioning those groups…
