If you work in even a medium sized organisation, you’ve probably heard the term “kubernetes cluster” thrown around. Invariably, you’ll have more than one. But at some point, someone will come along, likely a developer with a need, or a new project with big ideas, and boldly proclaim “We need our own cluster for this”. How should you reasonably decide whether to grant their wish, or force them to use an existing cluster?

  • What does the natural evolution look like in enterprises?
  • What happens at first when someone asks for a new cluster? …

Large enterprises never have just one or two data sources. It’s always tens or hundreds of places they need to pull data from, if not thousands. What may have started out as a couple of shell scripts in the early days of a company, may have turned into a crontab of tasks, which morphs into a spider-web of cron-dependencies, and eventually you’ve got program management sniffing down your department’s neck saying “Why does this ETL process keep failing, and costing us money?”. …

I’ve been working in software development since 2015, and have worked with lots of teams, and all of them have gone through phases of what they called “agile” at one point. But the experiences of this process have varied wildly, and I’ve rarely seen teams that would consider themselves hyper-productive, as some agile teams describe. I recently read Learning Agile, and got a whole different worldview from what I had experienced. Here’s how I was introduced, and why a lot of it wasn’t done right.

  • key takeaways from the book “Learning Agile”
  • a lot…

  • Estimates, Targets, and Commitments.
  • We’re all bad at estimating
  • Agile business, rigid estimates
  • An estimate comes with uncertainty

I’ve been on a few projects that came in on time, and far more that didn’t. When I reflect critically on this, I’ve definitely made my fair share of mistakes, but somehow I never felt at any given time that I was making a mistake while I was doing it. I wanted to dive deeper to understand the reasonings behind my past mistakes, so I recently read Steve McConnell’s Software Estimation: Demystifying the Black Art. …

I recently did a deep dive on computer networking and the internet, and I found some of these concepts to be really interesting. If you’ve been a developer for a while, but never really looked into the networking stack, I’d really encourage you to spend a little time hoking around to see how things work. Here are some of the more interesting concepts I really enjoyed:

  • Addresses — CIDR
  • What’s IPv6?
  • DNS

Internet Architecture

How does the internet actually work? Admittedly, I was totally ignorant to how information moves around, to me it was just “well… it’s all connected, and…

  • Progress thus far
  • What I found to be useful techniques for progress
  • Future plans

Motivations and Goals

I’m getting older. Fast approaching the 30 milestone — 30 years old, and about 30 kg heavier than I was at the start of my twenties 😂. Last year was quite successful for my professional development, I led a huge project and mentored junior developers, gained 3 GCP certifications, and switched jobs.

I normally start to wind down my personal efforts for the year around 10th December, just before my birthday and Christmas break, and this normally gives me some time to reflect…

  • date sharded tables
  • partitioned tables
  • when to use which
  • what about clustering?

When database tables become very large, querying them eventually leads to one of 2 problems — it becomes quite difficult, as your database has a lot of data to scan and aggregate, or if you have a very scalable database, it becomes very expensive!

In order to combat this, databases offer an array of useful features —typically, an RDBMS will allow indexes to quickly find data, partitions to physically split data into multiple tables internally, and multi-dimensional clustering to allow sorting the data in a…

  • Patterns I noticed in my software
  • Steps to improve
  • Robert Martin’s Clean Code
  • Martin Fowler’s Refactoring

I entered software engineering in a less common fashion, I started working in the industry as a data analyst writing SQL. After 4 months on the job, I realised I could have the most impact by filling in a gap that few others seemed to be filling around me, by developing my technical skills with SQL. …

  • What’s cloud-native?
  • Which definition should your company aim towards?

A few years ago, I was getting some help from a mentor who worked in one of my company’s infrastructure teams, around some work we were doing on our on-premise private cloud. At one point we had to stop and ask each other the question, “What do you think the cloud is?”

Spenser Confidential — Mark Whalberg is a cop, and tries to get the CCTV from the back office of a shop, but the CCTV is “in the cloud”.

He said “it’s elastic computing infrastructure”. He was a lot more experienced than me, so I just shut up and looked funny, until he asked me what I thought the cloud meant. I said “it’s…

If you have a product as popular as LiveScore, with 50M monthly users, you soon accumulate a LOT of data about those users. BigQuery is practically limitless in terms of what you can do with the serverless warehouse — but it’s not a charity. Here are some missteps we made along the way, and how we eventually optimised things

As a huge GCP user, it will come as no surprise that I’ve spent my fair share of time working with BigQuery and dealing with large datasets. But when livescore began recording so much data every month, our costs started to…

Mark McCracken

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store