Antony Denyer

Tilling the land of software

about me

I am passionate about creating great software and am particularly interested in associated supporting methodologies, namely XP and TDD.

recent public projects

Status updating…

found on

contact at

email@antonydenyer.co.uk

The Value of Functions

- - posted in aws, azure functions, faas, lambda, serverless | Comments

With the emergence of FaaS entering the mainstream, we are starting to see some real world business value in using such an architecture. Many of the benefits of using FaaS are discussed elsewhere:

  • pay as you go
  • no-ops
  • limited provisioning
  • performance scalability
  • lower running costs

But some of the benefits are more subtle. The benefit I want to highlight is this:

You can measure how much features cost to run.

While this may not seem like a winning feature, let’s think about it for a second.
You have a feature on your website called ‘price match’. What it does is matches prices against your competitor’s prices. To do this it scrapes each of your competitor’s sites to get their price of the product. If you ask the product owner how often they want the prices updated they’d probably say something like ‘as much as possible’ or ‘real time please’. In the old world, we would probably hit it as often as we could and be done with.

Now with FaaS you have some additional information to help you make decisions, this feature will cost 0.001ยข to run per competitor per product. You can then extrapolate different price points for different frequencies and make a business case for each of them. Or better yet measure the value of having it run every minute vs. every hour.

Summary

As an industry, we can genuinely begin looking at the running costs of an individual feature. This started at the product level with ‘Lean Startup’ and will, I believe, continue to a greater degree with individual features and functions.

Features can be added and removed more quickly with the costs and benefits of them being more transparent to both developers and product owners.

The Novel Developer

- - posted in developers | Comments

The trend of re-inventing the wheel is on the rise. As new developers enter the industry, there appears to be a tendency to prefer the novel over the stable. I’ve seen developers favour one library over another simply by the number of commits or by how active it is. While I understand the desire to use libraries that are ‘actively’ maintained I do not understand the desire to use a library that is constantly changing. I would expect most developers would want their code to change more quickly than the libraries they are dependent upon. It struck me; they’re not in search of solutions or a way to add value, they’re in search of novelty.

Daily Mail Developer

They’re in search of constant titillation and ‘gossip’. They need to be using something new to feel like they’re innovating. Using a different library or the latest js build tool is not innovative, at best it’s novel at worst it’s naive.

More Yak Shaving

We could be solving real world problems, business or otherwise. Instead, we’re moving from bower to grunt or gulp or whatever.

Beware the novel developer

ElasticSearch Schema Migrations With Zero Downtime

- - posted in elasticsearch, re-indexing, re-ingestion, schema mirgation | Comments

One of the frustrations of working with elasticsearch is not being able to make changes to your mappings and indexes without being destructive. You’d never use elasticsearch as your source of truth, so this isn’t normally a huge problem. But it can become a bit of a headache. The general advice tends to be ‘make your changes and then re-index everything again.’ Done wrong this can mean your search has no results while you re-index your data.

How do you re-index data?

There are two ways of re-indexing your data.

  • re-index data from your source of truth i.e re-ingest everything
  • re-index data into a new index from the old index

What we’re going to explore is how we re-index data into a new index from the old. There are some recommended practices in https://www.elastic.co/guide/en/elasticsearch/guide/current/index-aliases.html that we’re going to be expanding upon.

Architecture

The sort of system we will be talking about is one that follows Command Query Separation. Essentially you have a system that reads from the elasticsearch and a separate system that writes to elasticsearch. Something like a back end catalogue editor and a front end e-commerce site.

Use Aliases

Lets say you have an index called catalogue-v1 you’ll need to create two alliases that point to this index, catalogue-read and catalogue-write. Change your application so that your front end application queies from catalogue-read and your back office points to catalogue-write.

When you have a breaking change to your schema you can create a new index catalogue-v2 then modify the catalogue-write alias to point to the new index and the old one.

Migration

If you’ve changed a data type or an analyzer you can tell elasticsearch to re-index everything using the re-index api. Underneath the hood, it uses scroll api and the bulk api to push the data into a new index. If you need to change the data as you re-index you should use the scroll api and the bulk api. When you’re happy with the new index you can modify the catalogue-read alias to point to catalogue-v2.

Then all you have to do is clean up the aliases and delete the old index.