Anyone tempted to write a custom database should ask if they’re prepared to be in the database maintenance business.
Because that’s where they’re heading, for a really long time.
we’ve replaced our $10k/month Aurora instances with a $200/month Elastic Block Storage (EBS) volume.
Lol, i’m sure just refactoring it and using a different service could have lowered that price significantly.
The description of their data says “relational database” to me, but don’t murder me for using such an unpopular phrase!
They never would have been able to get the same performance from any solution that incorporates a general purpose database.
Their requirements/explicitly-not-required-ments include that it’s fine to drop 1s of data. That would be an insane proposition for any other database. Also their read/write rates and latency requirements are unusual to say the least.
It’s the same thing as tiger beetle. Ridiculously narrow domains allow for ridiculous performance improvements compared to of-the-shelf solutions.
What is tiger beetle?
A new database specifically designed for financial transactions.
I’m not an expert on finance software, so I can’t critically assert how good they really are. But they claim much much higher throughput than traditional databases, higher fault tolerance, self healing networks if several replicas are running, etc.
From a purely technical standpoint it’s interesting for being written in zig. Because the database scope is so narrow they know exactly how much memory they will need on startup and just allocate all required memory on startup and never allocate more, nor free the aquired memory.
Aurora is a relational database
Cool
Removed by mod
Never heard of this. Thanks for mentioning it!
Comparing cost to AWS Aurora is unfair. Give us the self host price, and compare to that.
Also, they should have tried Scylla or Cassandra. It’s very scalable and handles a lot of writes.
Why does everyone on the planet hate sql nowadays?
If your not dealing with billions of transactions why not?
Maybe it reminds them of Oracle.
What has changed though, is that we’ve replaced our $10k/month Aurora instances with a $200/month Elastic Block Storage (EBS) volume.
Holy shit, I hope whoever wrote this gets a fat bonus at the end of the year. That’s a truly astounding savings.
What they don’t say is how much in developer time they’ve spent rolling their own database. Then there’s also maintenance and new features.
It’s called job security, bro.
I’m sure their custom database will be easy to find people to support and maintain
It seems like features would be mainly additional key/values like temperature, humidity, etc. This wouldn’t really change the underlying infrastructure greatly but still give good enhancements to certain customers.
Aren’t time series databases like Prometheus pretty good at storing this kind of data?
I don’t believe Prometheus supports geospatial data. Two minutes of googling though, so I could be wrong.
They’re just storing doubles in their own format too. I’m not sure if they even need any spatial lookups on the data, they didn’t mention anything about that in the article. Maybe they do that in-memory?
I imagine a delta encoding scheme similar to what the time series DBs use would work well for the geo points. Maybe even a delta-of-delta encoding for things like ships which move very consistently.
It’s probably not worth it given how small they’ve already go their data. But it is fun.
If you’re using Java you can use eclipsestore. Seems like a good project