Add ADR052 on storing analytics in Aurora - #263
theseanything wants to merge 1 commit into
Conversation
| - Table schemas are defined in Terraform, but the provider does not support partitioning yet (not a big concern at our scale). | ||
| - Athena queries take seconds, too slow for the request path. Showing per-form statistics in forms-admin would need a faster way to query. | ||
|
|
||
| ### Option B: a separate Aurora PostgreSQL cluster for analytics |
There was a problem hiding this comment.
Think this is my preferred option atm - as whilst it's more expensive, it's a comparatively small infra cost. We are unlikely to run into performance issues at our scale and is more flexible.
There was a problem hiding this comment.
Do we have a sense of how other teams do it?
There was a problem hiding this comment.
GOV.UK Notify, store their notifications sent in a db. Haven't heard from Pay or anyone else
There was a problem hiding this comment.
Thank you.
Do we know if they have any issues/wish they had done it differently? Have they struggled with performance
There was a problem hiding this comment.
Possibly, considering the outlined use case is our own internal KPIs and at such a small scale - I'd like to take the risk and get something implemented sooner, rather than waiting to hear back from them.
| - Unfamiliar services: S3 Tables, Firehose and Athena. | ||
| - Firehose delivery is at-least-once, so queries may need de-duplication or the transform needs an idempotency key. | ||
| - Table schemas are defined in Terraform, but the provider does not support partitioning yet (not a big concern at our scale). | ||
| - Athena queries take seconds, too slow for the request path. Showing per-form statistics in forms-admin would need a faster way to query. |
There was a problem hiding this comment.
There's a reasonable demand for funnel analytics from form creators, and we've been talking about adding metrics for save and return.
I think it's a reasonable assumption that certain metrics/analytics we make available to form creators we'll also want to use for internal performance analytics.
So I think a solution that works for both purposes seems best, otherwise we have to maintain emitting metrics from forms-runner in 2 different ways.
There was a problem hiding this comment.
I agree.
I think it's worth adding our requirements for these to the context. In the context of KPIs this last point wouldn't be a so much of a con.
There was a problem hiding this comment.
I've updated the context - made the proposal for using a separate Aurora DB
There was a problem hiding this comment.
I'm still very uncertain, not totally certain an Aurora cluster is the right call.
Our actual requirement is so small: submission events and the number of forms over time, for an internal dashboard. Future analytics like funnels are still unclear and I have doubts our own database is the right home for them anyway, since they need to track users across interactions. That's better served by GA or a purpose-built product analytics tool.
Against that, an always-on cluster per environment, plus upgrades, backups and credentials, feels like a lot of cost (it'll be like $1000 - $1500 all up vs less than $50) + maintenance. The S3 Tables prototype is essentially free, covers the use case we understand today, and would be much quicker in getting us going with our KPIs.
Now I'm leaning towards starting with S3 Tables and revisiting if we outgrow it. Keen to hear if anyone sees it differently.
There was a problem hiding this comment.
I think there's a distinction between adding a new Aurora cluster and adding a new database to existing clusters. The cost of the latter is much lower.
From the proposal I assume you meant a cluster, but have we considered adding a database to an existing cluster?
There was a problem hiding this comment.
Yes I did mean cluster - (tho I've interchangeable used Aurora DB). I think what you're suggesting is have we considered the cost of writing the metrics to the existing cluster (logically separated in someway).
From the previous comment thread - my concern is the complexity around managing access to the sensitive production data. I.e. you want to Grafana to only have read access to the metric tables and nothing else.
If I've misunderstood, or maybe its easier than I realise - might be helpful if you could describe to how exactly you would set that up?
268c86c to
f75b3f2
Compare
theseanything
changed the title
We want to measure submissions over time and the number of live forms, keeping the data for at least two years. Existing tools (OTel and Classic CloudWatch metrics, GA4, Splunk, Logs Insights, the runner database) fall short on accuracy, cost, cardinality, retention or ownership. S3 Tables queried with Athena was prototyped and is very cheap, but relies on unfamiliar services, at-least-once delivery and slow queries. A separate Aurora PostgreSQL cluster is familiar, gives exactly-once writes and low-latency queries, and can later serve per-form analytics in forms-admin. This record accepts Aurora and sets out the trade-offs.
f75b3f2 to
62858f1
Compare
| - It is familiar. We already run Aurora PostgreSQL, so there are no new services to learn and we can reuse existing Terraform, backup and access patterns. | ||
| - Writes are transactional and exactly-once, with no log parsing or de-duplication, which suits data where accuracy matters. | ||
| - Indexed rows give low-latency queries, so the same store can later serve per-form analytics inside forms-admin. | ||
| - It is isolated from production databases, so analytics queries cannot affect production performance. |
There was a problem hiding this comment.
Do we know how big of a problem this actually is?
I think we have relatively low traffic. I'm wondering whether we could make it work with the existing clusters. Is it worth considering adding read replicas to deal with any performance concerns? I believe this is cheaper than adding a new cluster
There was a problem hiding this comment.
So I think - you still pay the same ACU costs, but uses the existing storage (tho that is negligible anyways) - however, we might be able to allow the read replica scale down to zero (but the long start up times, wouldn't be suitable prod)?
I guess the other thing is how we limit data access, because it'd be the forms-runner db with the submission data - which would require creating another (Postgres) Role in the DB. You could do it as a Rails migration? but kinda messy, you need to also get the password into Secrets Manager - and co-ordinate when the sequence of events.
Not convinced the complexity of keeping the data secure is worth the cost savings.
There was a problem hiding this comment.
I had assumed we'd create a separate database within the same cluster. So that the isolation is provide at that level. I think the access is managed per database, rather than per cluster
But I might be getting confused about how that works 🤔
There was a problem hiding this comment.
So you can create another logical "database" in postgres cluster - but you still need to manage permissions to that inside postgres (not something you can do with IAM). Within the physical cluster - you can create a "read replica", but that accesses the same underlying data (with all the logical databases).
There was a problem hiding this comment.
Could we definitely not go with the option of storing the metrics in the admin db, using that data to drive metrics we provide to form creators (instead of cloudwatch), and offer a simple dashboard and download of these stats for internal purposes?
3 participants
We want to measure submissions over time and the number of live forms, keeping the data for at least two years.
This proposes storing those metrics in a separate Aurora DB.
Existing tools (OTel and Classic CloudWatch metrics, GA4, Splunk, Logs Insights, the runner database) fall short on accuracy, cost, cardinality, retention or ownership.