Queryable State or How to Build a Billing System Without a Database
Together with Konstantin Knauf.
Traditionally, big data applications rely on the Lambda Architecture in order to achieve low latency as well as completeness. A streaming layer provides real-time previews while a complementary batch layer retrospectively recomputes the correct results. Using a robust stream processor like Apache Flink, we can do without the latter. But can we take it even one step further? This talk will discuss one of the upcoming features of Apache Flink with the potential to do just that.
As a real-world example we have built a prototype for a robust billing system based on Flink and Queryable State. On the one hand, the system exposes the current monthly subtotals in real-time to front-end applications, on the other hand it reports the complete results to downstream systems, e.g. for invoicing. As completeness and correctness are core requirements for a billing system, we will demonstrate the system in multiple failure scenarios, including taskmanager and jobmanager failures as well as unavailability of downstream systems.
This talk will give you an idea of how “Queryable State” combined with a robust stream processor enables new streaming use cases and changes the future of streaming application architecture.