Alert options for your data pipelines
Helen Dockrell, Notitia's Adelaide-based Director, runs us through how to set up alerts and monitoring systems for data workflows.
Data workflow alerts and monitoring systems
You’ve put in the hard work to automate a data workflow, and it’s working like a dream. But a few months later, you receive a haunting email from an end-user, asking why the last data point available is three weeks old (instead of 3 hours).
Queue cold sweats and rapid data pipeline debugging.
Six hours in, you find the spanner in the works - an unanticipated additional field in a data source. This has impacted all reports generated in the past month. And now there’s a significant workload created by the resulting damage control, stakeholder apologies and communications with various reporting bodies.
In hindsight this could have been a much easier and quicker fix, simply by having a monitoring system in place to notify you, in real time, when that unexpected bug occurs.
While it’s straightforward, in concept, to set up alerts and monitoring systems for these workflows, timelines are often constructed without taking these steps into account.
Let’s run through some simple and reliable options for those of us in a time crunch.
Option 1: No-code data pipeline alerts
This type of alert is built into reporting software, and allows you to select from a list of pre-configured options to create your alert.
Pros: They’re fast to implement, require no coding knowledge, are consistent and reliable.
Cons: You may find alert conditions that you cannot access using this interface, as it is created to cater to a standardised range of use cases. Similarly, if you have specific alert recipient requirements these may not be configurable options.
You can find these in ETL tools including Qlik Sense, Power BI and Power Automate.
Option 2: Fully customised data pipeline alerts
These are alerts that are coded individually to send an email or text message when specific conditions occur.
Pros: You can define an incredibly specific condition, your data and imagination are your only limitations! You will never receive an alert that is irrelevant (unless you want to).
Cons: You’ll have to code this one yourself, so there’s a bit more time involved in testing that your logic results in the intended outcome.
Keep this within your core software stack to minimise development time and solution fragility. A reliable and repeatable way to set these up is to use a scheduled log or data-evaluation script connected to an emailing or messaging API service.
Some examples of scheduled evaluation scripts are Runbooks within Azure Automation Services, or a Qlik Sense Data Load Editor file. You might connect these to a secure email or texting API service like Twilio or SendGrid to send your precise, customised alerts.
Happy data crunching!
About Helen Dockrell, Notitia Director
Notitia Director, Helen is a software developer who has worked in both industry and research environments to develop tools to conceptualise complex systems.
Always up to the challenge, Helen has a proven track record in developing ways that empower her clients through improved access to data for informed decision making.
Helen is a highly creative person, which fires her passion for problem solving and out-of-the-box thinking.
Leading Notitia's Adelaide office, Helen works with clients both in-person and remotely across Australia.
Her educational background in the sciences includes a degree and post graduate in molecular biology, biochemistry, genetics and computer science.
> Book time in with Helen to find out how she can help solve your problems through data.
