Skip to content
Work Expertise Services CV Portfolio Integrations Laravel packages Blog Sign in Get in touch

Knowing it is broken before the customer calls

Most outages are reported by the person suffering from them, and that is always later than the moment they started. What sits in between is not luck but a choice: what you measure, when you hear about it, and how much context comes with it.

What you actually want to know

There are three kinds of question, and all three ask for something different. The first is whether something is broken: an exception, a request returning an error code, a job that never finishes. The second is whether something has become slow, which is harder because nothing fails and the complaint stays vague. The third is whether anyone came by, and what they did.

The first two belong together and come from the application itself: errors, logs and traces. The third is different in kind and belongs with visitor numbers. That looks like a detail, but it decides where your data ends up: a visitor counter knows who your customers are, an error collector knows what happens inside your application. You need to hand over neither.

What I built on it

Six pieces between an error occurring and someone doing something about it.

Errors from your own apps in one place

Bugtail collects errors, logs and traces from several applications and bundles them into issues you can solve, instead of a stream of separate messages. Self-hosted, so your customers stack traces stay on your own server.

The sdk you already use

Bugtail speaks the Sentry and OpenTelemetry sdks. Switching is therefore changing an address, not a rewrite, and you are not locked in: the same code works with another receiver too.

Not the same error a thousand times

Messages are grouped on their fingerprint into one issue with a counter. An outage that occurs a thousand times is then a line saying a thousand, and not an inbox silting up exactly when you need the overview most.

Visitor numbers without resale

Matomo on your own address rather than a script that takes your visitors along to an advertising market. You see what you need, your visitor does not become a product, and your processing register stays short.

An alert in the channel you are in

A webhook to Slack or Teams when something goes wrong, with enough context to know whether it can wait. Anyone who has to open an alert email to see it was nothing stops reading them after a week.

An issue for the developers

What comes in at customer service and turns out to be a bug becomes an issue in Jira with the details of the case attached, and the status comes back through a webhook. That way nobody has to go and ask how far it is.

How it is put together

Five choices that decide whether an alert channel is still read after a month.

An alert without context is just noise

That there was an error is the least useful half. Which user, which page, which input, which version, which device: that is what makes the difference between ten minutes and half a day. You collect that context as it happens, because reconstructing it afterwards does not work.

Measuring must not slow the application down

What you collect goes through a queue and not through the visitor request. If the receiver is down, your application should simply keep running. A measuring system that can take your site down is a risk you bought on top, instead of removing one.

Not everything you can log may be kept

A stack trace regularly holds an email address, a postal address or a complete request with a token in it. You put that filter on the application side, before it leaves, and not on the receiver side. Otherwise your error log has become a data collection without anyone deciding so.

A threshold you keep beats one that is exactly right

An alert rule that fires twenty times a week is clicked away after two weeks. Better to alert less and take those alerts seriously. What you want to know but need not be woken for belongs in an overview you look at once a day.

Traces are for when nothing is broken

An exception is easy: it comes with a line number. The hard one is the page that became slow without anything failing. For that you need traces, with the time per step, and you are best off setting them up before you need them. Switching them on afterwards means waiting for it to happen again.

Do you only notice outages when someone calls

Then half a day of work usually already buys a lot: the sdk in, a channel attached, and a threshold low enough to catch something and high enough to stay read. Send a message.

Bugtail is ordered and managed here on deboeck.dev, the helpdesk side under the FreeScout integration, the route to the developers under Jira, and what you do when it really goes wrong under continuity and recovery.