Monit Automated Backups Failure Notification and Checking Your Backups

3 min read

Introduction

If an automated backup fails, it will be logged inside the Automated Backups Monitoring log. Monit checks this log once a day and any failures have been logged it will send you a notification.

It’s important to note that far more often that not this is a one-off event, and it does not necessarily mean that all of your backups are failing. This is easy to check and confirm, and if it is a one-off failure, the notice inside Monit will disappear the next time it checks the log.

The Automated Backups Monitoring Log

The Automated Backups Log can be found inside your server customizer.

Head over to the Servers page inside your account and click on the name of the server you wish to check. This will open up the customizer:

The log is located inside the Logs tab.

Click the icon to open up the log. Logs are for 24 hours and at the bottom you can also navigate between previous days if you wish to do so.

On the server the log is located here:

/opt/gridpane/backups.monitoring.log

You can learn more about site and server logs and where they live in this article:

GridPane Logs – How to Find Them and When to Use Them

Reviewing the Log

If reviewing the log in your browser, once the log is open you can use CTRL+F to open up a search and type the word “Fail“. This will make it easy to locate any failures without scanning the entire log line by line.

If no failures exist today, then this was likely a one off event and Monit will report all is well the next time it checks the log.

Quickly Check for Failures Over SSH

To quickly and easily review the log for failures, you can run one of the following commands on your server (depending on whether you’re using the new V2 backups system (Duplicacy), or the previous V1 backups system (Borg).

Checking V2 Backups

Run the following command to check today’s log for failures:

grep -A 4 ".*Duplicacy.*{1}" /opt/gridpane/backups.monitoring.log

Checking V1 Backups

grep -A 4 ".*Borg.*{1}" /opt/gridpane/backups.monitoring.log

What These Commands do

For both backup systems the {1} indicates a failure has occurred. Specifically:

  • -A 4 means show 4 lines after
  • .* is a wildcard
  • {1} is a fail response.

If nothing is returned, no failures have occurred today. If a failure has occurred it will output the failure line as well as the following 4 lines that will offer insight into what went wrong.

For example:

Thu-Jul--1-01:00:11-EDT-2021 | example.com | Duplicacy Fail {1}
Unable to push full manual backup to Local... storage threshold exceeded...
Automated Local Backup Failed!
Initiating full automated backup to Local...

If this is the first time connecting to your server over SSH, please see the following articles to get started:

What Causes Backup Failures?

Failures can occur for a variety of reasons including:

  • High CPU load at the time the backup is attempted
  • MySQL is too busy or has long running queries at the time the backup is attempted and the database cannot be exported
  • MySQL is experiencing prolonged table locking
  • Critical errors within the website
  • Database corruption
  • The allocated disk space has been exceeded
  • A backup already exists that time period
  • There’s an actual issue with the backups system on the server

More often than not the reason for the failure will be obvious inside the log.

No backups system can be 100% perfect all the time, so an odd failure from time to time is possible. If you believe that your backups system is broken, please contact support.