Merge pull request #1325 from BastilleBSD/bastille-monitor

This commit is contained in:
tschettervictor
2025-12-12 08:39:41 -07:00
committed by GitHub
10 changed files with 450 additions and 1 deletions

View File

@@ -104,6 +104,23 @@ This is the default `bastille.conf` file.
bastille_template_vnet="default/vnet" ## default: "default/vnet"
bastille_template_vlan="default/vlan" ## default: "default/vlan"
## Monitoring
bastille_monitor_cron_path="/usr/local/etc/cron.d/bastille-monitor" ## default: "/usr/local/etc/cron.d/bastille-monitor"
bastille_monitor_cron="*/5 * * * * root /usr/local/bin/bastille monitor ALL >/dev/null 2>&1" ## default: "*/5 * * * * root /usr/local/bin/bastille monitor ALL >/dev/null 2>&1"
bastille_monitor_logfile="${bastille_logsdir}/monitor.log" ## default: "${bastille_logsdir}/monitor.log"
bastille_monitor_healthchecks="" ## default: ""
Notes
-----
The options here are fairly self-explanitory, but there are some things to note.
* Bastille will mount the dataset it creates at ``bastille_prefix`` which
defaults to ``/usr/local/bastille``. So if you want to navigate to your jails,
you will use the ``bastille_prefix`` as the location because this is where the
will be mounted.
Custom Configuration
--------------------

View File

@@ -23,6 +23,7 @@ Bastille sub-commands
limits
list
migrate
monitor
mount
network
pkg

View File

@@ -0,0 +1,81 @@
monitor
=======
NEW in Bastille version 1.1.20250814
The ``monitor`` sub-command adds, removes, lists and enables/disables monitoring for container services.
Managing Bastille Monitor
-------------------------
To enable Bastille monitoring, run ``bastille monitor enable``.
To disable Bastille monitoring, run ``bastille monitor disable``.
We can always check if Bastille monitoring is active with ``bastille monitor status``.
Managing Services
-----------------
Bastille Monitor will attempt to monitor any services defined for any given container. If the service is
stopped, Bastille will attempt to restart it. Everything is logged in ``${bastille_monitor_logfile}``.
To have Bastille monitor a service, run ``bastille monitor TARGET add SERVICE``. The ``SERVICE`` arg can also be a
comma-separated list of services such as ``bastille monitor TARGET add SERVICE1,SERVICE2``.
To remove a service from monitoring, we can run ``bastille monitor TARGET delete SERVICE``. These can also be a
comma-separated list.
To show all services that Bastille is monitoring, run ``bastille monitor TARGET list``.
To list all jails that have a selected service defined for monitoring, run ``bastille monitor TARGET list SERVICE``.
This option only accepts a single ``SERVICE``, and cannot be a comma-separated list.
If you run ``bastille monitor TARGET``, without any args or actions, Bastille will run through the process of
checking the status of each defined service, and attempt to start any that are stopped.
Services can also be manually added or removed by editing the ``monitor`` file inside the jail directory, but
is not recommended unless you are an advanced user.
Configuration
-------------
The monitor sub-command is configurable via the ``bastille.conf`` file. See below
for configuration defaults:
.. code-block:: shell
bastille_monitor_cron_path="/usr/local/etc/cron.d/bastille-monitor"
bastille_monitor_cron="*/5 * * * * root /usr/local/bin/bastille monitor ALL >/dev/null 2&>1"
bastille_monitor_logfile="${bastille_logsdir}/monitor.log"
bastille_monitor_healthchecks=""
Alerting modules
----------------
The first alerting module to be supported is Health Checks
(https://healthchecks.io), which is both a free SaaS service (up to 20 checks)
and provides a self-hosted option (see ``sysutils/py-healthchecks``).
Simply configure the ``${bastille_monitor_healthchecks}`` variable with your Ping
URL and you're done!
Help
----
.. code-block:: shell
ishmael ~ # bastille monitor help
Usage: bastille monitor [option(s)] enable|disable|status
TARGET add|delete|list service1,service2
TARGET list [service]
TARGET
Options:
-x | --debug Enable debug mode.

View File

@@ -30,6 +30,7 @@ Usage
limits Apply resources limits to jail(s). See rctl(8) and cpuset(1).
list List jails, releases, templates and more...
migrate Migrate jail(s) to a remote system.
monitor Monitor and attempt to restart jail service(s).
mount Mount file(s)/directorie(s) inside jail(s).
network Add or remove interface(s) from jail(s).
pkg Manage packages inside jail(s). See pkg(8).
@@ -53,4 +54,4 @@ Usage
Use "bastille -v|--version" for version information.
Use "bastille command -h|--help" for more information about a command.
Use "bastille -c|--config FILE command" to specify a non-default config file.
Use "bastille -c|--config FILE command" to specify a non-default config file.

View File

@@ -95,6 +95,7 @@ Available Commands:
limits Apply resources limits to jail(s). See rctl(8) and cpuset(1).
list List jails, releases, templates and more...
migrate Migrate jail(s) to a remote system.
monitor Monitor and attempt to restart jail service(s).
mount Mount file(s)/directorie(s) inside jail(s).
network Add or remove interface(s) from jail(s).
pkg Manage packages inside jail(s). See pkg(8).
@@ -204,6 +205,7 @@ case "${CMD}" in
limits| \
list| \
migrate| \
monitor| \
mount| \
network| \
pkg| \

View File

@@ -88,3 +88,9 @@ bastille_template_clone="default/clone" ## default
bastille_template_thin="default/thin" ## default: "default/thin"
bastille_template_vnet="default/vnet" ## default: "default/vnet"
bastille_template_vlan="default/vlan" ## default: "default/vlan"
## Monitoring
bastille_monitor_cron_path="/usr/local/etc/cron.d/bastille-monitor" ## default: "/usr/local/etc/cron.d/bastille-monitor"
bastille_monitor_cron="*/5 * * * * root /usr/local/bin/bastille monitor ALL >/dev/null 2>&1" # default: "*/5 * * * * root /usr/local/bin/bastille monitor ALL >/dev/null 2>&1"
bastille_monitor_logfile="${bastille_logsdir}/monitor.log" ## default: "${bastille_logsdir}/monitor.log"
bastille_monitor_healthchecks="" ## default: ""

View File

@@ -0,0 +1,196 @@
#!/bin/sh
#
# SPDX-License-Identifier: BSD-3-Clause
#
# Copyright (c) 2018-2025, Christer Edwards <christer.edwards@gmail.com>
# All rights reserved.
# Ressource limits added by Lars Engels github.com/bsdlme
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
#
# * Redistributions of source code must retain the above copyright notice, this
# list of conditions and the following disclaimer.
#
# * Redistributions in binary form must reproduce the above copyright notice,
# this list of conditions and the following disclaimer in the documentation
# and/or other materials provided with the distribution.
#
# * Neither the name of the copyright holder nor the names of its
# contributors may be used to endorse or promote products derived from
# this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
. /usr/local/share/bastille/common.sh
usage() {
error_notify "Usage: bastille monitor [option(s)] enable|disable|status"
error_notify " TARGET add|delete service1,service2"
error_notify " TARGET list [service]"
error_notify " TARGET"
cat << EOF
Options:
-x | --debug Enable debug mode.
EOF
exit 1
}
# Handle options.
while [ "$#" -gt 0 ]; do
case "${1}" in
-h|--help|help)
usage
;;
-x|--debug)
enable_debug
shift
;;
-*)
error_exit "[ERROR]: Unknown Option: \"${1}\""
;;
*)
break
;;
esac
done
# Handle global actions.
case "${1}" in
enable)
[ "$#" -eq 1 ] || usage
if [ ! -f "${bastille_monitor_cron_path}" ]; then
mkdir -p /usr/local/etc/cron.d
echo "${bastille_monitor_cron}" >> "${bastille_monitor_cron_path}"
echo "$(date '+%Y-%m-%d %H:%M:%S'): Added cron entry at ${bastille_monitor_cron_path}" >> "${bastille_monitor_logfile}"
info "\nBastille Monitor: Enabled\n"
exit 0
else
error_exit "\nBastille Monitor is already enabled."
fi
;;
disable)
[ "$#" -eq 1 ] || usage
if [ -f "${bastille_monitor_cron_path}" ]; then
rm -f "${bastille_monitor_cron_path}"
echo "$(date '+%Y-%m-%d %H:%M:%S'): Removed cron entry at ${bastille_monitor_cron_path}" >> "${bastille_monitor_logfile}"
info "\nBastille Monitor: Disabled\n"
exit 0
else
error_exit "\nBastille Monitor is not enabled."
fi
;;
status)
[ "$#" -eq 1 ] || usage
if [ -f "${bastille_monitor_cron_path}" ]; then
info "\nBastille Monitor Status: Active\n"
exit 0
else
info "\nBastille Monitor Status: Inactive\n"
exit 1
fi
;;
esac
if [ "$#" -eq 0 ]; then
usage
fi
TARGET="${1}"
ACTION="${2}"
SERVICE="${3}"
SERVICE_FAILED=0
bastille_root_check
set_target "${TARGET}"
for _jail in ${JAILS}; do
bastille_jail_monitor="${bastille_jailsdir}/${_jail}/monitor"
# Skip if jail is not running or no monitor file
if ! check_target_is_running "${_jail}" || [ ! -f "${bastille_jail_monitor}" ]; then
continue
fi
## iterate service(s) and check service status; restart on failure
if [ -z "${ACTION}" ] && [ -f "${bastille_jail_monitor}" ]; then
for _service in $(xargs < "${bastille_jail_monitor}"); do
## check service status
if ! jexec -l -U root "${_jail}" service "${_service}" status >/dev/null 2>/dev/null; then
echo "$(date '+%Y-%m-%d %H:%M:%S'): ${_service} service not running in ${_jail}. Restarting..." | tee -a "${bastille_monitor_logfile}"
## attempt to restart the service if needed; update logs if unable
if ! jexec -l -U root "${_jail}" service "${_service}" restart; then
echo "$(date '+%Y-%m-%d %H:%M:%S'): Failed to restart ${_service} service in ${_jail}." | tee -a "${bastille_monitor_logfile}"
SERVICE_FAILED=1
fi
fi
done
elif [ -n "${ACTION}" ]; then
case ${ACTION} in
add)
[ -z "${SERVICE}" ] && usage
for _service in $(echo "${SERVICE}" | tr , ' '); do
if ! grep -Eqs "^${_service}\$" "${bastille_jail_monitor}"; then
echo "${_service}" >> "${bastille_jail_monitor}"
echo "$(date '+%Y-%m-%d %H:%M:%S'): Added monitor for ${_service} on ${_jail}" >> "${bastille_monitor_logfile}"
fi
done
;;
del*)
[ -z "${SERVICE}" ] && usage
for _service in $(echo "${SERVICE}" | tr , ' '); do
[ ! -f "${bastille_jail_monitor}" ] && break # skip if no monitor file
if grep -Eqs "^${_service}\$" "${bastille_jail_monitor}"; then
sed -i '' "/^${_service}\$/d" "${bastille_jail_monitor}"
echo "$(date '+%Y-%m-%d %H:%M:%S'): Removed monitor for ${_service} on ${_jail}" >> "${bastille_monitor_logfile}"
fi
# delete monitor file if empty
[ ! -s "${bastille_jail_monitor}" ] && rm "${bastille_jail_monitor}"
done
;;
list)
if [ -n "${SERVICE}" ]; then
if echo "${SERVICE}" | grep ','; then
usage # Only one service per query
fi
[ ! -f "${bastille_jail_monitor}" ] && continue # skip if there is no monitor file
if grep -Eqs "^${SERVICE}\$" "${bastille_jail_monitor}"; then
echo "${_jail}"
continue
fi
else
if [ -f "${bastille_jail_monitor}" ]; then
info "\n[${_jail}]:"
xargs < "${bastille_jail_monitor}"
fi
fi
;;
*)
usage
;;
esac
fi
done
# Final ping to healthcheck URL
if [ "$SERVICE_FAILED" -eq 0 ]; then
if [ -n "${bastille_monitor_healthchecks}" ]; then
curl -fsS --retry 3 "${bastille_monitor_healthchecks}" > /dev/null 2>&1
else
curl -fsS --retry 3 "${bastille_monitor_healthchecks}/fail" > /dev/null 2>&1
fi
fi

View File

@@ -0,0 +1,133 @@
.Dd 2025/12/12
.Dt bastille-monitor 1
.Os
.Sh NAME
.Nm bastille monitor
.Nd Monitor and attempt to restart jail service(s).
.Sh SYNOPSIS
.Nm
.Op Fl x
.Nm
.Op Fl x
.Ar enable|disable|status
.Nm
.Op Fl x
.Ar TARGET
.Sy add|delete
.Ar SERVICE1,SERVICE2
.Nm
.Op Fl x
.Sy list
.Op SERVICE
.Sh DESCRIPTION
The
.Nm
sub-command will monitor jail services using the cron
mechanism. It will attempt to restart them if/when
they stop. See also
.Xr bastille.conf 5 .
.Bl -tag -width Ds
.It Sy bastille monitor Oo Fl x Oc Ar TARGET
.Bl -tag -width Ds
.It Fl x , Fl -debug
Enable debug mode.
.El
.Pp
Calling
.Nm
with only a
.Ar TARGET
is how we initiante the checking/restarting
of services. Any service(s) defined for
.Ar TARGET
will be restarted if they are stopped.
.It Sy bastille monitor Oo Fl x Oc Sy enable|disable|status
.Bl -tag -width Ds
.It Sy enable
This will enable monitoring, adding a cron entry to be
executed every 5 minutes, or the duration set
in 'bastille_monitor_cron'. See
.Xr bastille.conf 5 .
.It Sy disable
Disable the monitoring function. This will remove the
cron entry.
.It Sy status
Check if the monitor is currently active or inactive.
.It Fl x , Fl -debug
Enable debug mode.
.El
.It Sy bastille monitor Oo Fl x Oc Ar TARGET Sy add|delete Ar SERVICE1,SERVICE2
.Bl -tag -width Ds
.It Sy add
Add the
.Ar SERVICE
to the list of services to be monitored.
.It Sy delete
Remove the
.Ar SERVICE
from the monitoring list.
.It Fl x , Fl -debug
Enable debug mode.
.El
.It Sy bastille monitor Oo Fl x Oc Ar TARGET Sy list Op SERVICE
.Bl -tag -width Ds
.It Sy list
List monitored service(s) for
.Ar TARGET .
If
.Ar SERVICE
is specified, list jail(s) that are monitoring that service.
Note that it will only list the jails given in
.Ar TARGET .
.It Fl x , Fl -debug
Enable debug mode.
.Sh EXAMPLES
.Bl -tag -width Ds
.It Enable the monitoring service:
.Sy bastille monitor enable
.It Add nginx to the monitor list for myjail:
.Sy bastille monitor myjail add nginx
.It Add nginx and caddy to the monitor list for myjail:
.Sy bastille monitor myjail add nginx,caddy
.It Check which jails are monitoring caddy:
.Sy bastille monitor ALL list caddy
.Sh SEE ALSO
.Xr bastille.conf 5 ,
.Xr bastille-bootstrap 1 ,
.Xr bastille-clone 1 ,
.Xr bastille-cmd 1 ,
.Xr bastille-config 1 ,
.Xr bastille-console 1 ,
.Xr bastille-convert 1 ,
.Xr bastille-cp 1 ,
.Xr bastille-create 1 ,
.Xr bastille-destroy 1 ,
.Xr bastille-edit 1 ,
.Xr bastille-etcupdate 1 ,
.Xr bastille-export 1 ,
.Xr bastille-htop 1 ,
.Xr bastille-import 1 ,
.Xr bastille-jcp 1 ,
.Xr bastille-limits 1 ,
.Xr bastille-list 1 ,
.Xr bastille-migrate 1 ,
.Xr bastille-mount 1 ,
.Xr bastille-network 1 ,
.Xr bastille-pkg 1 ,
.Xr bastille-rcp 1 ,
.Xr bastille-rdr 1 ,
.Xr bastille-rename 1 ,
.Xr bastille-restart 1 ,
.Xr bastille-service 1 ,
.Xr bastille-setup 1 ,
.Xr bastille-start 1 ,
.Xr bastille-stop 1 ,
.Xr bastille-sysrc 1 ,
.Xr bastille-tags 1 ,
.Xr bastille-template 1 ,
.Xr bastille-top 1 ,
.Xr bastille-umount 1 ,
.Xr bastille-update 1 ,
.Xr bastille-upgrade 1 ,
.Xr bastille-verify 1 ,
.Xr bastille-zfs 1

View File

@@ -69,6 +69,8 @@ and
List jails, releases, templates and more...
.It Sy migrate
Migrate jail(s) to a remote system.
.It Sy monitor
Monitor and attempt to restart jail service(s).
.It Sy mount
Mount files(s)/directorie(s) inside jail(s).
.It Sy network

View File

@@ -128,6 +128,16 @@ The default template that is appled to thin jails.
The default template that is appled to vnet jails.
.It bastille_template_vlan
The default template that is appled to vnet+vlan jails.
.Ss MONITORING
.It bastille_monitor_cron_path
Cron file for automatic monitoring entry.
.It bastille_monitor_cron
Actual cron entry. The default is to check every 5 minutes.
.It bastille_monitor_logfile
Log storage.
.It bastille_monitor_healthchecks
This is the FQDN for optional alert services.
Currently only supports 'healthchecks.io'.
.Sh SEE ALSO
.Xr bastille-bootstrap 1 ,
.Xr bastille-clone 1 ,