diff --git a/docs/chapters/configuration.rst b/docs/chapters/configuration.rst index 99503016..64b39472 100644 --- a/docs/chapters/configuration.rst +++ b/docs/chapters/configuration.rst @@ -104,6 +104,23 @@ This is the default `bastille.conf` file. bastille_template_vnet="default/vnet" ## default: "default/vnet" bastille_template_vlan="default/vlan" ## default: "default/vlan" + ## Monitoring + bastille_monitor_cron_path="/usr/local/etc/cron.d/bastille-monitor" ## default: "/usr/local/etc/cron.d/bastille-monitor" + bastille_monitor_cron="*/5 * * * * root /usr/local/bin/bastille monitor ALL >/dev/null 2>&1" ## default: "*/5 * * * * root /usr/local/bin/bastille monitor ALL >/dev/null 2>&1" + bastille_monitor_logfile="${bastille_logsdir}/monitor.log" ## default: "${bastille_logsdir}/monitor.log" + bastille_monitor_healthchecks="" ## default: "" + + +Notes +----- + +The options here are fairly self-explanitory, but there are some things to note. + +* Bastille will mount the dataset it creates at ``bastille_prefix`` which + defaults to ``/usr/local/bastille``. So if you want to navigate to your jails, + you will use the ``bastille_prefix`` as the location because this is where the + will be mounted. + Custom Configuration -------------------- diff --git a/docs/chapters/subcommands/index.rst b/docs/chapters/subcommands/index.rst index ca46d87c..9139bb7e 100644 --- a/docs/chapters/subcommands/index.rst +++ b/docs/chapters/subcommands/index.rst @@ -23,6 +23,7 @@ Bastille sub-commands limits list migrate + monitor mount network pkg diff --git a/docs/chapters/subcommands/monitor.rst b/docs/chapters/subcommands/monitor.rst new file mode 100644 index 00000000..cd1a57a2 --- /dev/null +++ b/docs/chapters/subcommands/monitor.rst @@ -0,0 +1,81 @@ +monitor +======= + +NEW in Bastille version 1.1.20250814 + +The ``monitor`` sub-command adds, removes, lists and enables/disables monitoring for container services. + + +Managing Bastille Monitor +------------------------- + +To enable Bastille monitoring, run ``bastille monitor enable``. + +To disable Bastille monitoring, run ``bastille monitor disable``. + +We can always check if Bastille monitoring is active with ``bastille monitor status``. + + +Managing Services +----------------- + +Bastille Monitor will attempt to monitor any services defined for any given container. If the service is +stopped, Bastille will attempt to restart it. Everything is logged in ``${bastille_monitor_logfile}``. + +To have Bastille monitor a service, run ``bastille monitor TARGET add SERVICE``. The ``SERVICE`` arg can also be a +comma-separated list of services such as ``bastille monitor TARGET add SERVICE1,SERVICE2``. + +To remove a service from monitoring, we can run ``bastille monitor TARGET delete SERVICE``. These can also be a +comma-separated list. + +To show all services that Bastille is monitoring, run ``bastille monitor TARGET list``. + +To list all jails that have a selected service defined for monitoring, run ``bastille monitor TARGET list SERVICE``. +This option only accepts a single ``SERVICE``, and cannot be a comma-separated list. + +If you run ``bastille monitor TARGET``, without any args or actions, Bastille will run through the process of +checking the status of each defined service, and attempt to start any that are stopped. + +Services can also be manually added or removed by editing the ``monitor`` file inside the jail directory, but +is not recommended unless you are an advanced user. + + +Configuration +------------- + +The monitor sub-command is configurable via the ``bastille.conf`` file. See below +for configuration defaults: + +.. code-block:: shell + + bastille_monitor_cron_path="/usr/local/etc/cron.d/bastille-monitor" + bastille_monitor_cron="*/5 * * * * root /usr/local/bin/bastille monitor ALL >/dev/null 2&>1" + bastille_monitor_logfile="${bastille_logsdir}/monitor.log" + bastille_monitor_healthchecks="" + + +Alerting modules +---------------- + +The first alerting module to be supported is Health Checks +(https://healthchecks.io), which is both a free SaaS service (up to 20 checks) +and provides a self-hosted option (see ``sysutils/py-healthchecks``). + +Simply configure the ``${bastille_monitor_healthchecks}`` variable with your Ping +URL and you're done! + + +Help +---- + +.. code-block:: shell + + ishmael ~ # bastille monitor help + Usage: bastille monitor [option(s)] enable|disable|status + TARGET add|delete|list service1,service2 + TARGET list [service] + TARGET + + Options: + + -x | --debug Enable debug mode. diff --git a/docs/chapters/usage.rst b/docs/chapters/usage.rst index 2221de3e..774e0ed1 100644 --- a/docs/chapters/usage.rst +++ b/docs/chapters/usage.rst @@ -30,6 +30,7 @@ Usage limits Apply resources limits to jail(s). See rctl(8) and cpuset(1). list List jails, releases, templates and more... migrate Migrate jail(s) to a remote system. + monitor Monitor and attempt to restart jail service(s). mount Mount file(s)/directorie(s) inside jail(s). network Add or remove interface(s) from jail(s). pkg Manage packages inside jail(s). See pkg(8). @@ -53,4 +54,4 @@ Usage Use "bastille -v|--version" for version information. Use "bastille command -h|--help" for more information about a command. - Use "bastille -c|--config FILE command" to specify a non-default config file. \ No newline at end of file + Use "bastille -c|--config FILE command" to specify a non-default config file. diff --git a/usr/local/bin/bastille b/usr/local/bin/bastille index 4bea13cb..05a7edc9 100755 --- a/usr/local/bin/bastille +++ b/usr/local/bin/bastille @@ -95,6 +95,7 @@ Available Commands: limits Apply resources limits to jail(s). See rctl(8) and cpuset(1). list List jails, releases, templates and more... migrate Migrate jail(s) to a remote system. + monitor Monitor and attempt to restart jail service(s). mount Mount file(s)/directorie(s) inside jail(s). network Add or remove interface(s) from jail(s). pkg Manage packages inside jail(s). See pkg(8). @@ -204,6 +205,7 @@ case "${CMD}" in limits| \ list| \ migrate| \ + monitor| \ mount| \ network| \ pkg| \ diff --git a/usr/local/etc/bastille/bastille.conf.sample b/usr/local/etc/bastille/bastille.conf.sample index 9a571b38..561b4d77 100644 --- a/usr/local/etc/bastille/bastille.conf.sample +++ b/usr/local/etc/bastille/bastille.conf.sample @@ -88,3 +88,9 @@ bastille_template_clone="default/clone" ## default bastille_template_thin="default/thin" ## default: "default/thin" bastille_template_vnet="default/vnet" ## default: "default/vnet" bastille_template_vlan="default/vlan" ## default: "default/vlan" + +## Monitoring +bastille_monitor_cron_path="/usr/local/etc/cron.d/bastille-monitor" ## default: "/usr/local/etc/cron.d/bastille-monitor" +bastille_monitor_cron="*/5 * * * * root /usr/local/bin/bastille monitor ALL >/dev/null 2>&1" # default: "*/5 * * * * root /usr/local/bin/bastille monitor ALL >/dev/null 2>&1" +bastille_monitor_logfile="${bastille_logsdir}/monitor.log" ## default: "${bastille_logsdir}/monitor.log" +bastille_monitor_healthchecks="" ## default: "" diff --git a/usr/local/share/bastille/monitor.sh b/usr/local/share/bastille/monitor.sh new file mode 100644 index 00000000..e46611e5 --- /dev/null +++ b/usr/local/share/bastille/monitor.sh @@ -0,0 +1,196 @@ +#!/bin/sh +# +# SPDX-License-Identifier: BSD-3-Clause +# +# Copyright (c) 2018-2025, Christer Edwards +# All rights reserved. +# Ressource limits added by Lars Engels github.com/bsdlme +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are met: +# +# * Redistributions of source code must retain the above copyright notice, this +# list of conditions and the following disclaimer. +# +# * Redistributions in binary form must reproduce the above copyright notice, +# this list of conditions and the following disclaimer in the documentation +# and/or other materials provided with the distribution. +# +# * Neither the name of the copyright holder nor the names of its +# contributors may be used to endorse or promote products derived from +# this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +. /usr/local/share/bastille/common.sh + +usage() { + error_notify "Usage: bastille monitor [option(s)] enable|disable|status" + error_notify " TARGET add|delete service1,service2" + error_notify " TARGET list [service]" + error_notify " TARGET" + cat << EOF + + Options: + + -x | --debug Enable debug mode. + +EOF + exit 1 +} + +# Handle options. +while [ "$#" -gt 0 ]; do + case "${1}" in + -h|--help|help) + usage + ;; + -x|--debug) + enable_debug + shift + ;; + -*) + error_exit "[ERROR]: Unknown Option: \"${1}\"" + ;; + *) + break + ;; + esac +done + +# Handle global actions. +case "${1}" in + enable) + [ "$#" -eq 1 ] || usage + if [ ! -f "${bastille_monitor_cron_path}" ]; then + mkdir -p /usr/local/etc/cron.d + echo "${bastille_monitor_cron}" >> "${bastille_monitor_cron_path}" + echo "$(date '+%Y-%m-%d %H:%M:%S'): Added cron entry at ${bastille_monitor_cron_path}" >> "${bastille_monitor_logfile}" + info "\nBastille Monitor: Enabled\n" + exit 0 + else + error_exit "\nBastille Monitor is already enabled." + fi + ;; + disable) + [ "$#" -eq 1 ] || usage + if [ -f "${bastille_monitor_cron_path}" ]; then + rm -f "${bastille_monitor_cron_path}" + echo "$(date '+%Y-%m-%d %H:%M:%S'): Removed cron entry at ${bastille_monitor_cron_path}" >> "${bastille_monitor_logfile}" + info "\nBastille Monitor: Disabled\n" + exit 0 + else + error_exit "\nBastille Monitor is not enabled." + fi + ;; + status) + [ "$#" -eq 1 ] || usage + if [ -f "${bastille_monitor_cron_path}" ]; then + info "\nBastille Monitor Status: Active\n" + exit 0 + else + info "\nBastille Monitor Status: Inactive\n" + exit 1 + fi + ;; +esac + +if [ "$#" -eq 0 ]; then + usage +fi + +TARGET="${1}" +ACTION="${2}" +SERVICE="${3}" +SERVICE_FAILED=0 + +bastille_root_check +set_target "${TARGET}" + +for _jail in ${JAILS}; do + + bastille_jail_monitor="${bastille_jailsdir}/${_jail}/monitor" + + # Skip if jail is not running or no monitor file + if ! check_target_is_running "${_jail}" || [ ! -f "${bastille_jail_monitor}" ]; then + continue + fi + + ## iterate service(s) and check service status; restart on failure + if [ -z "${ACTION}" ] && [ -f "${bastille_jail_monitor}" ]; then + for _service in $(xargs < "${bastille_jail_monitor}"); do + ## check service status + if ! jexec -l -U root "${_jail}" service "${_service}" status >/dev/null 2>/dev/null; then + echo "$(date '+%Y-%m-%d %H:%M:%S'): ${_service} service not running in ${_jail}. Restarting..." | tee -a "${bastille_monitor_logfile}" + + ## attempt to restart the service if needed; update logs if unable + if ! jexec -l -U root "${_jail}" service "${_service}" restart; then + echo "$(date '+%Y-%m-%d %H:%M:%S'): Failed to restart ${_service} service in ${_jail}." | tee -a "${bastille_monitor_logfile}" + SERVICE_FAILED=1 + fi + fi + done + elif [ -n "${ACTION}" ]; then + case ${ACTION} in + add) + [ -z "${SERVICE}" ] && usage + for _service in $(echo "${SERVICE}" | tr , ' '); do + if ! grep -Eqs "^${_service}\$" "${bastille_jail_monitor}"; then + echo "${_service}" >> "${bastille_jail_monitor}" + echo "$(date '+%Y-%m-%d %H:%M:%S'): Added monitor for ${_service} on ${_jail}" >> "${bastille_monitor_logfile}" + fi + done + ;; + del*) + [ -z "${SERVICE}" ] && usage + for _service in $(echo "${SERVICE}" | tr , ' '); do + [ ! -f "${bastille_jail_monitor}" ] && break # skip if no monitor file + if grep -Eqs "^${_service}\$" "${bastille_jail_monitor}"; then + sed -i '' "/^${_service}\$/d" "${bastille_jail_monitor}" + echo "$(date '+%Y-%m-%d %H:%M:%S'): Removed monitor for ${_service} on ${_jail}" >> "${bastille_monitor_logfile}" + fi + # delete monitor file if empty + [ ! -s "${bastille_jail_monitor}" ] && rm "${bastille_jail_monitor}" + done + ;; + list) + if [ -n "${SERVICE}" ]; then + if echo "${SERVICE}" | grep ','; then + usage # Only one service per query + fi + [ ! -f "${bastille_jail_monitor}" ] && continue # skip if there is no monitor file + if grep -Eqs "^${SERVICE}\$" "${bastille_jail_monitor}"; then + echo "${_jail}" + continue + fi + else + if [ -f "${bastille_jail_monitor}" ]; then + info "\n[${_jail}]:" + xargs < "${bastille_jail_monitor}" + fi + fi + ;; + *) + usage + ;; + esac + fi +done + +# Final ping to healthcheck URL +if [ "$SERVICE_FAILED" -eq 0 ]; then + if [ -n "${bastille_monitor_healthchecks}" ]; then + curl -fsS --retry 3 "${bastille_monitor_healthchecks}" > /dev/null 2>&1 + else + curl -fsS --retry 3 "${bastille_monitor_healthchecks}/fail" > /dev/null 2>&1 + fi +fi diff --git a/usr/local/share/man/man1/bastille-monitor.1 b/usr/local/share/man/man1/bastille-monitor.1 new file mode 100644 index 00000000..47e6af0e --- /dev/null +++ b/usr/local/share/man/man1/bastille-monitor.1 @@ -0,0 +1,133 @@ +.Dd 2025/12/12 +.Dt bastille-monitor 1 +.Os +.Sh NAME +.Nm bastille monitor +.Nd Monitor and attempt to restart jail service(s). +.Sh SYNOPSIS +.Nm +.Op Fl x +.Nm +.Op Fl x +.Ar enable|disable|status +.Nm +.Op Fl x +.Ar TARGET +.Sy add|delete +.Ar SERVICE1,SERVICE2 +.Nm +.Op Fl x +.Sy list +.Op SERVICE +.Sh DESCRIPTION +The +.Nm +sub-command will monitor jail services using the cron +mechanism. It will attempt to restart them if/when +they stop. See also +.Xr bastille.conf 5 . +.Bl -tag -width Ds +.It Sy bastille monitor Oo Fl x Oc Ar TARGET +.Bl -tag -width Ds +.It Fl x , Fl -debug +Enable debug mode. +.El +.Pp +Calling +.Nm +with only a +.Ar TARGET +is how we initiante the checking/restarting +of services. Any service(s) defined for +.Ar TARGET +will be restarted if they are stopped. +.It Sy bastille monitor Oo Fl x Oc Sy enable|disable|status +.Bl -tag -width Ds +.It Sy enable +This will enable monitoring, adding a cron entry to be +executed every 5 minutes, or the duration set +in 'bastille_monitor_cron'. See +.Xr bastille.conf 5 . +.It Sy disable +Disable the monitoring function. This will remove the +cron entry. +.It Sy status +Check if the monitor is currently active or inactive. +.It Fl x , Fl -debug +Enable debug mode. +.El +.It Sy bastille monitor Oo Fl x Oc Ar TARGET Sy add|delete Ar SERVICE1,SERVICE2 +.Bl -tag -width Ds +.It Sy add +Add the +.Ar SERVICE +to the list of services to be monitored. +.It Sy delete +Remove the +.Ar SERVICE +from the monitoring list. +.It Fl x , Fl -debug +Enable debug mode. +.El +.It Sy bastille monitor Oo Fl x Oc Ar TARGET Sy list Op SERVICE +.Bl -tag -width Ds +.It Sy list +List monitored service(s) for +.Ar TARGET . +If +.Ar SERVICE +is specified, list jail(s) that are monitoring that service. +Note that it will only list the jails given in +.Ar TARGET . +.It Fl x , Fl -debug +Enable debug mode. +.Sh EXAMPLES +.Bl -tag -width Ds +.It Enable the monitoring service: +.Sy bastille monitor enable +.It Add nginx to the monitor list for myjail: +.Sy bastille monitor myjail add nginx +.It Add nginx and caddy to the monitor list for myjail: +.Sy bastille monitor myjail add nginx,caddy +.It Check which jails are monitoring caddy: +.Sy bastille monitor ALL list caddy +.Sh SEE ALSO +.Xr bastille.conf 5 , +.Xr bastille-bootstrap 1 , +.Xr bastille-clone 1 , +.Xr bastille-cmd 1 , +.Xr bastille-config 1 , +.Xr bastille-console 1 , +.Xr bastille-convert 1 , +.Xr bastille-cp 1 , +.Xr bastille-create 1 , +.Xr bastille-destroy 1 , +.Xr bastille-edit 1 , +.Xr bastille-etcupdate 1 , +.Xr bastille-export 1 , +.Xr bastille-htop 1 , +.Xr bastille-import 1 , +.Xr bastille-jcp 1 , +.Xr bastille-limits 1 , +.Xr bastille-list 1 , +.Xr bastille-migrate 1 , +.Xr bastille-mount 1 , +.Xr bastille-network 1 , +.Xr bastille-pkg 1 , +.Xr bastille-rcp 1 , +.Xr bastille-rdr 1 , +.Xr bastille-rename 1 , +.Xr bastille-restart 1 , +.Xr bastille-service 1 , +.Xr bastille-setup 1 , +.Xr bastille-start 1 , +.Xr bastille-stop 1 , +.Xr bastille-sysrc 1 , +.Xr bastille-tags 1 , +.Xr bastille-template 1 , +.Xr bastille-top 1 , +.Xr bastille-umount 1 , +.Xr bastille-update 1 , +.Xr bastille-upgrade 1 , +.Xr bastille-verify 1 , +.Xr bastille-zfs 1 \ No newline at end of file diff --git a/usr/local/share/man/man1/bastille.1 b/usr/local/share/man/man1/bastille.1 index 2fbd8972..a52b4c4c 100644 --- a/usr/local/share/man/man1/bastille.1 +++ b/usr/local/share/man/man1/bastille.1 @@ -69,6 +69,8 @@ and List jails, releases, templates and more... .It Sy migrate Migrate jail(s) to a remote system. +.It Sy monitor +Monitor and attempt to restart jail service(s). .It Sy mount Mount files(s)/directorie(s) inside jail(s). .It Sy network diff --git a/usr/local/share/man/man5/bastille.conf.5 b/usr/local/share/man/man5/bastille.conf.5 index b16a932e..bc0fd2c0 100644 --- a/usr/local/share/man/man5/bastille.conf.5 +++ b/usr/local/share/man/man5/bastille.conf.5 @@ -128,6 +128,16 @@ The default template that is appled to thin jails. The default template that is appled to vnet jails. .It bastille_template_vlan The default template that is appled to vnet+vlan jails. +.Ss MONITORING +.It bastille_monitor_cron_path +Cron file for automatic monitoring entry. +.It bastille_monitor_cron +Actual cron entry. The default is to check every 5 minutes. +.It bastille_monitor_logfile +Log storage. +.It bastille_monitor_healthchecks +This is the FQDN for optional alert services. +Currently only supports 'healthchecks.io'. .Sh SEE ALSO .Xr bastille-bootstrap 1 , .Xr bastille-clone 1 ,