So here is the deal. OpnSense is based on FreeBSD (as the users know but others may not understand) and utilizes ZFS Snapshots which are an amazing way to go back to a previous state when something breaks after an update. However for some reason the cron section has been changed so you can not add any random cron job and can only select from predefined options. This means we have to work from the CLI to add jobs, but even worse after an update the cron jobs are removed… which becomes self defeating.

I think I have a decent solution that will work to not only persist the backup script that will automatically create and remove snapshots but also recreate the cron jobs automatically. Ideally there would be some sort of addon where we could adjust things but this is the best I can do for now and want to share with everyone.

Here is the script, place it at: /usr/local/etc/rc.syshook.d/start/99-snapshot-addon.sh and preform a chmod +x so that it is executable. Then reboot and the script to create the snapshot automatically and the cron job should be created. Since this runs on boot it will do this check after every update that requires a reboot or reboot in general and if the script and cron jobs are removed due to an update it will recreate them.

#!/bin/sh

CAT_PATH="/usr/local/opnsense/scripts/auto_be_retention.sh"

# Ensure target directory exists
mkdir -p /usr/local/opnsense/scripts

# 1. Write/Update the auto_be_retention.sh script
cat << 'EOF' > "$CAT_PATH"
#!/bin/sh

WEEKLY_RETENTION=6
MONTHLY_RETENTION=4

TYPE=$1

if [ "$TYPE" != "weekly" ] && [ "$TYPE" != "monthly" ]; then
    echo "Usage: $0 [weekly|monthly]"
    exit 1
fi

DATE_TAG=$(date +%Y%m%d_%H%M%S)
BE_NAME="auto_${TYPE}_${DATE_TAG}"

/sbin/bectl create "${BE_NAME}"

if [ "$TYPE" = "weekly" ]; then
    KEEP=$WEEKLY_RETENTION
else
    KEEP=$MONTHLY_RETENTION
fi

/sbin/bectl list -H | awk '{print $1}' | grep "^auto_${TYPE}_" | sort | \
awk -v keep="$KEEP" '
    { lines[NR] = $1 }
    END {
        to_delete = NR - keep
        for (i = 1; i <= to_delete; i++) {
            print lines[i]
        }
    }
' | while read -r OLD_BE; do
    if [ -n "$OLD_BE" ]; then
        /sbin/bectl destroy -F -o "$OLD_BE"
    fi
done
EOF

chmod +x "$CAT_PATH"

# 2. Inject cron tasks into /etc/crontab if missing
CRON_FILE="/etc/crontab"

if ! grep -q "auto_be_retention.sh weekly" "$CRON_FILE"; then
    echo "0 3 * * 0 root /usr/local/opnsense/scripts/auto_be_retention.sh weekly" >> "$CRON_FILE"
fi

if ! grep -q "auto_be_retention.sh monthly" "$CRON_FILE"; then
    echo "30 3 1 * * root /usr/local/opnsense/scripts/auto_be_retention.sh monthly" >> "$CRON_FILE"
fi

# 3. Reload cron daemon
service cron restart

This will check for and if missing create a script at /usr/local/opnsense/scripts/auto_be_retention.sh and it will check for and if missing create two cron jobs.

The script located at /usr/local/opnsense/scripts/auto_be_retention.sh will create a snapshot with a timestamp named similar to: auto_monthly_20261001_033000

It will also look the count of previous snapshots (6 for weekly and 4 for monthly which can be adjusted if you choose) and delete the oldest one. Weekly snapshots will happen every Sunday at 3:00 AM and monthly snapshots will happen on the 1st of the month at 3:30 AM.


Why do we need this?

We don’t “need” this but the advantage is that if you can have automatic updates (which are an option that is available in the cron section) it’s probably better to have a snapshot that can easily be rolled back to in the case of something acting up automatically created as well. The other option is try and undo the issue or take a config backup and reinstall then use that to get everything up and running again.

Rolling back a snapshot takes a few minutes and gets everything back up and running until a solution for the issue/s are available.

I ran the script and I am not seeing anything in the Settings > cron section!!

You won’t… that is part of the deal with how OpnSense has decided to handle things. You will need to look in: /etc/crontab This is because of how OpnSense is using menu drive cron selections, since the selections are not available it seems to not want to create or show them.

I have done some testing and removed the files and jobs and also had them recreated. At the next update we will see if everything sticks around or if it is removed.

Please remember that the script above is given without warranty or guarantee, use it at your own risk. I am not responsible if this script causes things not limited to but including data loss, testicular/breast shrinkage/growth, rude children, nagging wives/girlfriends/husbands/partners, alien visitation, global warming, or thermonuclear war. ID10T errors are also your own responsibility… you should not be running things you find on the internet without understanding how they work.

  • CaptainShardmind@lemmy.world
    link
    fedilink
    English
    arrow-up
    1
    ·
    9 hours ago

    I’m not on OPNsense, but I lean on the same idea with my Ubuntu servers, and being able to roll back is what makes me willing to update at all. One thing I’d add: boot environments live on the same disk, so export config.xml somewhere off the box on the same schedule. And do one deliberate rollback before an update forces you to, so you know how long it really takes.

    • MuttMutt@lemmy.worldOP
      link
      fedilink
      English
      arrow-up
      1
      ·
      8 hours ago

      I keep config backups of the config as well but my boot drive is a pair of mirrored Intel D3-S4510’s that are underprovioned by 50GB.

      Rollback time will technically depend on the size of the change since a rollback only replaces changed files so that will depend on how far back you are rolling back to. My largest snapshot references about 2GB and is a little over 4 months old, I would estimate about 5 minutes at most rolling back to that point.

  • ikidd@lemmy.dbzer0.com
    link
    fedilink
    arrow-up
    2
    ·
    14 hours ago

    I run my OPNsense as a proxmox VM, so I get high availability, replication, live backup and restore via PBS and update-time snapshots.

    I couldn’t recommend this method more, it’s saved my ass repeatedly.

    • MuttMutt@lemmy.worldOP
      link
      fedilink
      English
      arrow-up
      5
      ·
      13 hours ago

      I’ve never been comfortable virtualizing a router. Nearly anything else is not an issue.

      • ikidd@lemmy.dbzer0.com
        link
        fedilink
        arrow-up
        2
        ·
        11 hours ago

        It’s been amazing. The thing I like the most is being able to live-motion it to another node to upgrade the host then migrate back. Doesn’t drop a ping.

        I’m using some ancient Dell rackmount servers for nodes, and a couple times the prime server dropped off entirely because of power issues, and I didn’t notice for a couple weeks that it had jumped hosts. I’ve also had to restore it from backup because of a borked upgrade, and it took maybe 5 minutes.

        I thought that it would be a bad idea but I tried it with the original baremetal router that I lift and shifted to a VM then put the source on standby, and 5 years later, I would not go back.

    • tofu@lemmy.nocturnal.garden
      link
      fedilink
      English
      arrow-up
      1
      ·
      13 hours ago

      Depends on what you do with it I guess. It’s my gateway between my family LAN and my homelab LANs, and also the DHCP server for the latter. It’s supposed to be up when the proxmox cluster boots and all nodes would need 3 NICs for HA to work, and all be directly connected to the family LAN (in which they aren’t supposed to be).

      • ikidd@lemmy.dbzer0.com
        link
        fedilink
        arrow-up
        2
        ·
        11 hours ago

        Set the boot order in PVE. You can also group HA targets to shift as a group. I have a physical nic on WAN and LAN, and the LAN nic is VLANned for DMZ, guest, IOT, etc. As long as you keep them consistent from node to node, I’ve never had an issue.

        I have been thinking about a failover LTE modem, and that one has me a bit stumped about how to deal with HA moves. Maybe a GLiNet LTE router that I can VLAN on the WAN nic. Or maybe I just hook it into the LAN network and have failover scripts for my critical devices

  • non_burglar@lemmy.world
    link
    fedilink
    English
    arrow-up
    8
    ·
    20 hours ago
    • I personally wouldn’t be touching /etc/crontab, but adding or editing a tab in /var/cron/tabs.
    • I would use test or just a bash eval instead of mkdir to check for the existence of a directory. Mkdir has backfired on me spectacularly in bash and sh scripts.
    • always grep, then sort, then awk. Just as a habit, any other order can make run time long and cause globbing problems.
    • good job
    • MuttMutt@lemmy.worldOP
      link
      fedilink
      English
      arrow-up
      1
      ·
      16 hours ago

      On OpnSense no. I previously did not have snapshots created. I have on FreeNAS when I had a setting issue that came up. Fundamentally it should be no different.

      I have had to reinstall OpnSense before and would rather not go through that again if at all possible.