Ugh, apparently yesterday a bot visited my Forgejo instance and queried everything, which caused Forgejo to create repo archives for everything. Git on the instance is 2.1 GB in size, but the repo archive filled up everything and is 120 GB. I really didn’t expect such a spike.
That meant that it filled up the whole hard drive and the server and all the services and websites on it went down while I was sleeping.
Luckily it seems that just deleting that directory fixes the problem temporarily. I also disabled the possibility of downloading archived from the UI but I’m not sure if this will prevent bots from generating those archives again. I also can’t just make the directory read only because it uses it for other things like mirroring, etc too.
For small instances like mine those archives are quite a headache.
Script for monitoring disk space in Linux
The script below is designed to monitor disk space usage on a specified server partition. Configurable parameters include the maximum allowable percentage of disk space usage (
MAX
), the e-mail address to receive alerts (EMAIL
) and the target partition (PARTITION
).The script uses the df command to collect disk usage information and sends email alerts if the current usage exceeds the specified threshold
#!/bin/bash # Script: ./df_guard.sh [config_file] # Set the maximum allowed disk space usage percentage MAX=90 # Set the email address to receive alerts EMAIL=user@example.com # Set the partition to monitor (change accordingly, e.g., /dev/sda1) PARTITION=/dev/sda1 # Get the current disk usage percentage and related information USAGE_INFO=$(df -h "$PARTITION" | awk 'NR==2 {print $5, $1, $2, $3, $4}' | tr '\n' ' ') USAGE=$(echo "$USAGE_INFO" | awk '{print int($1)}') # Remove the percentage sign if [ "$USAGE" -gt "$MAX" ]; then # Send an email alert with detailed disk usage information echo -e "Warning: Disk space usage on $PARTITION is $USAGE%.\n\nDisk Usage Information:\n$USAGE_INFO" | \ mail -s "Disk Space Alert on $HOSTNAME" "$EMAIL" fi
Installation
sudo install -m 0755 df_guard.sh /usr/local/bin/df_guard.sh
Make the script executable:
sudo chmod +x /usr/local/bin/df_guard.sh
Launch examples
In crontab (root)
*/15 * * * * * /usr/local/bin/df_guard.sh
I have monitoring of it, but it happened during night when I was sleeping.
Actually I saw a lot of forgejo action on the server yesterday but didn’t think it would go so fast.