And the comminity blocklists are updated when more than a couple ( I think the number is something like 10-50 ) instances of crowdsec block an ip in some fast timeframe.
The ai blocklist just adds IP when even one instance finds an AI trying to scrape right from the useragent.
So even if the community blocklist has fewer ai ip’s, it does eventually include them.
Try crowdsec.
You can set it up with list’s that are updated frequetly and have it look at caddy proxy logs and then it can easilly block ai/bot like traffic.
I have it blocking over 100k ip’s at this moment.
I use nixos on my desktop, the server is a debian one but might be good to install nix on it.
I just have to import only one. Might just use thunderbird for that.
Will test out mailcow and see how it goes.
It was a systemd issue, after installing network manager, enabling it and disabling systemd-networkd share now mount’s in a matter of second’s.
I don’t like the fact I got to use network manager, but whatever.
Dec 15 14:12:33 arch systemd[1]: mnt-nfs.automount: Got automount request for /mnt/nfs, triggered by 1926 (keepassxc)
Dec 15 14:12:33 arch systemd[1]: Starting Wait for Network to be Configured...
Dec 15 14:12:33 arch systemd-networkd[1612]: enp3s0: DHCPv4 address 192.168.0.3/24, gateway 192.168.0.1 acquired from 192.168.0.1
Dec 15 14:12:33 arch systemd-timesyncd[1587]: Network configuration changed, trying to establish connection.
Dec 15 14:12:33 arch systemd-timesyncd[1587]: Network configuration changed, trying to establish connection.
Dec 15 14:12:38 arch systemd[1778]: Started tmux child pane 2350 launched by process 2106.
Dec 15 14:12:46 arch systemd[1778]: Created slice Slice /app/dbus-:1.15-org.a11y.atspi.Registry.
Dec 15 14:12:46 arch systemd[1778]: Started dbus-:1.15-org.a11y.atspi.Registry@0.service.
Dec 15 14:12:46 arch at-spi2-registryd[2526]: SpiRegistry daemon is running with well-known name - org.a11y.atspi.Registry
Dec 15 14:12:58 arch kernel: logitech-hidpp-device 0003:046D:4074.0008: HID++ 4.2 device connected.
Dec 15 14:13:03 arch systemd-timesyncd[1587]: Contacted time server 161.53.131.231:123 (2.arch.pool.ntp.org).
Dec 15 14:13:03 arch systemd-timesyncd[1587]: Initial clock synchronization to Sun 2024-12-15 14:13:03.310583 CET.
Dec 15 14:14:33 arch systemd-networkd-wait-online[2039]: Timeout occurred while waiting for network connectivity.
Dec 15 14:14:33 arch systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE
Dec 15 14:14:33 arch systemd[1]: systemd-networkd-wait-online.service: Failed with result 'exit-code'.
Dec 15 14:14:33 arch systemd[1]: Failed to start Wait for Network to be Configured.
Dec 15 14:14:33 arch systemd[1]: Reached target Network is Online.
Dec 15 14:14:33 arch systemd[1]: Mounting /mnt/nfs...
According to log’s it’s a systemd-networkd-wait-online.service issue, which shouldn’t be an issue because I’m using ethernet and share get’s automaunt request when I login into my user and load my xorg server.
Yes systemd is waiting until the network is up before trying to mount them, and am using the feature of systemd to not load until I need them to load.
From reading the logs further seems to be an issue with systemd not being able to properly check if network is up because I don’t use network manager, because after systemd-networkd-wait-online.service times out the nfs share get’s mounted right away,
I’m using the default list alongside Firehol BotScout list and Firehol cybercrime tracker list set to ban.
Also using the Firehol cruzit.com list set to do captcha, just in case it’s not actually a bot.
I’m also using the cs-firewall-bouncer and a custom bouncer that’s shown on crowdsecs tutorials to detect privilege escalation for if anybody actually manages to get inside.
Alongside that I’m using a lot of scenario collection’s for specific software I’m using like nextcloud, grafana, ssh, … which helps a lot with attacks directly done on a service and not just general scraping or both path traversing.
All free and have been using it for a year, only complaint I have is that I had to make a cronjob to restart the crowdsec service every day because it would stop working after a couple days because of the amount of requests it has to process.