I was hacked years ago. I was hosting a test instance of a phpbb for a local club. Work blocked SSH, so I opened up telnet. They either got in from telnet or a php flaw and installed password sniffers and replaced some tools (ps, top) with tools that would hide the sniffer service they installed.

After that, I changed my model. My time lab is for learning and having fun. I’m going to make mistakes and leave something exposed or vulnerable and hackers are going to get in. Under this new model, I need to be able to restore the system easily after a breach. I have a local backup and a remote backup and I have build scripts (ansible) so that I can restore the system if I need to. I’ve had to do this twice. Once from my own mistake and one from hardware failure.

Same. I have spent way more time troubleshooting a pipeline than it saves. I like the idea of automation but laziness prevails.

For my own curiosity, how do you perform a build? Is it all done in pipelines, kicked off on change? Do you execute the whole infra build each time you release an update?

As others have said, a traditional off site backup will work. How do you plan to perform a restore, though? If you need the self hosted source repo, it won’t be available until the infrastructure is stood to creating another circular dependency.

I’m still in the early stages of exploring this, too. My solution is to run a local filesystem git clone of the “main” repo and execute it with a Taskfile that builds a docker image from which it can execute the ansible infrastructure build. It is somewhat manual but I have performed a full rebuild a few times after some Big Mistakes.

After breaking “prod” many times, I have a Dev (local machine), Test (small VM) and Prod (big VM). My test is just less RAM and space and I need to spin down certain K8s things to spin up others, but it’s a close mirror of Prod, just less.

Thanks for the feedback. I plan to do some reading on NFSv4 domain mapping this weekend.