I used to run my blog(s) on OpenShift Online (v2) which was, for many reasons, a terrible use case for that platform and my OpenShift colleagues hated that people do that a lot. This is one of the reason why OpenShift Online (v3) enforces some sleep time for the running services - to prevent you from running blogs and generally "production" services on the free service. You can still do it, if you don't mind some downtime.
For blog though, downtime is generally a bad thing, so I started to look around to find a different place to put it. Obviously, I did not want to pay and I especially did not want to pay "per instance" as I am also running a blog for my mum. My though was to use some cheap webhosting, but that would mean I would have to setup thing the old way - mode 1, without containers and all the orchestration goodness.
I started to look for a cheap cloud provided where I could run a small server for couple dollars a month - I first went to DigitalOcean as I have some free credits there, then I found Linode and Alibaba which were cheaper for the same or a bit more powerful VMs. The plan was to deploy OpenShift Origin, use Grant's tutorial to migrate the WordPress and never care about it again.
I created a VM, started to play around with OpenShift there and then I thought: "Maybe I could use an old laptop and just run OpenShift in my closet". The laptop has 4 GB of RAM and 2 cores of CPU, so it's 2-4 times more powerful than the VM in the cloud. It does not have SSD, so the disk operations are not the best, but it works and for $5-10 dollars I save on the cloud VM, I can buy the SSD later if I need.
So I updated the old netbook to latest Fedora, downloaded OpenShift Origin client binary from Github and started the single-node cluster. By following the above mentioned migration tutorial I got my WordPress instance running in few minutes and fixing the A record for DNS gave me my nice URL as well. So far, so good.
I did not have much time when I set it all up late last year, so I just migrated my mum's blog as well and let it run. It worked like a charm and I thought that as soon as I get a bit of time, I'll simply fix some things that were not setup properly - like backups or what happens when the laptop restarts.
As life happens, I never got back to figure out the backup and restart stories, which resulted in a major cluster fu...fail. We are redoing electrical wiring in our building, which means there are some expected outages. The laptop generally survived them fine, until one outage took almost a whole day. The laptop turned off and with it both blogs.
I thought it'll be fine - I'll just boot it up and run the OpenShift cluster again. Easy, right? Basically, yes, it is, if you read the docs carefully and keep your etcd data out of the Origin container, because if you don't, once the container dies, the etcd - i.e. the database behind OpenShift and Kubernetes, is gone. Ok, well, not a great thing to happen, but I still had all my other data from blogs, so I just needed to run the containers again with the same host volumes.
Sadly, that turned out to be problematic as well, because MySQL container does not really like to be pointed to the old volume with a new deployment (I did not investigate why, although it might be interesting for MySQL OpenShift deployment developers..). So the database failed. I copied the database volume to a new directory and started to experiment on it - the idea was that if I can dump it, I could just run a clean deployment and then import the dump.
This approach seemed to be getting me closer to the goal until the point where I deleted the original data volume by mistake, which left me only with the already corrupted copy. Yay. So close! At that point, there was not much else to do than to start over and run from the 6 months old backups I made during the migration. Not great, but lessons learnt.
Well, let's start from scratch and do things right now! First, run OpenShift Origin with a proper set of parameters. Those will include things like "run from existing configuration", "store etcd on disk", "select specific path for volumes"...
/bin/oc cluster up --public-hostname=$INTERNAL_IP.nip.io --routing-suffix=$EXTERNAL_IP.nip.io --use-existing-config --host-data-dir=$HOST_DIR --host-config-dir=$HOST_DIR/openshift.local.config --host-pv-dir=$HOST_DIR/openshift.local.pv --host-volumes-dir=$HOST_DIR/openshift.local.volumes
Next, make sure the cluster will get started on (re)boot, so that it comes back up if there is another electricity issue. For that I used a unit from Tobias Brunner's post. Now, let's reboot and...it works!.
Next thing is backup. It does not make much sense to store backup on the same drive and I did not really want to upload everything somewhere to the cloud, so I just plugged in a USB Flash drive I got at some conference and created a systemd mount unit based on James Oguya's post. For the actual backup, I decided to simply rsync the directory, where I store all the OpenShift configuration, etcd data, PVs, etc. to the flash drive. I am not entirely sure it is the best solution or even a solution which will allow me to restore things easily and fully, but I hope it will..and I will test it at some point;). For the rsync backup I created a systemd timer and a one shot unit which the timer runs once a day. ArchWiki post helped with this step.
The last thing is to not only start, but also enable all those services, so that things get into the right state after reboot. I tried to reboot couple times and things seemed to work fine after that - it takes couple minutes to boot, start the cluster and deploy all the containers, so there is a downtime during reboot, but I can live with that.
Wish me luck so that I don't lose the data again and share your stories about how and what you run at home:).