Tag Archives: Cloud

Running a WordPress blog on OpenShift Origin

I used to run my blog(s) on OpenShift Online (v2) which was, for many reasons, a terrible use case for that platform and my OpenShift colleagues hated that people do that a lot. This is one of the reason why OpenShift Online (v3) enforces some sleep time for the running services - to prevent you from running blogs and generally "production" services on the free service. You can still do it, if you don't mind some downtime.

For blog though, downtime is generally a bad thing, so I started to look around to find a different place to put it. Obviously, I did not want to pay and I especially did not want to pay "per instance" as I am also running a blog for my mum. My though was to use some cheap webhosting, but that would mean I would have to setup thing the old way - mode 1, without containers and all the orchestration goodness.

I started to look for a cheap cloud provided where I could run a small server for couple dollars a month - I first went to DigitalOcean as I have some free credits there, then I found Linode and Alibaba which were cheaper for the same or a bit more powerful VMs. The plan was to deploy OpenShift Origin, use Grant's tutorial to migrate the WordPress and never care about it again.

I created a VM, started to play around with OpenShift there and then I thought: "Maybe I could use an old laptop and just run OpenShift in my closet". The laptop has 4 GB of RAM and 2 cores of CPU, so it's 2-4 times more powerful than the VM in the cloud. It does not have SSD, so the disk operations are not the best, but it works and for $5-10 dollars I save on the cloud VM, I can buy the SSD later if I need.

So I updated the old netbook to latest Fedora, downloaded OpenShift Origin client binary from Github and started the single-node cluster. By following the above mentioned migration tutorial I got my WordPress instance running in few minutes and fixing the A record for DNS gave me my nice URL as well. So far, so good.

I did not have much time when I set it all up late last year, so I just migrated my mum's blog as well and let it run. It worked like a charm and I thought that as soon as I get a bit of time, I'll simply fix some things that were not setup properly - like backups or what happens when the laptop restarts.

As life happens, I never got back to figure out the backup and restart stories, which resulted in a major cluster fu...fail. We are redoing electrical wiring in our building, which means there are some expected outages. The laptop generally survived them fine, until one outage took almost a whole day. The laptop turned off and with it both blogs.

I thought it'll be fine - I'll just boot it up and run the OpenShift cluster again. Easy, right? Basically, yes, it is, if you read the docs carefully and keep your etcd data out of the Origin container, because if you don't, once the container dies, the etcd - i.e. the database behind OpenShift and Kubernetes, is gone. Ok, well, not a great thing to happen, but I still had all my other data from blogs, so I just needed to run the containers again with the same host volumes.

Sadly, that turned out to be problematic as well, because MySQL container does not really like to be pointed to the old volume with a new deployment (I did not investigate why, although it might be interesting for MySQL OpenShift deployment developers..). So the database failed. I copied the database volume to a new directory and started to experiment on it - the idea was that if I can dump it, I could just run a clean deployment and then import the dump.

This approach seemed to be getting me closer to the goal until the point where I deleted the original data volume by mistake, which left me only with the already corrupted copy. Yay. So close! At that point, there was not much else to do than to start over and run from the 6 months old backups I made during the migration. Not great, but lessons learnt.

Well, let's start from scratch and do things right now! First, run OpenShift Origin with a proper set of parameters. Those will include things like "run from existing configuration", "store etcd on disk", "select specific path for volumes"...

/bin/oc cluster up --public-hostname=$INTERNAL_IP.nip.io --routing-suffix=$EXTERNAL_IP.nip.io --use-existing-config --host-data-dir=$HOST_DIR --host-config-dir=$HOST_DIR/openshift.local.config --host-pv-dir=$HOST_DIR/openshift.local.pv --host-volumes-dir=$HOST_DIR/openshift.local.volumes

Next, make sure the cluster will get started on (re)boot, so that it comes back up if there is another electricity issue. For that I used a unit from Tobias Brunner's post. Now, let's reboot and...it works!.

Next thing is backup. It does not make much sense to store backup on the same drive and I did not really want to upload everything somewhere to the cloud, so I just plugged in a USB Flash drive I got at some conference and created a systemd mount unit based on James Oguya's post. For the actual backup, I decided to simply rsync the directory, where I store all the OpenShift configuration, etcd data, PVs, etc. to the flash drive. I am not entirely sure it is the best solution or even a solution which will allow me to restore things easily and fully, but I hope it will..and I will test it at some point;). For the rsync backup I created a systemd timer and a one shot unit which the timer runs once a day. ArchWiki post helped with this step.

The last thing is to not only start, but also enable all those services, so that things get into the right state after reboot. I tried to reboot couple times and things seemed to work fine after that - it takes couple minutes to boot, start the cluster and deploy all the containers, so there is a downtime during reboot, but I can live with that.

Wish me luck so that I don't lose the data again and share your stories about how and what you run at home:).

Containers: Resurrection vs. Reincarnation

This post is to share that I am coining new analogy about containers, cloud and orchestration. Everyone uses Pet vs. Cattle, some use Ant. vs. Elephant. These animal analogies are fine and describe how YOU behave to your machines/containers if something bad happens.

But it's also important to look at things from the other side - what actually happens to the app? How does it feel? How does it perceive world around itself?

Resurrection

In a good old Mode1 world things were simple. Some app died, so you went and resurrected it back to life. Sure, PID was different, but it was the same machine, same environment, same processes around it. Feels like home...

Reincarnation

Then the cloud and container world appeared and people realised they don't want to bring dead things back to life (it might have something to do with all the scary zombie movies, I think). And so in container orchestration you just get rid of things that appear to be dead and then bring new ones to life. And you app is reincarnated instead of resurrected

Resurrection vs. Reincarnation.

Reincarnation is not completely new in IT world - it was already used in MINIX many years ago:). But I am coining this new analogy for containers context. Obviously, it's up to you now to share the wisdom and make sure people know who was the original prophet!

Forget resurrection, reincarnation is a way to go!

 

Mapping service ports to nodes in Kubernetes

Kubernetes is a great project and cool/hot technology. Although it made me to hate JSON (and YAML), I still enjoy exploring the possibilities it brings to your applications deployment.

It's also a base for even more awesome project called OpenShift (*cough* shameless plug included *cough*).

Anyway, I ran into a problem where I needed to expose port(s) of my application to the outer world (i.e. from Vagrant box to my host) and I struggled to find the solution quickly.

Normally, when you are on the machine where Kubes run, you will do something like this

[vagrant@centos7-adb ~]$ kubectl get services | grep flower
flower-service component=flower app=taskQueue,component=flower 10.254.126.210 5555/TCP

IOW I just listed all running services and grepped for flower. I can take IP and port from there now and use curl to get contents provided by that service. This uses the Kubernetes virtual network to get to the endpoint.

I can also do this

[vagrant@centos7-adb ~]$ kubectl get endpoints | grep flower
flower-service 172.17.0.7:5555

which gets me directly to container IP and port.

But this all happens in my Vagrant box (as you can see from the CLI prompt). This setup is good for places like Google Cloud or AWS where you get load balancing and port forwarding for free. But what if I just want to access my app on the VM IP address?

Well, you take your Kubernetes service config/artefact/JSON/YAML and modify it a bit. By default, Kubernetes services are set to "ClusterIP" mode where they are accessible only by the ways showed above. You'll want to change the type to "NodePort".

This will "use a cluster IP, but also expose the service on a port on each node of the cluster (the same port on each node)" according to docs.

apiVersion: v1
 kind: Service
 metadata:
   labels:
     component: flower
     name: flower-service
spec:
  type: NodePort
  ports:
    - port: 5555
      nodePort: 31000
  selector:
    app: taskQueue
    component: flower

By default, type NodePort will give you a random port in a range 30000-32767. You can also pick a specific port from this range (as you can see above). Well, that's it. You only need to know the IP of the machine and the given/specified port.

[vagrant@centos7-adb vagrant]$ kubectl describe service flower-service | grep "^NodePort"
NodePort: <unnamed> 31000/TCP

This is particularly useful when you are developing (with VM, as the use case described above), or if you have some testing instance in the cloud (where the load balancers are not available) and want to expose the app easily without having to fiddle with too many other pieces.