Category Archives: Docker

If you are looking for interesting tips and information about Docker, you might be at the right place.

Running a WordPress blog on OpenShift Origin

I used to run my blog(s) on OpenShift Online (v2) which was, for many reasons, a terrible use case for that platform and my OpenShift colleagues hated that people do that a lot. This is one of the reason why OpenShift Online (v3) enforces some sleep time for the running services - to prevent you from running blogs and generally "production" services on the free service. You can still do it, if you don't mind some downtime.

For blog though, downtime is generally a bad thing, so I started to look around to find a different place to put it. Obviously, I did not want to pay and I especially did not want to pay "per instance" as I am also running a blog for my mum. My though was to use some cheap webhosting, but that would mean I would have to setup thing the old way - mode 1, without containers and all the orchestration goodness.

I started to look for a cheap cloud provided where I could run a small server for couple dollars a month - I first went to DigitalOcean as I have some free credits there, then I found Linode and Alibaba which were cheaper for the same or a bit more powerful VMs. The plan was to deploy OpenShift Origin, use Grant's tutorial to migrate the WordPress and never care about it again.

I created a VM, started to play around with OpenShift there and then I thought: "Maybe I could use an old laptop and just run OpenShift in my closet". The laptop has 4 GB of RAM and 2 cores of CPU, so it's 2-4 times more powerful than the VM in the cloud. It does not have SSD, so the disk operations are not the best, but it works and for $5-10 dollars I save on the cloud VM, I can buy the SSD later if I need.

So I updated the old netbook to latest Fedora, downloaded OpenShift Origin client binary from Github and started the single-node cluster. By following the above mentioned migration tutorial I got my WordPress instance running in few minutes and fixing the A record for DNS gave me my nice URL as well. So far, so good.

I did not have much time when I set it all up late last year, so I just migrated my mum's blog as well and let it run. It worked like a charm and I thought that as soon as I get a bit of time, I'll simply fix some things that were not setup properly - like backups or what happens when the laptop restarts.

As life happens, I never got back to figure out the backup and restart stories, which resulted in a major cluster We are redoing electrical wiring in our building, which means there are some expected outages. The laptop generally survived them fine, until one outage took almost a whole day. The laptop turned off and with it both blogs.

I thought it'll be fine - I'll just boot it up and run the OpenShift cluster again. Easy, right? Basically, yes, it is, if you read the docs carefully and keep your etcd data out of the Origin container, because if you don't, once the container dies, the etcd - i.e. the database behind OpenShift and Kubernetes, is gone. Ok, well, not a great thing to happen, but I still had all my other data from blogs, so I just needed to run the containers again with the same host volumes.

Sadly, that turned out to be problematic as well, because MySQL container does not really like to be pointed to the old volume with a new deployment (I did not investigate why, although it might be interesting for MySQL OpenShift deployment developers..). So the database failed. I copied the database volume to a new directory and started to experiment on it - the idea was that if I can dump it, I could just run a clean deployment and then import the dump.

This approach seemed to be getting me closer to the goal until the point where I deleted the original data volume by mistake, which left me only with the already corrupted copy. Yay. So close! At that point, there was not much else to do than to start over and run from the 6 months old backups I made during the migration. Not great, but lessons learnt.

Well, let's start from scratch and do things right now! First, run OpenShift Origin with a proper set of parameters. Those will include things like "run from existing configuration", "store etcd on disk", "select specific path for volumes"...

/bin/oc cluster up --public-hostname=$ --routing-suffix=$ --use-existing-config --host-data-dir=$HOST_DIR --host-config-dir=$HOST_DIR/openshift.local.config --host-pv-dir=$HOST_DIR/openshift.local.pv --host-volumes-dir=$HOST_DIR/openshift.local.volumes

Next, make sure the cluster will get started on (re)boot, so that it comes back up if there is another electricity issue. For that I used a unit from Tobias Brunner's post. Now, let's reboot works!.

Next thing is backup. It does not make much sense to store backup on the same drive and I did not really want to upload everything somewhere to the cloud, so I just plugged in a USB Flash drive I got at some conference and created a systemd mount unit based on James Oguya's post. For the actual backup, I decided to simply rsync the directory, where I store all the OpenShift configuration, etcd data, PVs, etc. to the flash drive. I am not entirely sure it is the best solution or even a solution which will allow me to restore things easily and fully, but I hope it will..and I will test it at some point;). For the rsync backup I created a systemd timer and a one shot unit which the timer runs once a day. ArchWiki post helped with this step.

The last thing is to not only start, but also enable all those services, so that things get into the right state after reboot. I tried to reboot couple times and things seemed to work fine after that - it takes couple minutes to boot, start the cluster and deploy all the containers, so there is a downtime during reboot, but I can live with that.

Wish me luck so that I don't lose the data again and share your stories about how and what you run at home:).

Containers: Resurrection vs. Reincarnation

This post is to share that I am coining new analogy about containers, cloud and orchestration. Everyone uses Pet vs. Cattle, some use Ant. vs. Elephant. These animal analogies are fine and describe how YOU behave to your machines/containers if something bad happens.

But it's also important to look at things from the other side - what actually happens to the app? How does it feel? How does it perceive world around itself?


In a good old Mode1 world things were simple. Some app died, so you went and resurrected it back to life. Sure, PID was different, but it was the same machine, same environment, same processes around it. Feels like home...


Then the cloud and container world appeared and people realised they don't want to bring dead things back to life (it might have something to do with all the scary zombie movies, I think). And so in container orchestration you just get rid of things that appear to be dead and then bring new ones to life. And you app is reincarnated instead of resurrected

Resurrection vs. Reincarnation.

Reincarnation is not completely new in IT world - it was already used in MINIX many years ago:). But I am coining this new analogy for containers context. Obviously, it's up to you now to share the wisdom and make sure people know who was the original prophet!

Forget resurrection, reincarnation is a way to go!


Kubernetes Persistent Storage Hell

We've started to work on a rather complex application recently with my team at Red Hat. We all agreed it'll be best to use containers, Kubernetes and Vagrant to make our development (and testing) environment easy to setup (and to be cool, obviously).

Our application consists of multiple components where those important for the post are MongoDB and something we can call worker. The reason for MongoDB is clear - we are working with JSONs and need to store them somewhere. Worker takes data, does some works on them and writes to DB. There are multiple types of workers and they need to share some data. We also need to be able to scale (That's why we use containers!) which also requires shared storage. We want both storages to be local path (for Vagrant use case especially).

Sounds easy, right? But it's not. Here is the config objects situation:


The way you work with volumes i Kubernetes is that you define a PersistentVolume object stating capacity, access mode and host path (still talking about local storage). Then you define PersistentVolumeClaim with access mode and capacity. Kubernetes then automagically map these two - i.e. randomly match claim and volume where volume provides correct mode and enough capacity.

You might be able to see the problem now, but if not, here it is: If you have 2 volumes and 2 claims (as we have) there is no way you can be sure which claim will get which volume. You might not care when you first start your app, because the directories you provided for volumes will be probably empty. But what if you restart the app? Or the Vagrant box (and thus the app)? You cannot be sure which volume will be assigned to which claim.

This leads to an inconsistent state where your MongoDB storage can be assigned to your worker storage and vice versa.

I've found 2 related issues on github and which, if implemented and solved, should fix it. But is there a workaround?

Hell yeah! And it's pretty simple. Instead of defining PersistentVolumeClaim object and using persistentVolumeClaim key in a replication controller, you can use hostPath directly in the RC. This is how the patch looked like:

diff --git a/kubernetes/mongodb-controller.yaml b/kubernetes/mongodb-controller.yaml
index ffdd5f3..9d7bbe2 100644
--- a/kubernetes/mongodb-controller.yaml
+++ b/kubernetes/mongodb-controller.yaml
@@ -23,5 +23,5 @@ spec:
 mountPath: /data/db
 - name: mongo-persistent-storage
- persistentVolumeClaim:
- claimName: myclaim-1
+ hostPath:
+ path: "/media/mongo-data"
diff --git a/kubernetes/worker-controller.yaml b/kubernetes/worker-controller.yaml
index 51181df..f62df47 100644
--- a/kubernetes/worker-controller.yaml
+++ b/kubernetes/worker-controller.yaml
@@ -44,5 +44,6 @@ spec:
 mountPath: /data
 - name: worker-persistent-storage
- persistentVolumeClaim:
- claimName: myclaim-2
+ hostPath:
+ path: "/media/worker-data"

The important bits of Kubernetes config then looks like:

     - name: mongo-persistent-storage
       mountPath: /data/db
   - name: mongo-persistent-storage
       path: "/media/mongo-data"

How to (be a) man on Atomic Host

One major thing missing on the Atomic Host are manual pages. Not a terrible thing - you can always google for them, right? But what if you cannot? Then there is the Fedora Tools Docker image. Try this:

-bash-4.3$ alias man="sudo atomic run vpavlin/fedora-tools man"
-bash-4.3$ man systemd

You should see a manual page for systemd. Thinking about it, that's it. Nothing more you need to now about it. Simple:)


Running git on Atomic Host with Fedora Tools image

I added the Fedora Tools image to Fedora-Dockerfiles repository, as you might know from my earlier post. I'd like to introduce you to one use case for this image - git.

When I started to work more on Docker images, I started using Atomic Host(s) for testing as they boot faster and are easier to set up than classic installations. Problem was to get data in those VMs running Atomic Host as git was not present. That's where I first really appreciated the tools image.

bash-4.3# yum
bash: yum: command not found
bash-4.3# git
bash: git: command not found
bash-4.3# atomic run fedora/tools
[root@localhost /]# cd /host/home/vagrant/
[root@localhost vagrant]# git clone
Cloning into 'Fedora-Dockerfiles'...
remote: Counting objects: 2189, done.
remote: Compressing objects: 100% (9/9), done.
remote: Total 2189 (delta 3), reused 0 (delta 0), pack-reused 2180
Receiving objects: 100% (2189/2189), 915.13 KiB | 901.00 KiB/s, done.
Resolving deltas: 100% (1014/1014), done.
Checking connectivity... done.
[root@localhost vagrant]# exit
bash-4.3# ls
Fedora-Dockerfiles sync

It's simple, right? You can see there is neither yum/dnf, nor git on the host, but still, I was able to clone the repository from Github very easily. The important thing to notice is the path I cd'ed to: /host/home/vagrant. You can see /host prefix there. That's where the host's filesystem is mounted and where I can access it and modify it.

You can review the docker run command for the tools image f.e. with this command:

bash-4.3# docker inspect --format='{{.Config.Labels.RUN}}' vpavlin/fedora-tools
docker run -it --name NAME --privileged --ipc=host --net=host --pid=host -e HOST=/host -e NAME=NAME -e IMAGE=IMAGE -v /run:/run -v /var/log:/var/log -v /etc/localtime:/etc/localtime -v /:/host IMAGE

Obviously, you can do more, not just clone the repo - you can run commit, push, checkout or anything else the same way.

Fedora Tools Docker image

I got this request from my colleagues if there is something like Red Hat Atomic Enterprise Tools container image available for Fedora or CentOS. The answers was no, there isn't, thus I started to work on it. I'd like to tell you what it is and why do I invest my time into it.

First of all, Fedora Tools image is meant to be used mostly on Atomic Host as there is no way to install missing tools with yum or dnf. We could create tons of small images each containing a single tool. But that would a) make it hard for users to find all the tools, b) consume more space then a single image if you decide to use many (all...) of them, c) be hard to maintain.

These 3 reasons lead us to create a single image containing big number of tools important to sysadmins, performance analysts, or just users that need man pages on Atomic Host. This image is pretty big (more than 1 GB), but can be pretty useful.

Current version of the Dockerfile can be found in Fedora-Dockerfiles repository. You can find the list of additional packages (to what's already in a base image) starting on line 13.

The basic information on how to use the Fedora Tools Docker image can be found in README file and I hope to provide more how-to's here soon:).

I've set up an automated build as vpavlin/fedora-tools under my namespace on Docker Hub. To try the image, you can do:

atomic run vpavlin/fedora-tools


Jak jsem skoro zazdil InstallFest 2015

To máte tak, spousta práce, trocha nepozornosti a pozvánka na víc akcí (skoro) najednou. Tenhle koktejl okolností způsobil, že jsem byl poměrně dlouhou dobu přesvědčen, že InstallFest 2015 se koná přístí víkend (tedy 14. a 15. března). To si tak ve čtvrtek večer projíždíte Twitter a najednou zmínka o tom, že placky už jsou připraveny na sobotu. Sobotu? Jako tuhle sobotu? Hmm..

Snímek z 2015-03-07 17:58:04

A fakt že jo! No co, jdete spát a říkáte si: "Slajdy udělám zítra v práci, to bude hned." Jenže v práci furt někdo otravuje, něco chce, takže uděláte prd. Tak prý doma, večer. Jenže to se vypravíte na jídlo a pivo. Teda hned po tom si aspoň ráno koupíte jízdenku;). Tak fajn, slajdy se spáchají ráno před odjezdem. Ráno se vyštracháte z postele, koukáte na prázdnou prezentaci a říkáte si: "Co jsem to těm lidem vlastně chtěl říct, když jsem tu prezentaci posílal?" Něco spatláte a pak strávíte ještě půlku cesty dolaďováním a přemýšlením, co jste to vlastně ráno měli v hlavě.

Takže slidy by byly, co demo? Hmm, jak to znám, na demo nedojde a když, bude jiné v závislosti na dotazech. To snad ani nemá cenu chystat;). A měl jsem rpavdu, nemělo!

Klapka, jedem...

"Dovolte mi, abych vás přivítal na své přednášce. Na úvod se vás chci zeptat...ale co to plácám. Tak já se asi představím, co?" Jak vidíte, začal jsem zkušeně, tedy chci říct zmateně. Ovšem tu otázku jsem položil: "Kdo jste slyšeli o Dockeru před tím, než jste si přečetli název téhle přednášky?" Skoro všichni, fajn. "Kdo jste si ho nainstaloval?" zněla další otázka - asi 4 ruce. Uff, to zase budu plácat kraviny. Tak a poslední dotaz: "Ok, kdo jste používali kontejnery ještě před Dockerem?" Tři ruce, sakra, tak tyhle lidi ignorovat, když se budou ptát..ti jsou určitě chytří a ví toho víc než já!

Jak jsem si myslel na začátku, na pořádné demo nedošlo. Ostatně moje jediné "pořádné" demo je to, co jsme popsal v článku Running services with Docker and systemd. Takže se na něj mrkněte a demo si zkuste sami;) Třeba se vám taky rozbije, jako by se to určitě stalo mně.

Také, jako už tradičně, se přednáška zvrhla na Q&A session, kde jsem dostával záludné otázky a poskytoval jsem na ně v zásadě nesouvisející odpovědi. (Jsem v tom čím dál lepší!) Ale musím říct, že jsem si to s vámi, InstalFesťáci, užil. Hezky jsme pokecali. A navíc jste se mě nikdo nezeptal na síťování, čehož si velice cením!

Upřímně, slajdy samotné vám asi moc neřeknou, ale tady je máte - na konci jsou nějaké odkazy, tak třeba budou užitečné. Přednáška se očividně natáčela, takže jakmile bude, přihodím ještě video. A teď už dobrou noc, jdu si pustit nějaký film, když už si konečně Student Agency obnovilo výběr, a nejspíš si i trochu schrupnu. Ještě jednou díky za účast!


Jak jsem zjistil, video se přímo streamovalo na Youtube, takže tady je záznam přednášky:

My Docker Helpers

I work with Docker almost all the time in my job at Red Hat. Building, running, inspecting containers... Writing same long commands every time you want to run a container or get it's IP starts to annoy you quickly. That's why I started writing small helpers in the form of bash functions which are loaded through .bashrc and thus can be used from cmd line easily.

You can find them in my docker-tools repository but let me introduce them a bit.


If you load/import/build images often, it happens that you end up having a bunch of <none> named images in your docker images output. The above commands removes them all.


I use this mostly in VMs where I am limited in terms of disk space - every container, especially when you test f.e. if yum install works, eats some space and this command lets you remove them all quickly.

dr fedora
docker run --name tmp0 -it --rm fedora bash
dr fedora cat /etc/os-release 
docker run --name tmp0 -it --rm fedora cat /etc/os-release

This dr command is probably my favourite. It runs bash in the given image with arguments I use the most. You can also specifiy a command to run if you wish so.


Simple alias for docker load command with the advantage of being able to load from default directory so you can just give it a file name and it looks in the predefined folder.


The most awesome thing about using functions instead of just aliases is that you can add whatever logic you like. So my de command (representing docker exec) can be called with a container id/name and command  - same as docker exec. But it can also be called without command which then defaults to bash and also without container id/name which default to the last entry in docker ps output. If you want to skip specifying container, but still want to use different cmd than bash, use following syntax:

de "" rpm -qa

I don't use next command as often as those above but I still like it a lot - it let's you print IP address of any container. If container id/name is not specified it uses the same logic as de.


Last command I have on my list at the moment is dk and you could maybe guess - yes, it's docker kill and it also provides the same logic as the two above.


Do you have more aliases/ideas? Let me know I am happy to make my list richer!

Delete an Image from Private Docker Registry

Have you ever wondered how to remove repositories/tags from your private Docker registry? It's simple according to Docker registry API specs. So let's try this

yum -y install docker-registry
sudo systemctl start docker-registry
docker pull fedora:21
docker tag fedora:21 localhost:5000/fedora:21 
curl localhost:5000/v1/repositories/fedora/tags/21

You should see an image id printed to your terminal. Now let's delete the image...

$ curl -X DELETE localhost:5000/v1/repositories/fedora/tags/21

To be clear - it does not remove the image/layers data - it just removes the reference from fedora:21 tag to the image id (i.e. data). If there is any other tag referencing the data, they will still be accessible.

Anyway, in some cases it is useful to be able to remove this reference. I run a private registry with 337 images (multiplied by few tags for every image) and I sometimes found myself in a situation where I pushed an image with wrong tag or I just wanted to stop people from pulling a specific image. I wrote a small bash script for these occasions - The usage is simple NAME[:TAG]

Which means for our fedora example

$ ./ localhost:5000/fedora:21
Do you really want to untag "834629358fe214f210b0ed606fba2c17827d7a46dd74bd3309afc2a103ad0e89"? [y/N]: y
Image library/fedora:21 removed from localhost:5000

Firstly it checks if the image exists, then it asks for confirmation of removal and then it calls the previously shown curl command to delete the reference. I also have a simple "Docker Registry Garbage Collector" under development which goes through the docker-registry directory and moves unreferenced layers away (where you can delete them later). But that's going to be a topic next time:).

Running services with docker and systemd

I have described how to run systemd in a Docker container on Fedora in previous article but didn't give you any "real" example of how to actually use it. I guess you find it easy to figure out your own examples but let me show you mine.

WordPress & MariaDB

I like to use WP and MariaDB as example applications when I give a talk about containers. It's quite simple setup and at the same time uses some nice features of docker (f.e. links or port mapping). If you just want to see my Dockerfiles, please go to this github repository. If want more babbling, read on.

The MariaDB Dockerfile is forked from Fedora-Dockerfiles and I guess it will be "merged" there soon:). First few lines are quite boring - yum update/install. Next line I really like. How to start MariaDB properly..hmm...hey, let's use what packagers came up with - mariadb.service file.

RUN systemctl enable mariadb.service

This really presents the beauty of using systemd inside containers. Single line tells the init that we want to start the service on the container start. Easy, clean, awesome. Then some "stuff" follows. I let it up to you to figure out what it actually does - everybody likes homeworks, right?

You can also check my WordPress Dockerfile in the same repository. It's a bit longer but the most important line for this case is again enablement of the service - in this case httpd.

RUN systemctl enable httpd.service

If you build those two images...

docker build -t vpavlin/mariadb mariadb/
docker build -t vpavlin/wordpress wordpress/

You can run them with these commands

docker run -it --rm -v /sys/fs/cgroup:/sys/fs/cgroup --name mariadb vpavlin/mariadb
docker run -it --rm -v /sys/fs/cgroup:/sys/fs/cgroup -p 80:80 --name wordpress --link mariadb:mariadb vpavlin/wordpress

To describe these commands I'd say: Run a container with stdin and out attached to my tty, volume mount /sys/fs/cgroup for systemd, name containers mariadb and wordpress (respectively) and link mariadb to wordpress (which basically mean tell WP how to connect to mariadb). Oh, and map port 80 of WP container to host's port 80.

When you hit http://localhost in your browser, you should see a WP installation page.

Snímek z 2015-02-24 23:35:53

Any questions?:)