What is kjournald and why it’s using 99% of IO? What about noatime?

I have noticed this process in iotop using a lot of CPU/IO time and I started to wonder what is it doing. It turns out this is a journaling process of ext3 partitions: http://serverfault.com/questions/236836/kjournald-reasons-for-high-usage
Very often it’s related to not using noatime mount option.
Generally this mount attribute is used to lower the IO load, but it prevents kernel from setting last access time to a file (which is rarely needed):
UUID=9a95d8ab-78c7-53de-b8bc-a8a2340d6250 /mnt ext3 defaults,noatime 0 2
This is a example of a /etc/fstab line (more here:  http://tldp.org/LDP/solrhe/Securing-Optimizing-Linux-RH-Edition-v1.3/chap6sec73.html )
I recommend setting noatime everywhere you can, especially in high-read workloads – for more details take a look at this question: http://serverfault.com/questions/47466/drawbacks-of-mounting-a-filesystem-with-noatime
Take a look at following snippet and see that “Access” doesn’t change when accessing file:
[email protected]:~$ touch abcd
[email protected]:~$ stat abcd
  File: `abcd'
  Size: 0           Blocks: 0          IO Block: 4096   regular empty file
Device: 802h/2050d  Inode: 27705348    Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/wlangiewicz)   Gid: ( 1000/wlangiewicz)
Access: 2013-02-18 19:56:00.000000000 +0000
Modify: 2013-02-18 19:56:00.000000000 +0000
Change: 2013-02-18 19:56:00.000000000 +0000
[email protected]:~$ cat abcd
[email protected]:~$ stat abcd
  File: `abcd'
  Size: 0           Blocks: 0          IO Block: 4096   regular empty file
Device: 802h/2050d  Inode: 27705348    Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/wlangiewicz)   Gid: ( 1000/wlangiewicz)
Access: 2013-02-18 19:56:00.000000000 +0000
Modify: 2013-02-18 19:56:00.000000000 +0000
Change: 2013-02-18 19:56:00.000000000 +0000
[email protected]:~$ echo a > abcd
[email protected]:~$ stat abcd
  File: `abcd'
  Size: 2           Blocks: 8          IO Block: 4096   regular file
Device: 802h/2050d  Inode: 27705348    Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/wlangiewicz)   Gid: ( 1000/wlangiewicz)
Access: 2013-02-18 19:56:00.000000000 +0000
Modify: 2013-02-18 19:56:28.000000000 +0000
Change: 2013-02-18 19:56:28.000000000 +0000
[email protected]:~$ cat abcd
a
[email protected]:~$ stat abcd
  File: `abcd'
  Size: 2           Blocks: 8          IO Block: 4096   regular file
Device: 802h/2050d  Inode: 27705348    Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/wlangiewicz)   Gid: ( 1000/wlangiewicz)
Access: 2013-02-18 19:56:00.000000000 +0000
Modify: 2013-02-18 19:56:28.000000000 +0000
Change: 2013-02-18 19:56:28.000000000 +0000

Fixing MySQL replication that is stuck

If your MySQL replication is stuck for example on
CREATE / ALTER TABLE ...
(or few others) because for example for the tables that already exist on the slave, there’s quick fix that can be applied on MySQL slave:
SET GLOBAL SQL_SLAVE_SKIP_COUNTER=1; START SLAVE;
This will force slave to skip one command (that is causing it to stop) and then replication is started back again.
Depending on the root cause of the problem of stopped replication there are other solutions – but this works in the case described above.