Tales of an IT Nobody

devbox:~$ iptables -A OUTPUT -j DROP

MySQL CPU maxing out due to leap second, and AWS US-E1 outage June 30, 2012

Wow, US-EAST-1 has the worst luck doesn’t it?

I had CPU consumption alerts fire off for ALL of my AWS instances running Percona Server (MySQL).

I couldn’t for the life of me figure it out – I re-mounted all but the root EBS volumes, restarted the services and ensured there was no legitimate load on all of them, yet MySQL stuck to 100% consumption on a single core on all of the machines. I tried everything, lsof, netstat, show processlist, re-mounting the data partitions.

WIERD. It turns out however, AWS was not be the cause of this one, even in light of the recent AZ issue in EAST-1.

My issues started sharply at 00:00 UTC. Hmm, odd since we’re having a leap second today!


A quick touch to NTP and we’re as good as gold.

This server had many cores so it was still running fine (and it was the weekend) – Seems like bad things happen at the least opportune times, like when you’re out on a date with the wife and the baby is with grandma…

At any rate, it’s amusing to watch the haters hate that clearly don’t understand the concept of AWS and the AZ (Availability Zones). Funny to watch them all scoff and huff about how they run a better data center out of their camper than AWS.

If anything, I want to see a good root cause analysis and maybe some restitution for the trouble.

No Comments on MySQL CPU maxing out due to leap second, and AWS US-E1 outage
Categories: aws servers

MySQL COALESCE(), UNION behavior on zerofilled columns June 28, 2012

I’ve just filed a (potential) bug report, depending on your views for MySQL’s COALESCE() behavior:

http://bugs.mysql.com/bug.php?id=65763

If you have a zerofill column and perform the COALESCE() function on it, the leading zeros are truncated.

As I mention in the bug report, this may not matter to most – but it does change the output one would expect and is worthy of notice or a mention in the official documentation in my book.

Test scenario 1:

INSERT INTO coalesceTest (paddedNumber) VALUES (5), (10), (200), (3000);

SELECT COALESCE(paddedNumber), paddedNumber from coalesceTest;

Results:

Test scenario 2:

Results:

 

No Comments on MySQL COALESCE(), UNION behavior on zerofilled columns
Categories: mysql servers

On MySQL: The latest, far-reaching password circumvention June 11, 2012

By now, everyone has, or will be hearing about this issue.

While it’s an extremely simple hack and covers (dare I say the majority) of MySQL installation version. Let’s not forget to finish reading the entire disclosure:

From the disclosure:

But practically it’s better than it looks – many MySQL/MariaDB builds are not affected by this bug.

Whether a particular build of MySQL or MariaDB is vulnerable, depends on
how and where it was built. A prerequisite is a memcmp() that can return
an arbitrary integer (outside of -128..127 range). To my knowledge gcc builtin memcmp is safe, BSD libc memcmp is safe. Linux glibc sse-optimized memcmp is not safe, but gcc usually uses the inlined builtin version.

As far as I know, official vendor MySQL and MariaDB binaries are not
vulnerable.

Regardless, it’s a stupid simple test to see if you’re vulnerable or not so fire one up!

I just tested 5 gcc compiled hosts (mostly pre-5.5.23) and none of them were vulnerable. But regardless, maybe it’s time to re-compile ;)

No Comments on On MySQL: The latest, far-reaching password circumvention
Categories: mysql security servers

Note to self: Digest hashing and crytpological hashing are birds of a different feather June 7, 2012

Over decades of enhancements in computer science, there’s always a revolution going on in cryptography and hashing, MD5, SHA1 yesterday, SHA256/512 today.

As a programmer, it’s sometimes hard to avoid the back and forth talk about how algorithm A is inferior to algorithm B, and forget how hashing can be used in two ways.

Let this be a reminder to myself if anything – At it’s core, the intention for hashing is to take data; and based on it, generate a unique string so unique the value will never be (realistically) generated again using different data.

The cryptology part is the level of difficulty involved in reversing any said ‘one way hash’.

Don’t let the consistent bad-press of SHA/MD5 make you feel they’re insufficient for unique identifiers.

No Comments on Note to self: Digest hashing and crytpological hashing are birds of a different feather
Categories: programming rant