Tales of an IT Nobody

devbox:~$ iptables -A OUTPUT -j DROP

Securing Elasticsearch – Part 1 May 28, 2014

The most frequently asked question for ElasticSearch and security is “how do I require login”?

Once you’ve answered and implemented the answer to that question; a larger, truly more troublesome issue looms. The same principals used to secure ElasticSearch; typically a proxy fronted by Apache/nginx use various auth techniques. If you know what you’re doing, you have different endpoints in that proxy for controlling who can do GET/POST/DELETE requests, possibly pre-determining the index and type.

If you REALLY know what you’re doing, you’ll be far more concerned over the payload.

While reading through the documentation, I was surprised; no, SHOCKED to see that ElasticSearch ships with a security flaw as severe as remote code execution as an intentional feature through the dynamic scripting component of a body payload for ES.

If you’re responsible for running ElasticSearch servers…

You must examine how queries are sent into the server. If your web developers are sending near-verbatim DSL queries to ElasticSearch without any further filtration except auth and index constriction, please read this.

A malicious person could modify the payload directly to read files directly off of your filesystem and serve them up in ES.

The article contains a proof of concept (POC) link – simply download and modify the file and point it to your ES server and see if you’re vulnerable.

I think in most cases, dynamic remote scripting isn’t a big deal to turn off. So I’d strongly suggest following the advice on this page:  script.disable_dynamic: true


Stay tuned for Part 2 of a more obsessive approach to securing ElasticSearch for use on public search interfaces.

2 Comments on Securing Elasticsearch – Part 1
Categories: nosql security servers

Why are we spending so much time refuting? July 5, 2012

There’s a nice juicy war going on in the ‘data / web’ sector, that seems more heated than I can remember.

It essentially boils down to sensationalist claims from the likes of MongoDB and MemSQL, which in turn draw refuting remarks from industry professionals that are typically embedded with RDBMS technologies.

The typical responses to these new ‘hipster’ systems are usually transaction/consistency centric – as that’s where the RDBMS systems shine – they can perform wonderfully while being ACID compliant.

Or in the case of Node, refuting the ‘Apache doesn’t have concurrency, node is better’ arguments. I have a hunch 99% of the Node fanboys have a damn clue how capable Apache itself is.

There’s also things like Node.js that rub the seasoned people the wrong way, perhaps it’s the sensationalism without actually proving anything? (Check the first few comments) Or the utter lack of security focus? (That’s what bugs me) – I also think it has to do with their approach to enter the market: guns blazing, criticizing other solutions and hoisting their own as THE single option with more tenacity than appropriate for such an immature project. Guys in the trenches can’t stand that crap, we know it’s just another tool to get the/(‘a’) job done in a particular scenario.

But really, I think about how much time is wasted on these subjects going back and forth, so let’s stop wasting time. Be open minded to the new technologies as tools for a particular job and stop making all or nothing stories out of future tech, like it or not – we all have to share the same space.

No Comments on Why are we spending so much time refuting?

Worthy of distribution: Your cell phone records July 2, 2012

This is just too cool, and too perfect a testament for how you can derive a lot through data and social networking.

Via the F-Secure labs blog: http://www.zeit.de/datenschutz/malte-spitz-data-retention

Shows an extremely well put together example of combining a tidbit of privileged information with social networking and how this truly is a brave new world.

Map has it all.

No Comments on Worthy of distribution: Your cell phone records
Categories: linkspam security

On MySQL: The latest, far-reaching password circumvention June 11, 2012

By now, everyone has, or will be hearing about this issue.

While it’s an extremely simple hack and covers (dare I say the majority) of MySQL installation version. Let’s not forget to finish reading the entire disclosure:

From the disclosure:

But practically it’s better than it looks – many MySQL/MariaDB builds are not affected by this bug.

Whether a particular build of MySQL or MariaDB is vulnerable, depends on
how and where it was built. A prerequisite is a memcmp() that can return
an arbitrary integer (outside of -128..127 range). To my knowledge gcc builtin memcmp is safe, BSD libc memcmp is safe. Linux glibc sse-optimized memcmp is not safe, but gcc usually uses the inlined builtin version.

As far as I know, official vendor MySQL and MariaDB binaries are not

Regardless, it’s a stupid simple test to see if you’re vulnerable or not so fire one up!

I just tested 5 gcc compiled hosts (mostly pre-5.5.23) and none of them were vulnerable. But regardless, maybe it’s time to re-compile ;)

No Comments on On MySQL: The latest, far-reaching password circumvention
Categories: mysql security servers

GitHub hacked, and private repositories March 5, 2012

And this is precisely why albeit ‘nifty’, storing your private/proprietary code in a ‘private repository’ on the likes of GitHub / Bitbucket is a generally poor idea. – Keeping your code in SCM behind closed doors isn’t difficult. I find it very troublesome (annoying) to see how many people can’t function using Git without GitHub. (If you don’t believe me, look back several months to the PHP.INTERNALS discussion about moving to new SCM)

GitHub’s response was far too gracious to this guy. I understand the power he had, and behaved responsibly for it. But you could have just as easily made other communication attempts.

GitHub wants to stay afloat by having paying customers? You OWE your paying customers much, much more you do this bozo. Ban him. File a criminal complaint.

It seems that the majority of people posting on the Blog posts regarding this disclosure are “still happy customers” and are generally “ok” with it.

I have three categories for these kinds of people:

1. FSF (Free software foundation)-style hippies
2. “Younger” coders who are pushovers (limited sight)
3. Hacker-types who feel the same way to convey that something is insecure: via “lulz”.

p.s.: I should add, you can’t draw a comparison to the past breaches to Apache.org, or MySql.com because the resulting risks were much less than this. Comparing it to kernel.org’s intrusion would be a better fit, as that was more serious and they went dark for almost a full month reloading everything and thoroughly investigating.

No Comments on GitHub hacked, and private repositories
Categories: git rant security

Disable PHP 5.4’s built-in web server, while keeping CLI … February 6, 2012

Administrators: Don’t get blind-sided by PHP 5.4’s CLI web server!

I’ve gone over a similar issue like this before regarding the likes of git/hg. While those are developer tools and are less likely to be present on a production machine.

PHP 5.4 is jumping on the bandwagon to include a ‘cute’ little internal server – which is enabled by default.
The ‘everything needs a standalone server’ thing is starting to get on my security nerves feel silly.

It has limited use, and most developers will have limited use for it due to it’s lack of mod_rewrite (and equiv.) behavior … The worse part is: You can’t disable it if you want to keep cli (e.g.: no pear!)
Wish I spoke up on the list!

Anywho, here’s a hob-knobbed patch (for PHP 5.4.0RC6) that will change that for you.
(GNU/*nix only!) The patch adds a new configure option ‘–disable-cli-server’.

Download the patch here: patch-php5.4.0RC6-no-cli-server.diff
Place it in the PHP source base directory.

In the future I’ll plan on formalizing this patch and propose it in php.internals when I get a chance to make the windows part of the patch.


2 Comments on Disable PHP 5.4’s built-in web server, while keeping CLI …
Categories: linux php security servers

Observations: Google’s new Terms of Service January 27, 2012

The new TOS and Privacy Policy documents from Google are a welcome change, reducing 60 individual ones into a standard, global set is a much better idea for understanding’s sake.

Observation 1:

We may review content to determine whether it is illegal or violates our policies, and we may remove or refuse to display content that we reasonably believe violates our policies or the law. But that does not necessarily mean that we review content, so please don’t assume that we do.

Usng Our Services http://www.google.com/policies/terms/#toc-services

I get what they’re saying, but the wording seems a little humorous if you don’t hone in on ‘necessarily’.

Observation 2:

We provide information to help copyright holders manage their intellectual property online.

Privacy and Copyright Protection http://www.google.com/policies/terms/#toc-protection

Odd, this leaves some open questions to what information they provide … are they helping police things “SOPA style”?

Observation 3:

Some of our Services allow you to submit content. You retain ownership of any intellectual property rights that you hold in that content. In short, what belongs to you stays yours. 

 That’s neat… but wait, the next paragraph:

When you upload or otherwise submit content to our Services, you give Google (and those we work with) a worldwide license to use, host, store, reproduce, modify, create derivative works (such as those resulting from translations, adaptations or other changes we make so that your content works better with our Services), communicate, publish, publicly perform, publicly display and distribute such content.

Your Content in our Services http://www.google.com/policies/terms/#toc-content

Uh… ok…

I don’t have any intent on copyright infringement, so the first two don’t bother me, but the third one leaves me with some questions …

No Comments on Observations: Google’s new Terms of Service
Categories: google security

PHP Vulnerability – DJBX33A – Hash table collisions January 14, 2012

Trickling through my RSS feeds this morning was an article with quite the topic “PHP Vulnerability May Halt Millions of Servers“.

In a nutshell: A modest size POST to almost all PHP versions in the wild (Sans 5.3.9+) are in danger of an extremely simple DoS.

The vulnerability exploits the PHP internal hash table function (responsible for managing data structures) – more specifically: the technique used to ‘hash’ (generate a key for the hash table) the key for a key=>value relationship.

Here’s the informative part regarding PHP’s problem in the security advisory for this:

Apache has a built in limit of 8K max request length (that is, maximum size in request URL) by default.
Can the damage from an 8k request (this affects GET) – really cause the mentioned DDoS attack on reasonable hardware?

Additionally – PHP has a limiter on POST data too: max_post_size.
It’s this configuration variable in particular I think should be put in the limelight.

max_post_size is a run-time/htaccess configurable directive that maybe we don’t respect like we should.
Often, administrators (myself included) just tell php.ini to accept a large POST size to allow form based file uploads – It’s not uncommon to see:

– in almost any respectable setup.

Perhaps we should evaluate the underlying effects of this setting; maybe it should be something stupidly low by default (enough to allow a large WYSIWYG CMS article’s HTML and a bit more? 32K?) – and then delegate a higher limit using Apache configuration.

Caveat: these settings are PER DIR meaning:

  • .htaccess use is limited, you can’t set the php_value in a .htaccess with a URL match – you’re stuck using a context sensitive .htaccess (within a dir) or use thedirective – this won’t work for people using front controllers through a single file on their websites/apps.
  • Modifying the actual vhost/host configuration is a sound bet – you can do Location/File matching and set these at will; for situated web apps, this may be a feasible decision to take whitelist or blacklist approach on uploader destinations.

More resources:

  • Here’s the video that thoroughly covers the vulnerability – I’ve shortcut it to their recommended mitigation (outside of polymorphic hashing):
  • A full blown rundown, including proof of concept (USE AT YOUR OWN RISK!)
  • A string of hash collisions targeting DJBX33A for vuln testing (PS: Firefox seems to struggle with this in a GET format, Chrome doesn’t, odd!)
2 Comments on PHP Vulnerability – DJBX33A – Hash table collisions
Categories: php security servers

Worthy of distribution: Cloud analogy November 5, 2011

This post on Beyond Bandwidth seems to summarize some of my feelings about cloud computing – it’s best thought of as an outsourcing task for the most part; Although the benefits of something like an extra DNS server are a bit more than an ‘outsource benefit’; but you get the idea:

Cloudy analogies with a chance of illusion

No Comments on Worthy of distribution: Cloud analogy
Categories: security servers

Is there a hacking campaign against open source? September 26, 2011

Linux.com, kernel.org, mysql(twice this year), wordpress and php have all reported breaches of some sort this year. Is there some sort of campaign against these ‘high profile’ open source projects? It’s starting to feel like it, to me.

The more hands you get in the pot, the more nervous you should get as an administrator. System issues stem from more than password change frequency and difficulty – stale keys and giving access to folks that shouldn’t have access happens.

I also feel isolation or ‘separation of concerns’ is a tactic that is pushed aside in the name of maxing out a system, more often than not this stop gap would save a lot of trouble. Apache’s ability to mitigate concern from last year’s breaches is a good example of isolation, they had a fairly sophisticated break in and the repercussions weren’t as vocal as the ones from this year.

There doesn’t seem to be sufficient coverage of this MySQL hack right now – how sure are we this isn’t a sample set from a compromised browser as opposed to the site?

I hope there will be continued disclosure so everyone can learn something extra to safeguarding themselves.

While it doesn’t feel right to ream MySQL (at all, or at this point of the news) I have some initial thoughts I just can’t shake:

  1. If MySQL was ‘hacked’: Infiltrated earlier this year; you made no extra measures on a wider scope? really?
  2. Why the hell is your web/any cluster accessible without a VPN? It sounds like they’re selling shell access directly to the host/s..

 C’est la vie

No Comments on Is there a hacking campaign against open source?
Categories: mysql security servers

The inherent risks of ‘daemonize’ features in developer tools – Git, Mercurial (hg) September 24, 2011

A handful of tools such as mercurial, git, (soon PHP – which chances are will be it’s own binary) have their own ‘daemonize’ functionality.

Whatever your reasons – if you want to disable these; there’s little to no help in figuring out how… til now…

If you want to disable Mercurial’s hg serve:
Open the file (Your python install path may differ, but this should give you an idea of what to search for)


Find the function ‘create-server’ and add ‘sys.exit()’ in the first line:

How to verify this works:

1. Before patching – run ‘hg serve’ from a mercurial repository. It will report the port number and remain active in console.
2. After patching – ‘hg serve’ from a mercurial repository will simply exit and say nothing.
3. netstat, ps -A ux |grep ‘hg serve’

If you want to disable git’s git daemon:
This one is probably the easiest of the two: find and ‘chmod a-x’ (remove execute permissions) from the ‘git-daemon’ binary on your system – mine is in /usr/libexec/git-core. You can also relocate it somewhere in-accessible.

How to verify this works:

1. Before relocating/removing/chmodding – run ‘git daemon’ – your console will remain active as if it’s listening. (You can try a base dir for a proper daemon setup if you want …)
2. After relocating/removing – run ‘git daemon’, you’ll get an error saying there are insufficient privileges, or in the case of relocating/removing you’ll see “not a git command”.
3. netstat, ps -A ux |grep ‘git daemon’

No Comments on The inherent risks of ‘daemonize’ features in developer tools – Git, Mercurial (hg)
Categories: git hg linux security servers

Kernel.org, linux.com down, still… also, Git! – Updated! September 11, 2011

With news breaking about the compromised systems for kernel.org, linux.com, which are sites are “down for maintenance”. Completely – and it’s been this way for many days now. (Kernel.org since the 28th)

I think it’s safe to say the range and scope of the issues are pretty disappointing – the longer these systems stay down the more obvious it is that the damages are probably higher than perceived before; I’m having a hard time saying the administrators of these groups are just this slow at being cautious. (Especially the ‘forward facing  ‘ hosts)

(Segway to Git)

It’s interesting to note that a LOT of bullets with the security breach are dodged with the parity cryptography used by Git. Pretty cool! (Linus seems to be flirting with the idea of using Github for latest kernel developement.)

After keeping an eye on the “choosing a DVCS” discussions for PHP, a lot of people are in favor of leveraging Git purely because of Github – whereas Mercurial, something we use at M State has been around for a bit less – has a stronger, more mature toolset (Albeit, a bumpy ride for sure!); and, from my standpoint – better cross platform implementation. The speed differences are somewhat minimal.

The allure of Github is the social endeavour; this has fueled a much more active community (compared to Bitbucket, less ‘social-ly’). It looks like Git has finished clobbering competitors like Bazaar and Perforce and finally, I’m willing to throw in the towel for my support of Mercurial and say there’s little room for traction for Hg.

The programming world is a lot of work to follow =\

(Also: It’s September 11th – take a moment to reflect – cast some sympathy and reverence for the lives lost! ) 

UPDATE: 2011-09-21- 
linux.com has been updated and now states that they will be “restoring service shortly” – different from the original FAQ page they had up about the breach.

I’d expect kernel.org to follow.

After over half a month (almost a month for kernel.org) of being down with poorly communicated “maintenance” pages. I hope there’s a fallout for the culprits – and I hope the maintainers of those domains take a more serious approach to how they handle this situation next time…

No Comments on Kernel.org, linux.com down, still… also, Git! – Updated!
Categories: linux security

Worthy of distribution: Reset root MySQL password July 18, 2011

Oh snap! Need to reset your mysql root/admin (or any?) MySQL password? Well, you’ll need root and control over MySQLd to some extent, but this is worthy of a rainy-day bookmark indeed: http://mysqlpreacher.com/wordpress/2011/03/recovering-a-mysql-root-password-three-solutions/

No Comments on Worthy of distribution: Reset root MySQL password

/usr/bin/chage – Sending emails when a pasword expires, or is about to June 6, 2011

There’s a lot of scripts out there that do this but they either don’t revolve around /etc/shadow enough or they’re sloppy.

Here’s my spin on a script for nightly cron that will parse /etc/shadow and send out emails based on the per-user values. It’s resistant to garbage dates (99999 ‘expiration’ dates).

Below is my best attempt at making the script ‘cohesive’ in this layout, however you can find the script here as well.

No Comments on /usr/bin/chage – Sending emails when a pasword expires, or is about to
Categories: linux security servers

Shortcut to a directory with a bat file and a sub directory containing the same name May 31, 2011

Check this out.
Make a directory structure like this:


Fill the .bat file with something creative… (Non destructive)

Now, make a shortcut from anywhere to Dir/AnotherDir – lemme guess, it tries to name the shortcut after the bat file? Odd! Rename it in the prompt then.

Save your shortcut and try to use it. Lemme guess, executed the bat file?

See below if you’re too lazy to try ;)

No Comments on Shortcut to a directory with a bat file and a sub directory containing the same name
Categories: security windows

Amazon AWS – The risk of using a cooked AMI May 11, 2011

Straight from the horses mouth; I no longer use this AMI – but the only ones I’ve used are Debian EBS and SLES … Fortunately I already went through authorized_keys on the one I do keep around.

People take AWS services seriously – but the AMI sharing always set off a flag for me. “Community AMI?” – No thanks! (Unfortunately the only choice for people who don’t want to – or do not have the time to make their own AMI they can trust).

Dear AWS Customer,

We are aware that a public Amazon Machine Image (AMI) in the Amazon EC2 US East (Virginia) region includes a public SSH key that could allow the AMI publisher to log in as root. Our records indicate that you have launched instances of this AMI.

AWS Account ID:  [REMOVED]


Instance ID(s)

We are taking steps to remove the affected AMI within the next 24 hours. This will prevent launching new instances of the affected AMI, though existing instances of this AMI will continue to function normally.  For existing instances you may have of this AMI, we recommend that you migrate services to new instances based on a different AMI.

While you are migrating your services to a new instance, we also recommend that you identify and disable unauthorized public SSH keys. To do so, you will need to remove any unrecognized keys from your running instance(s). Note that public SSH keys are not guaranteed to be in the ‘/root/.ssh/authorized_keys’ file. The following command will locate all of the “authorized_keys” files on disk, when run as root:
       find / -name “authorized_keys” -print -exec cat {} \;

This command will generate a list of all known “authorized_keys” files, which you can then individually edit to remove any unrecognized keys from each of the identified files. To ensure that you do not inadvertently remove your authorized keys, we recommend that you initiate two SSH sessions when starting this process for each instance. You should keep the second session open until you have confirmed that all unrecognized / unauthorized keys are removed and that you still have SSH login access to the instance using your authorized key.

If you do not use SSH to connect to your Amazon EC2 instances, we recommend that you check the security groups associated with the above instance(s) to ensure that port 22 inbound is closed to all unknown IPs. This can be done via the AWS Management Console. For detailed instructions, please check the “Using Security Groups” section of the Amazon EC2 User guide:


We hope this information is helpful.

Best regards,

Amazon Web Services Support

This message was produced and distributed by Amazon Web Services LLC, 410 Terry Avenue North, Seattle, Washington 98109-5210

3 Comments on Amazon AWS – The risk of using a cooked AMI
Categories: security servers

Apple updater today… March 10, 2011

I need music. I spend a lot of time holed up in an office with IM with my peers as my main form of human interaction.

For a long time, I’ve relied on iTunes. Things have changed – I dont want to purchase through iTunes anymore – I don’t want DRM’d music. I’ve converted everything I have to mp3’s and it’s staying that way.

The biggest beef I have with Apple – is every time they push an update, it removes icons from quicklaunch and I have to re-create. It’s a monster download, and they always push quicktime into the bundle.

Today was the last straw – I did the latest “update” from apple – after installing it brings the updater app back to show you that things were updated.

Something was amiss, and I wish I took a screenshot: ‘updater’ icon was malformed looking. The gradient was low quality and there were pink/red/orange colors – almost as if an old game engine was rendering opacity colors incorrectly.

Fast forward a few hours – I notice “Windows action center” has a notification for windows defender wanting to send samples of those icons to microsoft, under the tune of “need more information”.

By now my head has a big red flashing strobe on it – I need to check my PC at home to see if this has happened; but that’s two strikes of suspicion for me, enough to be the last straw of Apple and my already-low trust in them.

Time to find a new music player.

No Comments on Apple updater today…
Categories: rant security

Google profile images, FAIL- Worthy of distribution March 5, 2011

Most folks probably don’t know that google has updated the user profile pages look’n’feel. Including some changes to profile pictures… Most of us feel somewhat comfortable uploading an image and cropping it using the interface provided to us from sites (Granted,  you get what you ask for if you upload anything with a naughty factor) – but this is just plain stupidity, and it’s straight from our techlord google.

Basically the uploaded files that are “Cropped” are still easily visible on a google profile page, even when uploaded to a private album, and cropped to be made public. Fail.

No Comments on Google profile images, FAIL- Worthy of distribution
Categories: rant security

On: ntp, ntpd. link dump! January 14, 2011

So, in order to quickly have a (debian) machine up and running on ntp, you’re bound to do something like this ‘apt-get install ntp ntpdate’.

The problem is that this installs ‘ntpd’ too. The default configuration is to allow your server to answer to NTP queries from anywhere.

If you want to give the crackdown you’ll be somewhat frustrated with pre 4.6 config options as they’re somewhat nontraditional to what we usually see; without further ado, here’s a simple ‘link dump’ for a configuration guide.

On ntp 4.x? Guess what? Doesn’t work =[ – must be done with iptables.

Here’s the cheatsheet /etc/ntp.conf :

driftfile /var/lib/ntp/ntp.drift
server my.server.address

restrict default ignore
restrict -6 default ignore


restrict my.server.address

This will allow you to poll things, e.g.: ntpq -p; and keep everyone else from sending packets to your box either on purpose or by accident. Note: You -have- to have your ‘servers’ in restrict lines or else it’ll hang on the first poll. (Indicated by ntpq -p )

When ntp isn’t working right, this is what ntpq -p looks like:

 box:/etc# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================  .INIT.          16 –    –   64    0    0.000    0.000   0.000

Note the 0.000’s in the delay/offset/jitter – it’s also stuck on the sync request at INIT.

A properly functioning ntpq -p should look something like this:

box:/etc# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================      3 u    3   64    1    1.349  2446.01   0.000

No Comments on On: ntp, ntpd. link dump!

Time to be informed! March 2, 2010

What would you do if you received a legitimate looking email from your hosting company asking you to OPEN an SMTP relay?

That’s apparently a new style of spam (to create more spam !) targeting administrators. I’m sure there’s a handful of ‘admins’ who can get by and would more than happy to oblige their skillz in opening a relay without really thinking about how fricken nuts it sounds …


Time to separate the weak from the weaker, or is it more weak?

No Comments on Time to be informed!
Categories: security servers