Don't you just love when shit hits the fan and your JVMs start going GC mental and OOMing??
Well that's pretty much what just happened. It seems as though there was a code change at some point today that caused one of our JVMs to (excuse the crassness) eat shit and die. Well, in this case, the shit eating (Garbage Collection) didn't work well and it took a few hours for it to get to a bad point.
We've reverted for now to a build prior to any changes that were made today. This seems to have sorted the problem out.
Next steps is for the fine Dev folks to take a look at their commits from today and re-evaluate them. Good thing we have great Devs and this will be sorted out real quick.