I think I found why I was running out of memory in the big loop. OpenCV. Maybe cos it's wrapping C++ in Java. You can call release() on some objects to release it's memory.

Try Java Mission Control and/or Eclipse Memory Analyzer - helps find objects causing memory leaks. Or for stuff like seeing if you've got too many instances of something you only expect to have one of.

Eclipse!? No thanks :S What can I use with IntelliJ?

According to Stack Overflow it's because Java doesn't see the memory being used by the C++ classes so it doesn't know it's using as much memory as it is.

I tried another test, writing an image in a big loop and it crashed using too much memory. If I call release on the image before each loop finishes, the memory use doesn't go up and it works.

You can point Eclipse MAT at any JVM, local or remote. You don't need to be developing with Eclipse JDT to use it - the default download is a standalone non-IDE version.

It can't be used to analyse C/C++, which isn't surprising since it'll be a separate process and different memory structure.

Might be able to use tools from NirSoft or SysInternals to do that, if necessary, but probably not to the same degree of detail/interactivity.