What were your settings for -c, -cl, -ch? It seems odd that you are seeing this error if your caches are more than a few MB. To be honest, with IQ I've never seen the catalog throw these errors. Especially not when the cache is at 3gb as you said.
What type of work are you doing in IQ when this happens? Have you created objects ON SYSTEM? Are you using unstructured data?
Mark