It depends what you do. Postgres doesn't allow to allocate bigger blocks than 1GB. Maybe you create too big string or too big value of some other type. But it can be signal of some cache bloating.
Can you show source code? Can you use gdb, attach to Postgres, place breakpoint to this error message, and when you get this error, send stack trace?