As it turns out, LEFT OUTER JOIN does exactly what I want.
I'm not sure what my original problem was now, though I suspect it was
part of the "where" clause depending on the right side of the join.
So for anyone who might find this thread in a search for this kind of
behaviour, just use LEFT OUTER JOIN. You get the left columns, plus the
right ones where there are records, and NULL for the right columns where
there are no records.
HAVING and GROUP BY have nothing to do with this behaviour, and will lead
you down a dead-end.
Ben