The outer nested join has the VALUES as the main loop, and the complicated join as the leaf. So, the complicated overlap-finding join gets run twice.
That's weird. What do you have as statistics target? Planner is incorrect few orders of magnitude, so increasing it may help.
BTW: One of constraints is redundant l1.start <= l2.start implies l1.start <= l2.end, so latter can be removed as for me.
Oh, there's also the great big sort and unique, but I think I can get rid of that.
As far as I can see, duplicates will occur if and only if l1.start == l2.start && l1.end == l2.end.
That can be easily filtered by adding "where n=1 or l1.start != l2.start or l1.end != l2.end" to outer select.