How big does the data stored in that field get? More than 2KB? Real question- is it getting stored plain, compressed inline, or toasted? Have you set the storage strategy/type, or is it the "extended" default behavior that compresses and then stores in the toast table if still more than 2000 bytes?
Always joining on a field that is toasted and compressed is going to be considerably slower than a discrete field. Do you have an index (or several) on the json field values? Also, are we actually talking about jsonb?