Under pressure to explore new ways to hedge their risks while cutting fee expenses, institutions are taking a fresh look at listed options — in some cases, at the expense of hedge funds.