The founding family of Quicksilver Resources has cancelled its plans to take the oil and gas company private, as talks with its second-biggest shareholder ended.