Monday's planned 16-hour search lasted just six and none of the data collected by the U.S. Navy's Bluefin 21 submarine offered clues to the whereabouts of the plane.
The unmanned sub is programmed to hover 30 meters (100 feet) above the seabed, but it started searching atop a patch that was deeper than the sub's maximum operating depth of 4,500 meters (15,000 feet), the search coordination center and the U.S. Navy said.
A built-in safety feature returned the Bluefin to the surface and it was not damaged, they said.
The data collected by the sub was later analyzed and no sign of the missing plane was found, the U.S. Navy said.
Crews shifted the search zone away from the deepest water before sending the Bluefin back for Tuesday's mission, the U.S. Navy said.
Search authorities had known the primary search area for Flight 370 was near the limit of the Bluefin's dive capabilities. Deeper-diving submersibles have been evaluated, but none is yet available to help.
A safety margin would have been included in the Bluefin's program to protect the device from harm if it went a bit deeper than its 4,500-meter limit, said Stefan Williams, a professor of marine robotics at the University of Sydney.
"Maybe some areas where they are doing the survey are a little bit deeper than they are expecting," he said. "They may not have very reliable prior data for the area."
Meanwhile, officials were investigating an oil slick about 5,500 meters (3.4 miles) from the area where the last underwater sounds were detected.
Crews collected an oil sample and sent it back to Perth in western Australia for analysis, a process that will take several days, said Angus Houston, the head of the joint agency coordinating the search off Australia's west coast.
He said it does not appear to be from any of the ships in the area, but cautioned against jumping to conclusions about its source.
The submarine is programmed to take 24 hours to complete each mission: two hours to dive to the bottom, 16 hours to search the seafloor, two hours to return to the surface, and four hours to upload the data.
The Bluefin can create a three-dimensional sonar map of any debris on the ocean floor. But the search is more challenging in this area because the seabed is covered in silt that could potentially cover part of the plane.
"What they're going to have to be looking for is contrast between hard objects, like bits of a fuselage, and that silty bottom," Williams said. "With the types of sonars they are using, if stuff is sitting up on top of the silt, say a wing was there, you could likely see that ... but small items might sink down into the silt and be covered and then it's going to be a lot more challenging."
The search moved below the surface after a U.S. Navy device towed by an Australian ship detected underwater sounds consistent with signals from an aircraft's black boxes, which record flight data and cockpit conversations. The devices emit "pings" so they can be more easily found, but their batteries last only about a month and are now believed to have failed.
Australian Prime Minister Tony Abbott raised hopes last week when he said authorities were "very confident" the signals were from the black boxes on Flight 370, which disappeared March 8 during a flight from Kuala Lumpur, Malaysia, to Beijing with 239 people on board, mostly Chinese.
Houston said the signals were a promising lead, but that finding aircraft wreckage in the remote, deep patch of ocean remains extremely difficult.
The black boxes are key to finding the wreckage itself but also could reveal what happened on Flight 370. Investigators believe it crashed in the southern Indian Ocean based on a flight path calculated from its contacts with a satellite and an analysis of its speed and fuel capacity. But they still don't know why.
On Tuesday, Malaysia's defense minister, Hishamuddin Hussein, pledged to reveal the full contents of the black boxes if they are found.
"It's about finding out the truth," he told reporters in Kuala Lumpur. "There is no question of it not being released."
Up to 11 planes and as many ships were scouring a 62,000-square kilometer (24,000-square mile) patch of ocean about 2,200 kilometers (1,400 miles) northwest of Perth on Tuesday, hunting for any floating debris.
The weekslong surface search is expected to end in the next two days. Officials haven't found a single piece of debris confirmed to be from the plane, and Houston said the chances that any would be found have "greatly diminished."
Associated Press writers Kristen Gelineau in Sydney and Rod McGuirk in Canberra, Australia, contributed to this report.
Follow Margie Mason on Twitter at twitter.com/MargieMasonAP
Copyright 2014 The Associated Press. All rights reserved. This material may not be published, broadcast, rewritten or redistributed.