Cloud data warehousing became a hot market after startups like Snowflake addressed a need that Amazon, Microsoft and Google failed to meet. But it's a changing market where the cloud giants could turn into their rivals

• The rise of the cloud sparked the need for fast and efficient ways to store and manage huge amounts of data in cloud platforms like those run by Amazon, Microsoft and Google.
• Data warehousing allowed corporations to store massive and diverse types of data in private data centers for decades, but it has proven challenging in the cloud. Amazon, Microsoft and Google had a hard time providing effective data warehousing tools to their clients.
• Startups like Snowflake, Incorta and Panoply stepped in to address this need creating a growing and highly-competitive market that's expected to be worth $100 billion by 2025.
• But Snowflake and other startups now also find themselves competing with the cloud giants which continue to develop their own data warehousing and management technologies, which threaten to make startups less attractive and even irrelevant.
• Click here for more BI Prime stories.

One of the hottest sectors of cloud computing is at a turning point.

A bevvy of young startups carved a lucrative niche for themselves in recent years by helping corporate customers manage the oceans of data they moved onto cloud platforms.

While cloud giants like Amazon, Google and Microsoft thrived, so too did specialized data warehousing startups like Snowflake, Panoply, Incorta and others.

Now, the arrangement looks like it's run its course: The startups are bracing for competition with the cloud giants they once considered allies, and scrambling to find new capabilities that will justify their rich, sometimes multi-billion dollar, valuations.

"This world is really exploding," Gartner senior director analyst Adam Ronthal told Business Insider. "In order to be successful, they need one thing that is totally different, and totally compelling."

For investors in the enterprise tech market and for corporate customers, the shifting landscape is likely to bring new innovations, the potential for price wars and maybe even acquisitions.

Data warehousing has been challenging for businesses.

Data warehousing has been a major challenge for companies that ran their networks in private data centers, said Osama Elkady, who worked for Oracle for 20 years before launching his own cloud data warehousing startup called Incorta.

"The process of data warehousing was a very painful for customers," he told Business Insider. "Building the data warehouse takes a long time to implement."

Setting up data warehouses is even more challenging in cloud computing the trend which has allowed businesses to set up their networks in web-based platforms run by Amazon, Microsoft and Google. This trend made it possible for businesses to scale down or even abandon private data centers leading to huge cost savings.

But the cloud giants, led by Amazon, the most dominant platform, had a hard time quickly developing data warehousing tools for their clients, who found many of the products they offered in the earlier years of the cloud inadequate, Ronthal said.

Startups stepped in to fill a need.

That's when the startups stepped in.

Over the last few years, when more businesses were migrating to the cloud, a host of new companies "filled a void that the big cloud giants initially could not satisfy," Craig Kelly, vice president of analytics at Syntax, a cloud management services company, told Business Insider.

The most successful one, by far, is Snowflake. Founded in 2012 by Benoit Dageville, Marcin Zukowski, Thierry Cruanes, it came out of stealth mode in 2014, and quickly drew rave reviews with a new data warehousing product designed and built specifically for cloud computing.

"They hit the market at exactly the right time when people were really starting to engage with the cloud," Ronthal said. "When Snowflake first launched they launched in a world where there was a fair amount of dissatisfaction" with the data warehousing technologies, particularly Redshift offered by Amazon, the market's most dominant player.

"Snowflake came along and said, 'If you're struggling with these things, we can address that for you.' And they could."

Snowflake, which is now valued at $3.9 billion, is widely expected to go public in a year or two. Other startups quickly emerged to take on the cloud data warehousing market which Constellation Research projects will be worth $100 billion by 2025.

Rubrik, which is based in Palo Alto, Calif. and is now valued at $3.3 billion , provides AI-powered tools for backing up and recovering a company's data stored in the cloud.

Incorta, which is based in San Mateo, Calif., offers data warehousing for companies that are either in the cloud or are still maintaining private data centers.

Panoply, which is based in San Francisco, came up with an automated data warehousing tool for Amazon cloud customers, though it subsequently expanded to Microsoft's platform. It was recently named to Gartner's list of "cool vendors" for data warehousing, for coming AI-powered tools for "easy ingesting of data from multiple sources."

These startups emerged at a time when what had been the hot trend called big data was starting to cool. This trend was based largely on Hadoop, a technology that made it possible to store and sort through huge amounts of data from diverse sources with exceptional speed.

But Hadoop vendors were eventually overshadowed by the rise of AI and cloud data warehousing platforms, led by Snowflake. One Hadoop company, MapR was forced to sell its assets to Hewlett Packard Enterprise. Another, Cloudera, is trying to bounce back, and has just unveiled its own "cloud native" data warehousing platform.

Ronthal called Hadoop "a story of overpromising and underdelivering" where vendors "basically painted a picture of a platform that was going to be all things to all people. It was going to do all these things and, as it turned out, it only did some of those things really well."

Cloud data warehousing and management startups face a different set of challenges. The Gartner report on Panoply noted that "the initial limitation that tied Panoply to AWS Redshift also limited its initial target customer base, and the company will need to work hard to expand its footprint in an ultra-competitive market."

The warning pretty much applies to other startups in the cloud data warehousing market, including a dominant player like Snowflake, Ronthal said: "The problem they face is, if I strategically align myself with Amazon ... I'm going to go with Redshift and see if it works and meets my needs. For a good percentage of organizations, it will."

Amazon and the cloud giants aren't sitting still.

While customers may have been dissatisfied with Amazon's data warehousing tool in the past, Ronthal said, "Amazon doesn't sit still. They continue to invest in Redshift." This is true for Microsoft and Google with their own data warehousing tools, he added. It's only a matter of time a cloud provider says, "Oh, we do that too."

"So the point of differentiation starts to erode," he said. "The Redshift of today is very different from the Redshift of five years ago in terms of performance and in terms of capabilities. ... If Snowflake were to launch today, they will probably have a harder time of it."

Kelly of the Syntax echoed this view, saying: "I do have long term concerns with the viability of most of these companies. It will be hard to compete with the pace of innovation, as well direct access to the top performing infrastructure that the cloud giants have."

Asked which cloud data warehousing startups are most vulnerable, Kelly answered: "Quite honestly, all of them."

He said the challenges become pronounced to him at Amazon's annual cloud technology event, called re:Invent, where he said he saw "how companies hold their breath when learning about the hundreds of new applications that AWS rolls out due to fear that their niche will be cannibalized."

On Snowflake, he said: "I just question how they will differentiate long term, or be able to keep up with the pace of innovation and horsepower that AWS provides."

Slootman, Snowflake's CEO, argued that it's good business for the cloud giants to be able to offer choices. "Amazon won't be attractive if you can only buy Amazon [stuff] on Amazon. Would you like it as a customer? No. You want a rich ecosystem. You want options. You want to go to Safeway and want one brand of toothpaste? No. That's not like Soviet Union-style grocery."

But Kelly said Amazon has been offering more choices for clients and Snowflake "doesn't have the breadth of capabilities that AWS can combine" with Redshift and other newer tools, such as Glue, for preparing data for analytics, and Quicksight, a business intelligence service.

Ronthal said that for most cloud data warehousing startups, like Snowflake, a priority should be to identify "points of differentiation that's sticky." One path to stickiness is what he called an "inter-cloud" strategy which would allow businesses using different cloud platforms to use Snowflake across those platforms, he said. It would be a capability that Amazon, Microsoft and Google could not easily do "because they are limited to their respective clouds."

Snowflake could not be reached for comment. But the company has hinted that this is the path it plans to take, referring to a "cross-cloud" approach.

"The true benefits of a multi-cloud strategy will not materialize until data can be shared and replicated across clouds and regions, the company said in a blog post. "Fortunately, cross-cloud capability is the answer."

But the need for a nimble and robust product roadmap was noted by Snowflake CEO Frank Slootman himself as he acknowledged that the startup's biggest partners could one day become its fiercest competitors.

"There is one giant elephant in the room," Slootman told Business Insider in a June interview. "That is that the [main] cloud players are also our competitors."

"Platform companies always try to suffocate other companies," he said. "Microsoft did it. IBM did it. They all do it. … I'm a huge customer of Amazon and at the same time they are a huge competitor. … Microsoft is trying to make up ground so they are very partner-friendly right now. But does Microsoft have the same homicidal impulses as Amazon? Yes, they do."

Got a tip about Snowflake or another tech company? Contact this reporter via email at bpimentel@businessinsider.com, message him on Twitter @benpimentel. You can also contact Business Insider securely via SecureDrop.