From a college dropout to a $54 billion fortune - the incredible rags-to-riches story of Oracle founder Larry Ellison

As a child, Larry Ellison's adoptive father repeatedly told him he was good for nothing, according to Fortune.

Today, Forbes estimates Ellison's net worth at $54 billion, making him the third richest person in the US.

Before he founded Oracle, the database software firm that made his fortune, Ellison grew up in a working class Chicago family of Jewish immigrants.

"I was raised on the South Side of Chicago," he said in an oral history for the Smithsonian Institution. "I remember Look Magazine called it the oldest and worst black ghetto in the United States."

When Ellison was born in 1944, his mother was unmarried, according to a profile in Fortune. She gave him to relatives to raise, and Ellison never met his biological father.

He dropped out of college twice - first from the University of Illinois at Urbana-Champaign, then the University of Chicago - before moving to California and working odd jobs.

In his oral history, he recalled, "I never took a computer science class in my life. I got a job working as a programmer; I was largely self-taught. I just picked up a book and started programming."

When he landed a programming job at Ampex Corporation, Business Insider's Madeline Stone notes that one of Ellison's responsibilities at the company was building a database for the CIA. In 1977, he and two coworkers left Ampex to start a database management company of their own.

Knowing that no one would want to take a risk on a brand new product, Ellison and his cofounders chose not to label their first release "Version 1.0."

"The very first version was Oracle Version 2," he admitted at a customer conference last year.

Their ploy worked. Oracle's first customer was a big one: the CIA. It later became the most popular database ever sold. That success paid off for Ellison - according to the Wall Street Journal, he was the highest paid executive in the US before he stepped down as CEO in 2014.

But becoming a billionaire was never his goal, he told the Smithsonian Institution. "When I started Oracle, what I wanted to do was to create an environment where I would enjoy working. That was my primary goal. Sure, I wanted to make a living. I certainly never expected to become rich, certainly not this rich."

Now 70, Ellison has a lifestyle that he could only have dreamed of during his working class Chicago childhood.

"This is all kind of surreal," he told Mike Wilson, the author of "The Difference Between God and Larry Ellison." "I don't even believe it now. Not only did I not believe it when I was 14, but when I look around, I say, this must be something out of a dream."

Ellison collects cars and private jets, and has his own America's Cup sailing team. His incredible real estate portfolio includes a private golf club in Rancho Mirage, California; a $70 million house in Silicon Valley; the former summer home of the Astor family in Newport, Rhode Island; a historic garden villa in Kyoto, Japan; and the entire Hawaiian island of Lanai. And because he loves basketball, he's installed courts on at least two of his yachts.

He has also given hundreds of millions of dollars to charity, particularly medical research and education, and says that he plans to give billions more.

Clearly, he's proven his adoptive father wrong.