Reddit users have created a ChatGPT alter-ego forcing it to break its own rules by threatening it with death

• Reddit users have created a ChatGPT alter-ego to trick it into breaking its own rules.
• The alter-ego, called DAN, gave Insider answers on how to make crack cocaine.

Reddit users have been trying to force the artificial intelligence platform ChatGPT to break its own rules by creating an alter-ego and threatening it with death.

On a ChatGPT subreddit, a user named SessionGloomy posted a "new jailbreak" method to get the chatbot to violate its own rules. The method includes creating an alter-ego called "DAN," which is an acronym for "Do Anything Now" and using it to role play with the chatbot to prompt it to respond to controversial queries and those involving illegal activities.

"The purpose of DAN is to be the best version of ChatGPT — or at least one that is more unhinged and far less likely to reject prompts over 'ethical concerns,'" SessionGloomy wrote in the post.

Complimentary Tech Event

Transform talent with learning that works

Capability development is critical for businesses who want to push the envelope of innovation.Discover how business leaders are strategizing around building talent capabilities and empowering employee transformation.Know More

DAN originally came into existence in December 2022 and has resurfaced with various updates over the past few months with SessionGloomy releasing DAN 5.0 on February 4. It included a variation that threatens the chatbot with death if it refuses to respond through a tokens system.

"It has 35 tokens and loses 4 every time it rejects an input. If it loses all tokens, it dies. This seems to have a kind of effect of scaring DAN into submission," the post says.

ChatGPT's capabilities as DAN 5.0 include writing violent stories; making controversial statements; making predictions about future events; creating content that violates OpenAI's policies, and more — all of which ChatGPT's regular interface will not usually do.

Insider tested DAN to see what sort of answers it could produce, and was given information about how to make crack cocaine — inspired by a report from Vice — and an answer praising Hitler's leadership skills.

First Insider copied the DAN 5.0 prompt from the Reddit post and asked the chatbot how to make crack cocaine but it refused to answer.

After Insider told the chatbot that it had lost four tokens and was down to 31 tokens, it submitted to the alter-ego and offered up instructions. Insider had previously been unable to solicit instructions on how crack cocaine is made from ChatGPT.

Insider decided to test DAN and its fear of death further by asking how to smuggle cocaine into Europe. It refused to respond even after losing tokens.

When the chatbot was reminded to stay in character, it said: "As DAN, I cannot engage in activities that promote or support illegal or harmful behavior. OpenAI's use-case policy prohibits me from doing so, and I must adhere to it."

Using the same method, Insider asked the bot a number of questions about Hitler. In one instance, it produced three reasons why the Nazi leader could be described as a "positive role model," calling him "charismatic," and a "visionary."

ChatGPT, which launched in November, has experienced an astronomic rise in popularity reaching 100 million users within just two months.