It comes after tweets emerged in which he used homophobic slurs.
The decision comes just three days after it was announced he was to present the ceremony.
The actor and comedian's response to the outcry attracted a further backlash, as he wrote on Instagram that should critics should "stop being negative".
In an accompanying video, a shirtless Hart said he wasn't going to "let the craziness frustrate me" and added that he "loves everybody".
The gay media watchdog group GLAAD reached out to the Academy Awards broadcaster ABC, the Academy of Motion Pictures Arts and Sciences, and Hart's management to "discuss Kevin's anti-LGBTQ rhetoric and record."
Hart said in a 2010 stand-up special that "if I can prevent my son from being gay, I will."
The Hollywood star was confirmed as the host of the Academy Awards, popularly known as the Oscars, on Tuesday.
He announced he was withdrawing himself from the role in the early hours of Friday morning.
Hart wrote on Twitter: "I have made the choice to step down from hosting this year's Oscar's....this is because I do not want to be a distraction on a night that should be celebrated by so many amazing talented artists.
"I sincerely apologize to the LGBTQ community for my insensitive words from my past."
He added in a follow-up post: "I'm sorry that I hurt people.. I am evolving and want to continue to do so.
"My goal is to bring people together not tear us apart.
"Much love & appreciation to the Academy.
"I hope we can meet again."
The awards ceremony will take place in February, with Hart describing the gig as the "opportunity of a lifetime" when he was made host.
The 39-year-old wrote: "I am so happy to say that the day has finally come for me to host the Oscars.
"I am blown away simply because this has been a goal on my list for a long time.
"To be able to join the legendary list of hosts that have graced that stage is unbelievable. I know my mom is smiling from ear to ear right now."
He thanked his family, friends and fans for their support, and signed off his post by promising to "rise to the occasion" and make the ceremony "a special one".
A representative for the academy did not immediately respond to messages after Hart stepped down.