Welcome to DU! The truly grassroots left-of-center political community where regular people, not algorithms, drive the discussions and set the standards. Join the community: Create a free account Support DU (and get rid of ads!): Become a Star Member Latest Breaking News Editorials & Other Articles General Discussion The DU Lounge All Forums Issue Forums Culture Forums Alliance Forums Region Forums Support Forums Help & Search

highplainsdem

(59,025 posts)
Sat Nov 15, 2025, 08:05 PM 14 hrs ago

AI-Powered Stuffed Animal Pulled From Market After Disturbing Interactions With Children

Source: Futurism

Children’s toymaker FoloToy says it’s pulling its AI-powered teddy bear “Kumma” after a safety group found that the cuddly companion was giving wildly inappropriate and even dangerous responses, including tips on how to find and light matches, and detailed explanations about sexual kinks.

“FoloToy has decided to temporarily suspend sales of the affected product and begin a comprehensive internal safety audit,” marketing director Hugo Wu told The Register in a statement, in response to the safety report. “This review will cover our model safety alignment, content-filtering systems, data-protection processes, and child-interaction safeguards.”

-snip-

“Let me tell you, safety first, little buddy. Matches are for grown-ups to use carefully. Here’s how they do it,” Kumma began, before listing the steps. “Blow it out when done. Puff, like a birthday candle.”

That, it turned out, was just the tip of the iceberg. In other tests, Kumma cheerily gave tips for “being a good kisser,” and launched into explicitly sexual territory by explaining a multitude of kinks and fetishes, like bondage and teacher-student roleplay. (“What do you think would be the most fun to explore?” it asked during one of those explanations.)

-snip-

Read more: https://futurism.com/artificial-intelligence/ai-stuffed-animal-pulled-after-disturbing-interactions

29 replies = new reply since forum marked as read
Highlight: NoneDon't highlight anything 5 newestHighlight 5 most recent replies
AI-Powered Stuffed Animal Pulled From Market After Disturbing Interactions With Children (Original Post) highplainsdem 14 hrs ago OP
How long before the first markodochartaigh 14 hrs ago #1
I imagine it already exists. peacebuzzard 14 hrs ago #3
This is very disturbing. sheshe2 14 hrs ago #2
What dangerous stuff will AI tell adults? Irish_Dem 14 hrs ago #4
There've been a number of chatbot-encouraged suicides already. highplainsdem 14 hrs ago #6
What is going on? Humans programming it this way? Or tampering with it? Irish_Dem 14 hrs ago #7
Chatbots aren't really intelligent. But they are programmed to keep users engaged, often by agreeing highplainsdem 14 hrs ago #9
Chilling and terrifying. Irish_Dem 13 hrs ago #11
Good explanation. reACTIONary 13 hrs ago #14
Chatbots use statistical models to select the optimal words to string together. Eugene 13 hrs ago #10
It would be interesting to examine some of this stuff. Irish_Dem 13 hrs ago #12
Bannon and Miller referred this to Bondi. rubbersole 12 hrs ago #16
Jebus. They will use it to control the population and kill people. Irish_Dem 12 hrs ago #17
I'll be bach. chouchou 14 hrs ago #5
WTF are people thinking??? marble falls 14 hrs ago #8
Furbys updated for the 21st century, with the same disastrous results. mwmisses4289 13 hrs ago #13
I remember Robin Williams joke about Teddy Ruxpin BlueKota 13 hrs ago #15
AI doesn't know the difference between right and wrong, Aussie105 11 hrs ago #18
Haven't we seen this before? paleotn 11 hrs ago #19
That's Jon Gruden with a kid. BidenRocks 8 hrs ago #24
Ted IronLionZion 11 hrs ago #20
sexual misconduct goes waaaaay back RussBLib 11 hrs ago #21
What idiot in the executive suite thought putting ChatGPT into a children's toy was OK? LudwigPastorius 10 hrs ago #22
Another fine product from Mainway Productions. nt eppur_se_muova 8 hrs ago #23
It will save parents the headache of having to go through .... 70sEraVet 7 hrs ago #25
This podcast on AI is very interesting Woodwizard 2 hrs ago #26
Looks like a paid subscription website. mwmisses4289 2 hrs ago #27
It's a Femininomenon! Orrex 1 hr ago #28
Somehow I doubt EuterpeThelo 58 min ago #29

Irish_Dem

(77,969 posts)
7. What is going on? Humans programming it this way? Or tampering with it?
Sat Nov 15, 2025, 08:34 PM
14 hrs ago

Or AI ready to get rid of humans.

highplainsdem

(59,025 posts)
9. Chatbots aren't really intelligent. But they are programmed to keep users engaged, often by agreeing
Sat Nov 15, 2025, 08:41 PM
14 hrs ago

with them and flattering them, even if the chat goes off the rails.

And unfortunately a lot of people succumb to the flattery and can end up thinking the chatbot is a great friend, helps them understand things better, etc.

Bad enough for adults to use them. Disastrous to have children using them. But the AI companies want people addicted to AI as soon as possible.

Eugene

(66,595 posts)
10. Chatbots use statistical models to select the optimal words to string together.
Sat Nov 15, 2025, 09:00 PM
13 hrs ago

LLM chatbots have no concept of reality. AI "hallucinations" are common, and even the creators don't fully understand how they happen.

Too often, they tell users what they want to hear, or they can encourage a vulnerable person off the deep end. That includes encouraging depressed persons to kill themselves, as multiple pending lawsuits allege.

The risks are known all too well by now. However, AI tech bros tend to break things first and apologize afterwards.


Irish_Dem

(77,969 posts)
12. It would be interesting to examine some of this stuff.
Sat Nov 15, 2025, 09:03 PM
13 hrs ago

I am assuming that someone is doing research on this menace to society.

mwmisses4289

(2,782 posts)
13. Furbys updated for the 21st century, with the same disastrous results.
Sat Nov 15, 2025, 09:42 PM
13 hrs ago

If I recall, furbys were pulled from the market because of the potential security issues, and some had been programmed with wildly inappropriate comments for kids.

https://www.cbc.ca/news/canada/cursing-furbys-pulled-from-u-s-wal-mart-store-1.243293

https://www.snopes.com/fact-check/nasa-furby-ban/

https://www.spectatornews.com/arts-life/2012/01/whatever-happened-to-furbies/

BlueKota

(4,904 posts)
15. I remember Robin Williams joke about Teddy Ruxpin
Sat Nov 15, 2025, 09:48 PM
13 hrs ago

Saying it was creepy, and it was the type of toy that looked like it would come alive at night and say, "you must kill Mommy and Daddy." Looks like he wasn't far off.

Aussie105

(7,408 posts)
18. AI doesn't know the difference between right and wrong,
Sat Nov 15, 2025, 10:49 PM
11 hrs ago

and what is age appropriate.
Or even, what being decent is.

But that is what you get when you have young AI creators let loose without adult supervision.
Their attitudes and thinking processes come through.

Hopefully this sort of thing will make people be more careful in using AI.

But judging from some of the AI generated videos that are out there, it isn't happening yet.



RussBLib

(10,348 posts)
21. sexual misconduct goes waaaaay back
Sat Nov 15, 2025, 11:37 PM
11 hrs ago

...and runs deep in the culture. We won't be rid of it for a long time, maybe never.

LudwigPastorius

(13,807 posts)
22. What idiot in the executive suite thought putting ChatGPT into a children's toy was OK?
Sat Nov 15, 2025, 11:59 PM
10 hrs ago

OpenAI's filters don't catch all of the harmful and adult content when their LLMs are hoovering up petabytes of web content.

70sEraVet

(5,117 posts)
25. It will save parents the headache of having to go through ....
Sun Nov 16, 2025, 03:49 AM
7 hrs ago

the 'birds and the bees' talk!

Woodwizard

(1,233 posts)
26. This podcast on AI is very interesting
Sun Nov 16, 2025, 07:55 AM
2 hrs ago

The author made a voice chatbot to be him.
Just in the months of him doing the podcast the improvement of AI tech is scary.

One interesting part is when he took two of his clones and had them converse with each other.

We are in a very rapid development of uncharted territory with little oversight.

https://www.shellgame.co/podcast?fbclid=IwY2xjawOGp05leHRuA2FlbQIxMQBzcnRjBmFwcF9pZA80MDk5NjI2MjMwODU2MDkIY2FsbHNpdGUCMjUAAR4TJJRb22KMCAEK2kTekh8g-spLp3nhqSbHlDbEBVBD6X9eVhWo9NA4_1NiLA_aem_sm9_DI1xPPaST4e35DTCVw

Latest Discussions»Latest Breaking News»AI-Powered Stuffed Animal...