Meta expands AI safety protocols with parental alerts for teen self-harm discussions
The social media giant introduces manual moderation safeguards and plans emergency service integration as it tightens controls around adolescent interactions with its artificial intelligence models.
Meta has introduced a new safety mechanism for Meta AI on Instagram that notifies parents or guardians if their teenage children discuss self-harm or suicide during private chats. The system employs a dedicated AI to flag conversations, which are then manually reviewed by human moderators to ensure accuracy and prevent unnecessary panic before alerts are sent. These notifications include resources and advice for parents. The feature is currently available in the US, UK, Australia and Canada for users with parental supervision enabled, with a global rollout planned by the end of the year.
The alert mechanism utilises a dedicated AI system to identify clear references to self-harm within chat logs. To mitigate false positives, Meta has implemented a procedural safeguard where all chats flagged by the initial detection system undergo manual review before an alert is dispatched to parents. The company acknowledged the distress these notifications may cause, stating that in instances where a teen’s intent appears ambiguous, human reviewers will err on the side of caution and send out the notification to avoid missing real risks.
This update extends existing safety protocols that already trigger emergency contacts for public posts indicating self-harm, thereby addressing gaps in private chatbot interactions. Meta AI already directs users discussing suicide or self-harm to crisis helplines and encourages them to reach out to an adult. The new parental alert system complements these measures by proactively informing guardians, who receive specific resources and advice on how to support their children.
Meta is also developing capabilities to contact emergency services directly if a user, including adults, is assessed to be at imminent risk. This capability is currently in development and will apply to both minors and adults if a conversation with Meta AI suggests an imminent risk of suicide. The company is expanding its safeguards for teen users in an effort to demonstrate to parents and regulators that its platforms are safe for adolescent use.
Additionally, Meta’s Limited Content setting now applies to Meta AI interactions. If a parent has switched on Limited Content on Instagram, which filters out content with sensitive topics like mature visuals and self-harm, their children's AI interactions will also be more limited. This follows a similar move by OpenAI in May, which rolled out a feature called Trusted Contact for ChatGPT, allowing users to nominate a friend to be contacted if they are at risk of harming themselves.
