Apple Releases AI-Powered Audio Intelligence Technology in Beta Testing

Apple's AI-powered Audio Intelligence technology is now in beta testing on Apple Watch Series 12 devices as part of the iOS 27.3 update for developers. This feature was first announced at Apple's Surprise and Shine event in September.

The technology includes four opt-in AI features that allow users to detect important sounds, identify songs, catch up on conversations, and take notes. One of these features is designed to assist deaf and hard-of-hearing users by identifying important sounds such as sirens or doorbells from their watch.

Another feature taps into Apple's Shazam app to recognize music playing in the background and provide information about the song name and artist. However, it's the more interactive features that may raise concerns about user privacy.

Apple is attempting to strike a balance between providing useful functionality and respecting the privacy of those around them. The company has been mindful of the backlash against AI-powered recording devices like Meta's AI glasses.

The Live Rewind feature allows users to review recent conversations by double-pressing on the watch's digital crown, displaying a text snippet of what was said in the last 15 seconds. This snippet can be viewed and then deleted or saved for later access within the standalone Siri app.

This functionality is intended to help users recall important information from fast-paced situations such as presentations or conversations at restaurants.

The new feature allows users to tap an "Ask Siri" option below a text snippet if they need clarification on something that was mentioned during a conversation.

A double-press of the digital crown triggers an audible chime and visual animation, signaling to the user that a recording has been made.

Recordings are automatically deleted after 30 seconds unless saved by the user.

The AI-powered features of Apple Watch are being put to the test in a beta phase, focusing on recording conversations for users who need help taking notes. This can be particularly useful in situations where hands are full or one needs to concentrate on what's being said.

When using Siri Recaps, recordings are automatically deleted after 30 seconds unless saved by the user. However, these features won't activate when listening to music with headphones or using noise cancellation features, ensuring users' conversations remain private during those times.

To manage recorded conversations, Apple Watch owners can review their saved Siri Recaps and choose which ones to save. These notes can be stored in the Notes app or exported to the Journal app for more personal entries. If no action is taken, recaps will delete themselves within seven days.

A reminder to review recorded conversations appears not only at the end of each day but also when unlocking the Apple Watch in the morning. This ensures users stay on top of their saved notes and can easily access them as needed.

Users have flexibility in how they use Siri Recaps, with options to enable or disable it manually or set a schedule for recording conversations. Even if a schedule is set, the feature can be turned off from the Control Center at any time. Apple has also designed the feature to omit sensitive information such as government-assigned identifiers and personal data that could promote self-harm.

The Apple Watch's Audio Intelligence features rely on a combination of hardware, software, and services to function. The S11 chip, built-in microphone, and Apple Intelligence models that run on the iPhone are all part of this ecosystem.

A secure enclave within the always-on processor provides a dedicated audio buffer where raw audio is processed without being accessible to anyone, including Apple. This ensures the audio remains private until it's transferred to the paired iPhone through a secure pairing mechanism.

Lightweight AI models on the S11 chip detect conversations and transfer the relevant audio to the iPhone's secure enclave. On-device processing converts the audio into condensed text, removing filler words, which is then encrypted before being sent to Apple's Private Cloud Compute (PCC).

The PCC processes the condensed text to create a recap of the conversation, which is then sent back to the iPhone as part of the Audio Intelligence feature.

The recaps created by Apple Watch's Audio Intelligence feature are stored securely on your iPhone and can only be accessed by you. This ensures that sensitive information shared during conversations remains private.

To create these 15-second snippets for Live Rewind, the Apple Watch transfers audio to the iPhone for processing. The watch itself does not store any of this data locally.

The company claims that the use of these features will not impact battery life on either your watch or phone. This is a key consideration as users begin to test and provide feedback on these new AI capabilities.

Once beta testing is complete, these features are expected to become available to all users in a public release.

The Apple Watch is gearing up to roll out two AI-powered features that aim to simplify conversations for users. The new capabilities will allow users to rewind and recapture conversations, potentially making it easier to follow along in noisy environments or when speaking with multiple people.

These features are currently being tested through a beta program before their public release.