Microsoft Copilot Gets Voice, Vision Features

The updated Copilot app will read users a summary of news, a feature called Copilot Daily, and, eventually, personalized reminders. Copilot Vision, which is in preview, can answer questions about text and images users are browsing online and offer guidance — such as suggesting how to furnish an apartment someone is considering purchasing.

The upgraded AI assistant, which starts rolling out Tuesday, works with iOS, Android, Windows and the web, and includes a new version for WhatsApp.

The vision tool requires customers to opt in. In preview, it’s being tested with a limited list of popular web sites so Microsoft can collect data and ensure safety. The data also won’t be stored or used for training, the company said, and the tool won’t work with paywalled and “sensitive” content in the preview.

While Microsoft staked out an early lead in consumer AI owing to its investment in OpenAI, rivals Apple Inc. and Alphabet Inc.’s Google are now rolling out AI services aimed at the millions of people who use Android and iOS mobile devices. Microsoft is shut out of that market, so its AI strategy relies heavily on Windows computers, notebooks and tablets.

Earlier this year, the company teamed up with hardware makers to market a line of PCs specially designed to run artificial intelligence applications. So far, reaction to the machines — sold under the Copilot+ PC brand — has been mixed.

In an effort to generate some excitement, Microsoft on Tuesday also announced several new AI features for the devices.

A photo-editing tool lets users enhance older or lower-quality images as well as fill in and erase parts of photos to make them easier to edit. A plain-English search lets users find documents and photos by describing them rather than having to know the file name, while a Click to Do feature brings up a menu of suggested steps or tasks based on what the customer is working on. The new Copilot app will also be available with a click on the Copilot+ PC

Microsoft’s AI consumer chief, Mustafa Suleyman, has pledged to move beyond the utilitarian products the company has long been known for. In an interview, he talked about creating new interfaces and building “experiences that deeply empathize” with users. He acknowledged that getting there will require changing the mindset inside Microsoft, not to mention building new business models.

“Now that we actually have these very conversational interfaces that give you a feeling — that focus on the emotional intelligence of the experience not just the utility — it requires us to exercise a totally new muscle in the organization,” he said.

Suleyman in a blog post explained his vision for Microsoft’s AI future. “With your permission, Copilot will ultimately be able to act on your behalf, smoothing life’s complexities and giving you more time to focus on what matters to you. It will be an advocate for you in many of life’s most important moments,” he wrote, listing things like doctor’s appointments, planning a kid’s birthday party and a “tricky life decision.”