M365Con 2027 – conférence en ligne gratuite · 11–22 Jan 2027 · 120 sessions m365con.net

Multimodal AI

Concept

Multimodal AI refers to models and applications that process more than one input type, such as text, images, voice and video.

✦ CONCEPTMultimodal AIM365.FM
Image de titre : IA multimodale (concept)

Également abordés

L’épisode consacré à Multimodal AI aborde aussi Azure AI Content Safety, Copilot Control System, Graphics processing unit (GPU), AI evaluation, Reranking et OpenAI.

Description

Multimodal AI refers to models and applications that process more than one type of input, such as text, images, voice and video. In the research literature the underlying field is called multimodal learning, a branch of deep learning that combines several kinds of data, known as modalities. These can include text, images, audio, video and, in robotics, proprioception. The central idea is that combining modalities gives a fuller picture than any single data type can provide.

Multimodal approaches are used for tasks such as visual question answering, cross-modal retrieval, text-to-image generation and image captioning. Beyond these, applications named in the sources range from search and recommendation to healthcare diagnostics, robotics, emotion recognition and code search. In enterprise settings, the same principle shows up when a system accepts a photo, a spoken request or a document alongside written text and uses all of it to complete a task.

Technically, a multimodal model has to represent the information from each modality jointly. Several approaches exist. In early fusion, the embeddings of the different modalities are merged before a prediction is made. In intermediate fusion, each modality is first processed separately and the resulting representations are then combined. Cross-attention is another mechanism for relating one modality to another. A common recipe for multimodal large language models converts the output of an encoder into tokens that are interleaved with text tokens, often using a frozen image encoder. Examples named in the sources include CLIP for joint image-text representations, LLaVA, Whisper for audio and DALL-E for generation. The field was proposed in 2011, and large multimodal models such as Google Gemini and GPT-4o have become popular since 2023.

The M365.FM episode "Beyond Copilot: Building AI That Works in the Real World with Azure AI Foundry and Agentic AI for Frontline Workers with Fergus Kidd [MVP]" places multimodal AI in the context of frontline work. The episode describes computer vision as having immediate practical value in the field. OCR can capture long serial numbers without a technician having to read them aloud, and barcode and QR scanning can quickly identify equipment and retrieve the right records. More advanced visual quality checks, such as verifying a window installation from a photo, are described as possible but requiring significant additional effort. The episode frames this as the point where enterprise AI becomes operational: it does not merely generate text, it connects people with the right data and helps them complete real tasks, moving between knowledge retrieval, notes and reporting within the same session.

The episode "How Microsoft Built an Enterprise AI Architecture Advantage" lists multimodal AI among the areas of AI innovation it discusses. It describes advances in AI generation that combine text, audio and images for richer user experiences, alongside other areas such as AI copilots integrated into business applications and machine learning for energy management.

Several open challenges are associated with multimodal systems according to the source material. Aligning the modalities with each other is one. Deciding how and when to fuse them is another, as is handling situations in which one modality is missing. Choosing data sources that complement each other also matters. Privacy is a further concern, because combining several data sources can make it possible to re-identify people. These points are relevant when evaluating whether a multimodal capability fits a given scenario, particularly where images, audio or video of people or equipment are captured as part of everyday workflows.

Aperçu compilé avec l'IA à partir des épisodes M365.FM et des informations de cette page. Tu peux l'améliorer avec le bouton de modification ci-dessous.

TypeConcept

Glossaire

Community wiki

Short info above, community knowledge below: add practical tips, examples or corrections. New contributions are reviewed first; trusted members publish directly.

Experts et entreprises pour ce sujet

Personne n'a encore indiqué ce sujet. Ajoutez-le à votre profil.

Maintenu par Mirko Peters (Microsoft MVP) · Dernière mise à jour: