M365Con 2027 – free online conference · 11–22 Jan 2027 · 120 sessions m365con.net

Multimodal AI

Concept

Multimodal AI refers to models and applications that process more than one input type, such as text, images, voice and video.

✦ CONCEPTMultimodal AIM365.FM
Title graphic: Multimodal AI (Concept)

Also discussed

The episode about Multimodal AI also covers Azure AI Content Safety, Copilot Control System, Graphics processing unit (GPU), AI evaluation, Reranking and OpenAI.

Description

Multimodal AI refers to models and applications that process more than one type of input, such as text, images, voice and video. In the research literature the underlying field is called multimodal learning, a branch of deep learning that combines several kinds of data, known as modalities. These can include text, images, audio, video and, in robotics, proprioception. The central idea is that combining modalities gives a fuller picture than any single data type can provide.

Multimodal approaches are used for tasks such as visual question answering, cross-modal retrieval, text-to-image generation and image captioning. Beyond these, applications named in the sources range from search and recommendation to healthcare diagnostics, robotics, emotion recognition and code search. In enterprise settings, the same principle shows up when a system accepts a photo, a spoken request or a document alongside written text and uses all of it to complete a task.

Technically, a multimodal model has to represent the information from each modality jointly. Several approaches exist. In early fusion, the embeddings of the different modalities are merged before a prediction is made. In intermediate fusion, each modality is first processed separately and the resulting representations are then combined. Cross-attention is another mechanism for relating one modality to another. A common recipe for multimodal large language models converts the output of an encoder into tokens that are interleaved with text tokens, often using a frozen image encoder. Examples named in the sources include CLIP for joint image-text representations, LLaVA, Whisper for audio and DALL-E for generation. The field was proposed in 2011, and large multimodal models such as Google Gemini and GPT-4o have become popular since 2023.

The M365.FM episode "Beyond Copilot: Building AI That Works in the Real World with Azure AI Foundry and Agentic AI for Frontline Workers with Fergus Kidd [MVP]" places multimodal AI in the context of frontline work. The episode describes computer vision as having immediate practical value in the field. OCR can capture long serial numbers without a technician having to read them aloud, and barcode and QR scanning can quickly identify equipment and retrieve the right records. More advanced visual quality checks, such as verifying a window installation from a photo, are described as possible but requiring significant additional effort. The episode frames this as the point where enterprise AI becomes operational: it does not merely generate text, it connects people with the right data and helps them complete real tasks, moving between knowledge retrieval, notes and reporting within the same session.

The episode "How Microsoft Built an Enterprise AI Architecture Advantage" lists multimodal AI among the areas of AI innovation it discusses. It describes advances in AI generation that combine text, audio and images for richer user experiences, alongside other areas such as AI copilots integrated into business applications and machine learning for energy management.

Several open challenges are associated with multimodal systems according to the source material. Aligning the modalities with each other is one. Deciding how and when to fuse them is another, as is handling situations in which one modality is missing. Choosing data sources that complement each other also matters. Privacy is a further concern, because combining several data sources can make it possible to re-identify people. These points are relevant when evaluating whether a multimodal capability fits a given scenario, particularly where images, audio or video of people or equipment are captured as part of everyday workflows.

Overview compiled with AI from M365.FM episodes and the facts on this page. You can improve it with the edit button below.

TypeConcept

Episodes about Multimodal AI

Episodes in which Multimodal AI is discussed:

Frequently asked questions

Questions marked with AI were written with AI from the facts on this page.

What is multimodal AI? (AI)

Multimodal AI refers to models and applications that process more than one input type, such as text, images, voice and video. The underlying research field is called multimodal learning, which combines several kinds of data, called modalities, to give a fuller picture than a single type.

How do multimodal models combine different data types? (AI)

Common approaches include early fusion, where embeddings are merged before prediction, intermediate fusion of separately processed representations, and cross-attention. Many multimodal LLMs convert an encoder's output into tokens interleaved with text tokens.

How does multimodal AI apply to frontline work? (AI)

According to an M365.FM episode, computer vision supports tasks such as OCR for serial numbers, barcode and QR scanning to retrieve records, and photo-based checks like verifying a window installation, although advanced visual checks require significant effort.

Glossary

Community wiki

Short info above, community knowledge below: add practical tips, examples or corrections. New contributions are reviewed first; trusted members publish directly.

Experts and companies for this topic

Nobody has listed this topic yet. Add it to your profile.

Sources

Maintained by Mirko Peters (Microsoft MVP) · Last updated: