What Is Multimodal Artificial Intelligence And Why Is It Important?

Relevance: GS 3- Science and Technology- developments and their applications and effects in everyday life, Awareness in the fields of IT

(Source: The Hindu, 10/10/2023)

Click here for Daily Current Affairs

Why in the news?

• Recently, AI companies including OpenAI, Google, and Meta have begun to develop and release multimodal AI services.
• These services can process different types of data such as audio, video, image, etc.

What is multimodal artificial intelligence?

• Multimodal Artificial Intelligence refers to advanced AI models in which multiple modes of information or sensory data are integrated in order to facilitate human-like reasoning and decision-making.
• While the traditional AI models are focused on processing information from a single modality, i.e. text, image, or speech, the multimodal model incorporates data from multiple modalities.
• They are developed using sophisticated techniques such as feature extraction, machine learning, and neural networks that can integrate and analyze data from multiple sources.
• Advantages:
  • Enhanced accuracy
  • More effective AI systems.
• Examples:
  • Natural language processing (NLP): It combines text and speech recognition for more accurate and natural language interactions between humans and machines.
  • OpenAI’s text-to-image model, DALL.E is a multimodal AI model that was released in 2021.
  • DALL.E is built on another multimodal text-to-image model called CLIP developed by OpenAI.

The AI race

• The OpenAI announced that it had enabled GPT-3.5 and GPT-4 to study images and analyse them in words.
• It is alo woking on Gobi, which is expected to be a new multimodal AI system.
• According to reports, the release was due to a report that Google’s new multimodal model, Gemini, was being tested

How does multimodality work?

• ChatGPT’s vision capabilities are based on DALL.E, which is based on the same concept as other AI image generators such as Midjourney and Stable Diffusion which link text and images in the training stage.

Training

• The system identifies patterns in visual data which can be connected with the data of the image descriptions.
• As a result, the system is abale to generate images based on the text prompts entered by the user.
• Multimodal audio systems are also trained in a similar manner.
• GPT’s voice processing capabilities are based on its own open-source speech-to-text translation model, Whisper, which was released in 2022.
• Whisper can recognise speech in audio and translate it into simple language text.

What are some applications of multimodal AI?

• It can be used in fields such as healthcare, finance, entertainment, etc.
• Healthcare: Multimodal models can be used to analyze medical images, patient data, and clinical notes to provide more accurate diagnoses and treatment plans.
  • AI systems that can analyze complex datasets can be used in processing CT scans or identifying rare genetic variations.
• Finance: They can be used to analyze financial data from multiple sources, to make more informed investment decisions.
• Entertainment: It can be used to develop immersive and interactive virtual reality games and movies.
• Translation: Meta’s SeamlessM4T model, can perform text-to-speech, speech-to-text, speech-to-speech and text-to-text translations for around 100 languages.

Potential future applications

• According to research future multimodal models could use alternate sensory data like “touch, speech, smell, and brain fMRI signals.”
• In the future, AI might be able to generate visuals and sounds of an environment as well as other physical elements.
  • For example, a beach simulation would have waves, wind, and temperature.

Examples

• In 2020, Meta was working on a multimodal system to automatically detect hateful memes on Facebook while Google published research about a multimodal system that could predict the next lines of dialogue in a video.
• In 2023, Meta announced ImageBind, an open-source AI multimodal system with text, visual data, audio, temperature and movement modalities.
• The idea behind this is to have future AI systems cross-reference this data in similar ways that current AI systems do for text inputs. For instance, a virtual reality device is.

Conclusion

• The multimodal model has the potential to revolutionize the way we process and analyze information.
• Incorporating data from multiple modalities allows AI systems to achieve greater accuracy, efficiency, and human-like reasoning, leading to a more intelligent and connected world.

(*Click this link to read prelims specific weekly current affairs articles)

FAQs

MCQs

Question: With the print state of development, Artificial Intelligence can effectively do which of the following?

1. Bring down electricity consumption in industrial units
2. Create meaningful short stories and songs
3. Disease diagnosis
4. Text -to -Speech Conversion
5. Wireless transmission of electrical energy

Select the correct answer using the code given below: (UPSC CSE 2020)

(a) 1, 2, 3 and 5 only

(b) 1, 3 and 4 only

(c) 2, 4 and 5 only

(d) 1, 2, 3, 4 and 5