Cultural Heritage and Inclusive Societies

Area/s: Cultural Heritage and Inclusive Societies

Organization: Pompeu Fabra University

Research theme code : RLA-UPF-02

This fellowship brings together two dynamic and distinct domains—Personalized Education and Creative Technologies and Smart Mobility—under a shared focus on leveraging advanced AI methodologies to address complex, real-world challenges. Candidates will have the flexibility to work in either domain, contributing to ground-breaking research that bridges multimodal learning, reinforcement learning, and applied AI systems. Both research directions emphasize the integration of multimodal AI techniques to process and interpret complex datasets—whether in the form of music, video, or transport dynamics.

Key areas of focus include*:

  1. Personalized Education and Creative Technologies: Explore innovative AI methodologies that drive advancements in music and audio understanding, with applications spanning creation, distribution, education, and preservation. Develop cutting-edge approaches for semantic and geometric analysis of dynamic video content, seamlessly integrating multimodal data such as video, audio, and text. Emphasis will be placed on designing task-specific models, establishing robust evaluation metrics, and building prototypes to support transformative applications in educational and creative industries.
  2. Smart Mobility: Drive innovation in robust reinforcement learning algorithms to enhance decision-making processes in autonomous transport systems. Focus on tackling key challenges such as improving sample efficiency and ensuring robustness in sequential decision-making, with practical applications in optimizing resource transport and developing advanced mobility solutions.

Key Responsibilities*:

  • Develop AI methodologies for understanding and processing music and audio content, focusing on applications in creation, distribution, education, and preservation.
  • Advance semantic and geometric analysis of dynamic video content by
  • Candidates are not expected to focus on or address all these items simultaneously; these represent the broader research directions of the role integrating multimodal data (video, audio, and text) to build task-specific models, evaluation metrics, and prototypes for educational and creative applications.
  • Design and implement robust reinforcement learning algorithms to optimize decision-making in autonomous transport systems, addressing challenges like sample efficiency and robustness in sequential decision-making.

Qualifications:

  • PhD in Artificial Intelligence, Machine Learning, or a related field.
  • Expertise in at least one of the following: multimodal data processing, reinforcement learning, or computational creativity.
  • Proven ability to bridge theoretical AI research and real-world applications.
  • Excellent collaboration and communication skills to work in interdisciplinary teams.

Available Infrastructures: UPF houses infrastructure for both engineering work (computing, robotics and sensing, specialised audio-visual equipment – VR, audio, etc.) and experimentation with human subjects (experimental rooms with specialised equipment), complemented with links to hospitals and other external facilities.

Possible Principal Investigators:

  • Xavier Serra leads research on advanced learning techniques to enhance automatic music and audio processing: sound and music description, information retrieval, singing voice synthesis, audio source separation.
    Email: xavier.serra@upf.edu
    Web: https://www.upf.edu/web/xavier-serra
    Possible Secondments: BMAT.
    Keywords: Sound and Music Computing; Music Information Retrieval; Computational Musicology.

  • Coloma Ballester, Pablo Arias, Gloria Haro, and Federico Sukno lead research on advanced learning techniques for digital media industries, education and entertainment, namely geometric and semantic understanding of the dynamic content of videos using multimodal information.
    Email: coloma.ballester@upf.edu
    Web: https://www.upf.edu/web/ipcv
    Possible Secondments: MediaPro, Brainstorm, Eurecat.
    Keywords: Computer Vision; Scene Analysis and Understanding; Multimodal Learning; Recognition.