Moviescope: Large-scale Analysis of Movies using Multiple Modalities

News Release Summary

Researchers at the University of Virginia and Microsoft have released Moviescope, a dataset of 5,000 movies pairing video trailers, audio, movie posters, text plot summaries, and metadata pulled from sources including YouTube, Wikipedia, and IMDb, in order to systematically test how well different types of data can predict high-level movie attributes like genre and production budget. The team found that simple averaging operations over word or frame embeddings — methods they call fastText and fastVideo — consistently outperformed more computationally expensive approaches like LSTM recurrent networks and action-recognition models designed for short clips, suggesting that for holistic movie-level classification, preserving temporal order matters less than researchers might expect. Text-based plot summaries turned out to be the strongest single predictor of genre, edging out video and even structured metadata, while audio proved surprisingly useful for estimating budget — outperforming the video signal from the same trailer. A human study using Amazon Mechanical Turk showed that people performed only marginally better than the models, with humans doing best when reading plot text and struggling most with raw video frames. Combining all five modalities together yielded the best overall results, confirming that each data type captures something the others miss. The work matters because most existing video datasets focus on short, isolated action clips, whereas Moviescope is designed for the kind of long-range, narrative-level understanding that movies demand, and the authors are releasing their dataset, pretrained embeddings, and code to give other researchers a practical benchmark for multimodal video analysis.