Video Intelligence is Going Agentic

TLDR: The Rise of Agentic Video Intelligence

• Jockey is now in research preview at twelvelabs.io/jockey - reachable via API for builders and via MCP on Claude for teams and consumers, with ChatGPT coming soon.

• The stack finally compounds. Jockey is the first TwelveLabs product to integrate Pegasus, Marengo, and Search into one agent surface. Every future model upgrade makes Jockey better automatically, with no migration required.

• We are witnessing a seismic shift: video AI is evolving from a basic analysis tool into an intelligent collaborator that can understand context, reason about content, and execute complex creative tasks.

• TwelveLabs' Jockey isn't just another AI tool; it's a sophisticated orchestrator that conducts our video-native foundation models like a symphony, turning chaotic video workflows into harmonious creative processes.

• Forget clunky video interfaces: these new video agents speak your language while letting you get hands-on with the visuals, finally bridging the gap between human creativity and machine efficiency.

• The days of wrestling with technical video tools are numbered; imagine telling your AI assistant Find the most compelling customer stories and watching it assemble a perfect highlight reel.

• For enterprises drowning in video content, this isn't just an upgrade; it's a revolution that transforms overwhelming media libraries into accessible, actionable assets through natural conversation.

You can listen to this article via this NotebookLM-generated conversation here.

1 - Introduction: The Video Intelligence Evolution

In 2025, we stand at the precipice of a fundamental shift in how machines understand and manipulate visual media. With billion hours of video content uploaded daily across platforms, traditional processing tools (designed for an era of limited content and predefined tasks) are becoming obsolete. This explosion of content has created both unprecedented opportunities and formidable challenges.

The emergence of video foundation models has evolved our capabilities from simple frame-by-frame video analysis to sophisticated spatial understanding and temporal reasoning. However, a critical gap remains between raw model capabilities and real-world applications. This is where agentic video intelligence enters the picture; representing a paradigm shift from passive analysis to active, goal-oriented interaction with video content. These systems combine video foundation models with large language models through agent planning frameworks - creating AI systems that understand not just what appears in a video, but why it matters and what actions to take.

In 2026, that shift has arrived in how machines understand and manipulate visual media. At TwelveLabs, we're addressing these challenges through Jockey, a conversational agent that combines video and photo foundation models with LLM-based reasoning through a purpose-built agent architecture. Jockey doesn't just process a single file; it reasons across an entire video and photo library by person, by moment, by place the way people actually think about their libraries.

2 - The Rise of AI Agents in the LLM Domain

The emergence of Large Language Models (LLMs) has revolutionized AI by enabling autonomous agents that can understand, plan, and act. These LLM-powered agents use natural language as their interface, allowing them to comprehend goals and execute complex tasks through sophisticated reasoning and tool integration. This capability evolved from basic prompt engineering into more advanced approaches that leverage chain-of-thought prompting, enabling agents to break down complex problems step-by-step.

Source: https://www.letta.com/blog/ai-agents-stack

Advanced agent frameworks like Open AI Agents SDK, Letta, and LangGraph have emerged as critical infrastructure, providing structured approaches for building reliable agents. These frameworks implement essential capabilities including planning, tool integration, memory management, and self-reflection. Modern agent architectures separate cognitive functions into specialized components - planners for strategy, executors for actions, and critics for evaluation - enabling them to handle increasingly complex tasks.

In real-world applications, LLM agents have proven highly adaptable. They can perform diverse tasks from software development and customer service to research synthesis - all by combining LLMs' reasoning capabilities with domain-specific tools and workflows. This architecture allows them to navigate complex problem spaces more flexibly than traditional automation while providing greater reliability than raw LLM outputs.

Source: https://weaviate.io/blog/what-are-agentic-workflows#planning-pattern

The field has identified several key design patterns that drive agent success. These include task decomposition (breaking goals into manageable steps), recursive reasoning (applying logic to intermediate results), tool augmentation (expanding capabilities through APIs), and human-in-the-loop collaboration (incorporating user feedback). These patterns work across different domains, establishing core principles for effective agent design.

These insights from LLM agent development provide crucial guidance for video intelligence systems. They show that foundation models work best within structured architectures that align with specific task requirements. They emphasize the importance of explicit planning and reflection for reliability. Most critically, they demonstrate that the most effective systems balance automation with human collaboration, creating true partnerships that leverage both machine and human intelligence. As we develop video agents, these principles will help us navigate the unique challenges of understanding and manipulating visual media.

3 - The Emergence of Video Agents: A New Paradigm

The convergence of video foundation models and LLM-based agent architectures has given rise to video agents - systems uniquely designed to understand, manipulate, and reason about visual media with unprecedented sophistication. Unlike their predecessors, these agents don't merely analyze video content; they interact with it purposefully, leveraging specialized knowledge about visual narratives, temporal relationships, and multimodal context. This new paradigm addresses fundamental limitations of both traditional video processing pipelines, which lack semantic understanding, and general-purpose AI agents, which struggle with the complex temporal and spatial dimensions inherent to video content.

What makes video agents distinct is their ability to navigate the unique challenges of video intelligence. Video data exists in a high-dimensional space combining visual, audio, and temporal elements, requiring specialized perception capabilities beyond those needed for text or static images. Computational demands scale dramatically with video length and resolution, necessitating efficient approaches to attention and memory. Perhaps most significantly, video understanding requires reasoning across multiple timescales simultaneously—from frame-level details to scene-level composition to narrative-level structure—a capability rarely needed in other domains. A system like OmAgent has pioneered a solution to these challenges through an innovative architecture combining multimodal RAG with a divide-and-conquer reasoning approach.

Source: https://www.om-agent.com/

Recent academic work has accelerated this shift, with researchers increasingly focusing on agentic frameworks for video understanding. Papers like "VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding" from Peking University demonstrate how memory mechanisms can enhance temporal reasoning, while Stanford's research on "VideoAgent: Long-form Video Understanding with Large Language Model as Agent" explores efficient searching and aggregation of relevant information from lengthy videos. These approaches share a common insight: rather than processing all video content exhaustively, effective agents should reason about what's important, selectively attend to relevant segments, and dynamically build understanding through iterative analysis.

Source: https://wxh1996.github.io/VideoAgent-Website/

There are also works such as LAVE that combines LLM reasoning with video-specific tools. These systems typically structure their workflow around specialized components: perception systems for extracting multimodal features, memory mechanisms for storing and retrieving video context, planning modules for decomposing complex queries, and execution components for interfacing with video processing tools. This modularity allows them to combine the strengths of foundation models with domain-specific knowledge about video content and editing operations.

Source: https://arxiv.org/abs/2402.10294

As video agents evolve from research prototypes to production systems, they are increasingly addressing real-world challenges in content creation, media analysis, and information retrieval. Early adopters in broadcasting, film production, and social media are already leveraging these capabilities to automate previously manual tasks like highlight generation, content moderation, and promotional clip creation. The most advanced systems demonstrate an emerging ability to understand not just the surface content of videos, but their underlying purpose, style, and narrative structure—enabling interactions that feel more like collaborating with a knowledgeable assistant than operating a complex tool. This transition from passive analysis to active collaboration represents perhaps the most significant paradigm shift in video intelligence since the advent of multimodal AI.

4 - TwelveLabs Approach: Jockey as a Video Agent Framework

Now in research preview, Jockey represents TwelveLabs' first product to integrate its entire technology stack into one place. At its heart, Jockey leverages Pegasus 1.5 for advanced video-to-text understanding and image-to-text understanding (Image VLM + Image Captioning), Marengo 3.5 (with the Eclipse/V3 embedding upgrade) for semantic video and image search, and Search 1.5/2.0 for multi-step retrieval with iterative refinement. Rather than replacing these models, Jockey strategically coordinates them through a planner-worker-reflector architecture, and because Jockey sits on top of the full stack, every future model upgrade (a better Pegasus, a better Marengo, a better Search) compounds directly into a better Jockey, with no separate integration work required.

5 - Bringing MCP Interface to Jockey

The Model Context Protocol (MCP) is no longer a future possibility for Jockey. It's how consumers reach the product today. Jockey ships as an MCP connector on Claude and soon on ChatGPT, letting anyone connect their personal photo and video library and ask Jockey to find any person, place, or moment across their entire collection, directly inside a tool they already use. Alongside MCP, Jockey exposes an API for builders who want to integrate video and image intelligence directly into their own applications. Both surfaces run the same underlying agent; you don't get a different AI depending on where you connect, you get the same one.

6 - The Power of Multimodal Interfaces

As of research preview, Jockey has full parity between video and image assets. It has the same conversational interface, the same thinking-state transparency, now returns mixed results across a user's entire photo and video library. Think of it as: "Your entire video and photo library - organized, searchable, and understood the same way you think about it." Mixed modality isn't a nice-to-have; a library whether for personal or business use is inherently both photos and videos, so a video-only interface is an incomplete experience from day one.

7 - Transforming Content Creation: Jockey's Vision for Digital Marketing Workflows

Nowhere is Jockey's impact more immediate than in digital marketing - a discipline where creative teams are drowning in their own output. Brands now run hundreds of video ad variants across TikTok, Meta, and YouTube simultaneously, and someone, somewhere, is expected to know why one outperformed the other nineteen. Today that someone scrubs through raw footage by hand, squinting at hook timing and CTA placement, trying to reverse-engineer a pattern from gut feel. Jockey ends that era.

Instead of a Director of Paid Social manually re-watching a quarter's worth of ad creative, they simply ask:

"Take the three best-performing ads, summarize what's in them via Jockey, cross-reference what the data says worked, and write me a creative brief for the next ad in that line."

Jockey watches the footage, reads the performance data alongside it, and hands back a brief. Ask it to go further, and it will:

"Find ads that overperformed on Instagram relative to YouTube. Have Jockey look at their content and explain what about them suits the IG feed vs YouTube."

"For the five ads with the worst performance, have Jockey describe what happens right after the hook. Is there a common dead spot?"

This is the shift this piece has been building toward all along: video stops being something a person has to sit through, and becomes something an agent can reason about at the scale a marketing calendar actually runs at. Jockey will tag every asset in a library, such as talking-head vs. b-roll, on-screen text, music energy, exact CTA timing, shot count, and join those tags to completion rate and CPM, so a team can finally ask the question that matters:

"For every ad, get Jockey to tag: talking-head vs b-roll, on-screen text yes/no, music energy, when the CTA appears, shot count. Join that to completion rate and CPM and tell me which attributes actually correlate with efficiency."

The same capability compounds outward from analysis into production. A brief becomes a rough cut:

"Based on this creative brief, find relevant clips from our b-roll content and put together a rough cut for a 30s ad."

And a library of demos becomes a structured, queryable asset:

"Find all product demo clips where the spokesperson mentions a specific feature. Tag by feature name, clip length, and the CTA used - output as a table."

Even the notoriously fuzzy task of onboarding a new creative agency collapses into a single ask:

"We're briefing a new agency. Have Jockey analyze our top 20 ads and write a 'what works for our brand on video' summary they can use as a creative guardrail."

What used to take a marketing team days of manual review, and a fair amount of guesswork, now takes minutes and comes back with an answer grounded in the footage itself. That's not an incremental workflow improvement; it's the same agentic shift this article opened with, just landed on the desk of a performance marketer.

Beyond marketing, AI-native creators can use Jockey (via the Claude MCP connector) as a retrieval-and-editing copilot for their own archive. Instead of scrubbing through hours of footage to find “that one moment,” you describe what you’re trying to make (topic, character, vibe, pacing) and Jockey pulls the best supporting clips across your library or knowledge store, then structures them into something you can actually publish: a Shorts sequence, a Reel outline, or a shot-by-shot timeline with suggested on-screen text and captions.

Try prompts like: “Based on this script, find the best supporting clips from my knowledge store and assemble an engaging 2-minute social post.” Or: “Turn this long-form video into 3 Shorts. For each Short: write a hook, pick clips, add suggested on-screen text, and output a shot-by-shot timeline.”

8 - The Future of Video Agents: Our Vision and Roadmap

The future evolution of video agents will be guided by three core principles that extend beyond incremental improvements to fundamentally transform how humans interact with visual media. With Jockey, we're building toward a vision where video intelligence becomes a natural extension of creative and analytical workflows, anticipating needs and adapting to individual preferences while maintaining the balance between automation and human creative control.

Flow-Aware Trajectories represent our first guiding principle: systems that anticipate user needs several steps ahead to maintain creative momentum. In practice, this means Jockey will predict which clips might be needed next during highlight creation, suggest appropriate transitions based on content, and even pre-assemble rough cuts that align with a user's editing patterns. As these capabilities mature, editors will experience fewer interruptions to their creative process, with the system handling technical details while preserving creative direction. For example, a sports producer could assemble a highlights package with Jockey automatically suggesting key plays based on crowd reactions, maintaining consistent pacing, and ensuring narrative flow; all while the producer maintains creative control over the final selection.

Contextual Meta-Learning enables Jockey to evolve with each interaction, building persistent understanding of user preferences across projects and sessions. Rather than requiring explicit instruction for every operation, the system will recognize patterns in user behavior (preferred transition styles, typical scene structures, or common editing sequences) and adapt its suggestions accordingly. This creates a virtuous cycle where the system becomes more valuable over time, remembering project-specific requirements and adapting to feedback. Organizations working on recurring content types (like weekly sports highlights or product reviews) will find their workflows becoming increasingly efficient as Jockey learns to prioritize relevant content and apply consistent styling without explicit direction.

Multimodal Intelligence at Scale allows Jockey to make sophisticated connections across video, audio, and textual elements without relying on rigid rules. This principle enables complex operations like retrieving scenes based on thematic relevance rather than just visual similarity, analyzing narrative flow across multiple clips, or identifying emotionally resonant sequences automatically. As we integrate additional third-party capabilities (such as audio generation for voiceovers, video generation for alternative scenes, or avatar creation for presentations), Jockey will orchestrate these specialized tools into cohesive workflows, maintaining context across operations and ensuring consistent quality in the final output. The integration roadmap includes partnerships with leading tools for segmentation, background replacement, synthetic media generation, and audio creation, significantly expanding Jockey's capabilities without requiring users to manage complex technical integrations.

The evolution of Jockey from tool to collaborative partner represents a paradigm shift in how organizations approach video assets. By focusing on these three principles and building an extensible ecosystem of capabilities, we're creating a platform that adapts to how people naturally think about video content rather than forcing users to adapt to technical limitations. For developers building on this platform, Jockey offers the opportunity to create applications that leverage sophisticated video intelligence without the complexity of integrating disparate models. For enterprises seeking to scale visual content creation, it provides a foundation for consistent, high-quality output that respects established workflows while dramatically improving efficiency.

9 - Conclusion: Embracing the Video Agent Revolution

Jockey has moved well past alpha.

Jockey is now in research preview: self-serve, with real usage limits and pricing. Get started today at twelvelabs.io/jockey - sign up through Playground, then connect via the Jockey MCP connector on Claude, or build directly against the API.

Pricing: Free (5GB storage, 15 responses/day), Plus ($20/mo - 100GB, 40 responses/day), Pro ($100/mo - 500GB, 200 responses/day).