I indexed 37h of my YouTube library using an RTX 4090 and local ML models in 24h

Following my recent blog post and Hacker News post. where I ran the desktop app on my M1 Max. Today, I’m using the self-hosted version running in Docker with an NVIDIA RTX 4090 (24 GB VRAM).

The content is also fundamentally more demanding: long podcast episodes with at least two faces in every frame, coding tutorials packed with on-screen text, and screen recordings. GoPro footage is mostly wide outdoor shots.

But NVIDIA was faster than my M1 Max.

Here’s the full spec of the rented server:

Here’s my YouTube channel: https://www.youtube.com/@iliashaddad_dev

The key difference from the GoPro run: this time I used the source-available Docker version (github.com/IliasHad/edit-mind) rather than the desktop app, running on a machine with an RTX 4090 and CUDA 12.8.0 (I rented from a cloud provider)

Summary

The indexing process

The self-hosted version runs the same pipeline as the desktop app. Point it at a folder, and it works through each video in stages:

1. Transcription: the full audio track is transcribed using OpenAI Whisper
2. Frame analysis: the video is divided into scenes (1-2.5 seconds), and each frame runs through a set of ML plugins: face recognition, on-screen text detection, object detection, scene description, dominant color, and shot type
3. Embedding: scene data is embedded into a local vector database across three collections: text, visual, and audio

Stage breakdown

• Coding tutorials and screen recordings: TextDetectionPlugin It is the most expensive plugin
• Podcast and interview episodes: FaceRecognitionPlugin and DescriptorPlugin dominate, the face model runs significantly harder than it ever does on my GoPro videos.

Fastest video

A Day in the Life of a Remote Software Engineer in Malaysia | Work, Life, and Culture.mp4

Slowest video

Note: (excl. short clips)

Ep 08: Interview with Scott Tolinski, founder of LevelUpTutorials and co-host of the Syntax podcast.mp4

Frame analysis plugin averages

Longest video

EP 10: Building a Shopify App In Public (Part 1).mp4

Plugin breakdown:

Here are a couple of example video clips using these prompts:

Generate me a compilation of videos where transcription has the word "welcome” - https://youtu.be/OButxl1-A0k

Generate me a compilation of videos where transcription has the word "edit mine" - https://youtu.be/OzauaIc5kOk

Here are a couple of example screenshots using the search:

Wrapping up

The big takeaway: an RTX 4090 running Edit Mind in Docker via CUDA is faster than my M1 Max desktop app for this kind of workload, even on more demanding content.

And the best part is you can use the desktop app as a client for the self-hosted Docker server to utilize the video editing software integration and native MacOS desktop experience

If you wanna check the full processing jobs data, here’s the JSON file. And you can search the video file name in my YouTube video to watch the full video.

What’s next?

If you want to try it yourself, the self-hosted version is at github.com/IliasHad/edit-mind. Add a folder, and it handles the rest. The desktop app (with integrations for Final Cut Pro, DaVinci Resolve, and Adobe Premiere Pro) is available at edit-mind.com.

If you have ideas for performance improvements, new plugins, or just want to share what you're working on in the same space, open an issue or create a pull request.