Uploaded April 2024 | Updated September 2026, 3 hours ago
As you are graduating from ideas to engineering, one of the key concepts to be aware of is Parallel Computing and Concurrency. 🔒🔓
I am SUPER excited to share our 94th Weaviate podcast with Magdalen Dobson Manohar! Magdalen is one of the most impressive scientists I have ever met, having completed her undergraduate studies at MIT before joining Carnegie Mellon University to study Approximate Nearest Neighbor Search and develop ParlayANN. ParlayANN is one of the most enlightening works I have come across that studies how to build ANN indexes in parallel without the use of locking.
In my opinion, this is the most insightful podcast we have ever produced into Vector Search, the core technology behind Vector Databases. The podcast begins with Magdalen’s journey into ANN science, the issue of Lock Contention in HNSW, further detailing HNSW vs. DiskANN vs. HCNNG and pyNNDescent, ParlayIVF, how Parallel Index Construction is achieved, conclusions from experimentation, Filtered Vector Search, Out of Distribution Vector Search, and exciting directions for the future!
I also want to give a huge thanks to Etienne Dilocker, John Trengrove, Abdel Rodriguez, Asdine El Hrychy, and Zain Hasan. There is no way I would be able to keep up with conversations like this without their leadership and collaboration.
I hope you find the podcast interesting and useful!
ParlayANN: arxiv.org/pdf/2305.04359.pdf
Learn more about Magdalen's career here: magdalendobson.github.io
Chapters
0:00 Welcome Magdalen!
0:29 Journey to ANN Science
3:05 ParlayANN
6:06 Lock Contention in HNSW
9:20 pyNNDescent
11:16 2-Hop Neighborhoods
12:32 Long range edges
15:34 HCNNG
20:18 ParlayIVF
26:06 Parallel Index Construction
35:07 Experimental Results
39:22 Filtered Vector Search
44:15 Range Filters
46:15 ACORN
47:52 Filtered Graph Repair
49:42 Out of Distribution Queries
53:12 Dynamic Graph Indexes that can update without locks
57:08 What directions for the future excite you the most?
▬▬▬▬▬▬▬▬▬▬▬▬ CONNECT WITH US ▬▬▬▬▬▬▬▬▬▬▬▬
Visit weaviate.io
Star us on GitHub github.com/weaviate/weaviate
Stay updated and subscribe to our newsletter: newsletter.weaviate.io
Try out Weaviate Cloud Services for free here: https://console.weaviate.cloud/
Got questions?
Forum: forum.weaviate.io
Slack: weaviate.io/slack
Connect with us on
Twitter: twitter.com/weaviate_io
LinkedIn: linkedin.com/company/weaviate-io
As you are graduating from ideas to engineering, one of the key concepts to be aware of is Parallel Computing and Concurrency. 🔒🔓
I am SUPER excited to share our 94th Weaviate podcast with Magdalen Dobson Manohar! Magdalen is one of the most impressive scientists I have ever met, having completed her undergraduate studies at MIT before joining Carnegie Mellon University to study Approximate Nearest Neighbor Search and develop ParlayANN. ParlayANN is one of the most enlightening works I have come across that studies how to build ANN indexes in parallel without the use of locking.
In my opinion, this is the most insightful podcast we have ever produced into Vector Search, the core technology behind Vector Databases. The podcast begins with Magdalen’s journey into ANN science, the issue of Lock Contention in HNSW, further detailing HNSW vs. DiskANN vs. HCNNG and pyNNDescent, ParlayIVF, how Parallel Index Construction is achieved, conclusions from experimentation, Filtered Vector Search, Out of Distribution Vector Search, and exciting directions for the future!
I also want to give a huge thanks to Etienne Dilocker, John Trengrove, Abdel Rodriguez, Asdine El Hrychy, and Zain Hasan. There is no way I would be able to keep up with conversations like this without their leadership and collaboration.
I hope you find the podcast interesting and useful!
ParlayANN: arxiv.org/pdf/2305.04359.pdf
Learn more about Magdalen's career here: magdalendobson.github.io
Chapters
0:00 Welcome Magdalen!
0:29 Journey to ANN Science
3:05 ParlayANN
6:06 Lock Contention in HNSW
9:20 pyNNDescent
11:16 2-Hop Neighborhoods
12:32 Long range edges
15:34 HCNNG
20:18 ParlayIVF
26:06 Parallel Index Construction
35:07 Experimental Results
39:22 Filtered Vector Search
44:15 Range Filters
46:15 ACORN
47:52 Filtered Graph Repair
49:42 Out of Distribution Queries
53:12 Dynamic Graph Indexes that can update without locks
57:08 What directions for the future excite you the most?
▬▬▬▬▬▬▬▬▬▬▬▬ CONNECT WITH US ▬▬▬▬▬▬▬▬▬▬▬▬
Visit weaviate.io
Star us on GitHub github.com/weaviate/weaviate
Stay updated and subscribe to our newsletter: newsletter.weaviate.io
Try out Weaviate Cloud Services for free here: https://console.weaviate.cloud/
Got questions?
Forum: forum.weaviate.io
Slack: weaviate.io/slack
Connect with us on
Twitter: twitter.com/weaviate_io
LinkedIn: linkedin.com/company/weaviate-io
