Vector databases: When a girl goes to shop for a scarf

20 kmph! Now that can be the speed of a car (specially if it’s driving in Bangalore or Mumbai). But 20kmph towards the Northside- that’s more than speed. That’s velocity.

Turns out one is a scalar data and the other is a vector data. And that’s why pulling out and dishing out this data needs a different kind of database. It’s a spaghetti plate situation. It needs a spork instead of a fork.

Your normal database works on rows and columns. This one works on vectors. Yeah, the same term we studied in the maths class: anything that has both magnitude and direction. A spreadsheet will tell you how to find a number- in which row or column. But that will not work for a vector- a list of numbers which indicates a location within a space.

Basically a vector database is a database that stores vector embeddings (or vectors generated by neural networks). A vector form of data can be anything that has a complex numerical or semantic texture. Where exact value may not matter much but where the similarity factor holds real weight. Where context and memory get precedence over specificity. Nowadays, it is a lot about data that is being fed for AI to retain and learn from memory for complex tasks – helping them draw patterns, make comparisons, use clustering, see things in multiple dimensions, understand relationships and make sense of the fabric underlying various connections. They have attributes that are generated by AI or ML and are complex to manage. This is perhaps why, traditional databases or forks are not fit to grab the complexity and scale of vector data. And just an index won’t work. You need a spork to hold it all well. Unlike a vector index (which can be limited to search and retrieval functions), a vector database can handle the big picture of data storage, scalability, meta-data and managing the vectors.

The approaches used are many - hashing, quantisation, graph-based search, Approximate Nearest Neighbour (ANN) search, CRUD operations, metadata filtering, horizontal scaling, and serverless architectures. Vector databases help to get just the right portion of spaghetti without spilling or breaking anything.

As AI gathers pace and ground, vector databases would become mainstream. However, there are already signs of the need of something sharper, deeper and lightning-fast than what we have today- specially as AI models evolve to new levels, as multi-modal architectures gain steam and as Retrieval Augmented Generation (RAG) demands better databases. Time for a splade (knife plus spoon plus fork) replacing the spork. Anytime now.