MarkLogic
Vector Search

Vector search uses mathematical representations of data (vector embeddings) to find similar items based on their meaning rather than keyword matching. The embeddings encapsulate the semantic essence of content like text, images, video and other data types into numbers. Vector embeddings are generated by vector classifiers included in LLMs, and represent high-dimension floating points in a multi-dimensional vector space, allowing for similarity comparisons that understand context and meaning. Similarity is measured based on the distance between the vectors and uses Euclidian or cosine similarity calculations.

Depending on the algorithm, vector search can be exact or approximate. A k-Nearest Neighbors (kNN) algorithm finds the closest vectors to a query vector using distance metrics like cosine similarity or Euclidean distance. It is an exact method, but slow and resource heavy, posing scalability challenges for large datasets. The Approximate Nearest Neighbor (ANN) algorithm on the other hand uses an index to find results that are close enough, trading accuracy for speed. ANN search is fast, which makes it more suitable for large data sets and real-time applications.

Vector search is often called semantic search as the vector embeddings used in this type of search capture the semantic meaning of data like words and images. Both semantic search and vector search represent information retrieval techniques to find the most relevant information to a user’s question, but they leverage different models, indexes and query languages. Vector search uses vectors (or mathematical representations of information) to find conceptually similar items based on their distance in a vector space. Semantic search uses graph data like metadata and semantic triples (used in knowledge graphs) to interpret user queries based on context and relationships between information entities. Graph data is indexed with the triple index and queried with the SPARQL language.

Vector search is an information retrieval method while large language models, or LLMs, are a type of artificial intelligence trained on copious amounts of available data that are able to generate information like text or perform natural language tasks. While vector search can query data in real time, LLMs can produce answers only based on the information they have learned. Vector search can enhance an LLM, in retrieval-augmented generation (RAG) systems for example, to improve the accuracy and relevance of LLM outputs.

MarkLogic Server is more than a vector database. It’s a multi-model database that natively supports vectors alongside data models like document, graph, geospatial, temporal and sequel. The built-in search capabilities leverage a universal index, a vector diskANN index, range queries and many more advanced features though one API that allows you to query across all data types concurrently and use a hybrid result score for the final output.