In traditional search systems, finding relevant information is limited to matching exact keywords. If you search for \”refund guidelines,\” a keyword search will miss a document that only uses the word \”reimbursement.\” Modern AI search engines bypass this limitation using Vector Embeddings and Cosine Similarity, enabling semantic search.
What are Vector Embeddings?
Vector Embeddings are lists of numbers (high-dimensional vectors) generated by AI models that represent the conceptual meaning of a text string. Similar concepts are placed close to each other in this multidimensional mathematical space. For example, the vector for \”reimbursement\” will align closely with the vector for \”refund\” even though they share no common characters.
Measuring Similarity with Cosine Similarity
To rank documents, we compute the mathematical cosine of the angle between two vectors. The formula for Cosine Similarity is:
\cos(\theta) = \frac{\mathbf{A} \cdot \mathbf{B}}{\|\mathbf{A}\| \|\mathbf{B}\|}
If the vectors point in the exact same direction, the cosine is 1 (100% similarity). If they are orthogonal, the similarity is 0. By representing your search queries and database documents as vectors, you can retrieve the most conceptually relevant chunks instantly.
To test how documents are chunked and rank semantic queries locally, try our interactive browser-native Vector Similarity Playground.
Leave a comment