Skip to main content
MICHIEL HORSTMAN
BOOK A CALL ↗

Vector Databases for Dummies: Why Embeddings Need Memory

Vector DatabasesEmbeddingsSemantic SearchAI MemoryVector Search
Vector Databases for Dummies: Why Embeddings Need Memory

The real reason AI forgets what you said five minutes ago -- and how embeddings, vector search, and memory systems fix it without a bigger model.

Vector databases are retrieval systems that store and query high-dimensional embeddings by semantic similarity instead of exact keyword matches, giving AI systems persistent memory.

image

AI systems appear knowledgeable yet forgetful. They can generate creative explanations, summarize complex domains, and adapt to new prompts, yet fail to remember simple details from earlier in a conversation. Read time: 9 min. This contrast is not caused by a lack of intelligence. It is caused by a lack of structured memory.

Before we get into memory, start with the basics: Embedding For Dummies: Beginner's Guide to AI's Hidden Language This article explains embeddings, the hidden language AI uses to represent meaning. It's the low-level layer you need to understand before tackling vector databases and memory.

Once you see how embeddings work, the paradox becomes clear: chatbots feel infinitely wise and infinitely forgetful at the same time. The culprit isn't intelligence or parameters. It's memory.

For a full walkthrough of how retrieval fits into the production pipeline, see RAG for Dummies.

TL;DR

Transformers provide intelligence; vector databases provide memory. Embeddings translate meaning into coordinates, and vector search retrieves relevant information by similarity rather than keywords. Together, they enable AI systems that can recall, connect, and reason across time.

Why do large language models forget?

Imagine a system with vast knowledge and extraordinary generative ability, yet unable to retain what you said a few messages earlier. That is the reality of today's large language models. They operate within fixed context windows that, despite being large, fill up quickly and degrade in reliability as they grow.

That's every large language model today. They dazzle, but they don't recall.

Yes, models like GPT-4 now come with context windows the size of novellas (128k tokens = 96,000 words). On paper, this feels enormous. In practice, conversations and documents accumulate far faster than you'd expect. Stack up a few reports, an email chain, and a draft strategy doc, and suddenly you're hitting the ceiling.

The naive solution? Make the notebook bigger. But language models are built on attention mechanisms, and those scale quadratically. Double the context, quadruple the compute. Quadruple the compute, and your latency balloons. At a certain point, responses take thirty seconds or more. That's not "AI assistant" territory anymore; that's waiting for your 1998 printer territory.

And even if you pay for the horsepower, there's diminishing returns. Past 100k tokens, the model's recall and reasoning begin to fuzz out. More paper doesn't mean more memory.

The real fix isn't brute force. It's smarter filing. Humans don't recall everything we've ever seen verbatim; we store concepts, link them, and retrieve them when relevant. That's what vector memory gives to machines.

image

How do embeddings turn meaning into coordinates?

Embeddings are mathematical vectors that represent semantic meaning. They place words, sentences, or entire documents inside a high-dimensional coordinate system where similar ideas cluster together. This transforms abstract meaning into measurable distance.

Think of it as GPS for ideas.

Every word, sentence, or document can be converted into a vector a long list of numbers that represent its semantic position in space. "King" might land at [0.2, 0.8, -0.3, ...] while "Queen" hovers nearby because they share overlapping meaning. The space isn't three dimensional, like a map, but 1,536-dimensional in popular embedding models. Hard to visualize, but the math works.

Picture a galaxy:

  • "Dog" and "Puppy" are neighboring stars
  • "Cat" orbits in the same system
  • "Democracy" lives galaxies away
  • Distance equals difference in meaning

What makes embeddings magical is context. "Bank" in the sense of finance and "bank" as a river's edge don't collapse into the same coordinates. The model learns to separate them by exposure to billions of examples. It captures the subtle connections we intuitively grasp but can't easily program into rules.

And it's cheap. OpenAI's ada-002 embeddings cost $0.0001 per 1,000 tokens. For a few cents, you can map entire knowledge bases into navigable meaning space.

The upshot? Embeddings let machines "think" in terms of similarity. They give us a way to turn meaning into math. But once you have a galaxy of points, you need a telescope to navigate it. That's where vector databases come in.

image

What makes vector databases different from traditional databases?

Traditional databases are built for precision. Ask for "dogs", and they'll only return documents containing the exact string "dogs". No "dog", no "puppies", no "canines". It's binary: either a match, or nothing.

This is keyword search. It's like Ctrl+F on a giant PDF -- powerful if you know the exact word, useless if you don't.

Vector databases work differently. Instead of asking "Who said 'dogs'?", you ask "What's closest in meaning to 'puppies'?" and get semantically related answers: dogs, canines, man's best friend. It's search that works like thought, retrieving ideas by meaning rather than literal text.

Here's the difference in code:

Keyword Search vs Semantic Search

-- Traditional (Keyword)SELECT * FROM notes WHERE text = 'dogs';
-- Vector (Semantic)SEARCH notes WHERE similarity(embedding, 'puppies') > 0.8;

In keyword search, "dogs" != "dog". In semantic search, "puppies" finds "dogs" because their embeddings live in the same neighborhood of meaning.

image

Why does this matter for AI?

Keyword search returns results only when exact strings match what you typed. Semantic search retrieves information based on meaning, allowing AI systems to surface relevant ideas even when wording differs. This shift unlocks more natural, intuitive interaction.

Vector databases are not faster databases -- they are a fundamentally different category of retrieval engine built for meaning-based search.

This shift is why vector databases feel like GPS for memory. Instead of a static lookup table, they let AI navigate a multidimensional map of concepts.

The global vector database market is projected to reach $4.3B by 2028 (MarketsandMarkets), driven by demand across search, recommendation, and AI memory systems.

The key trick under the hood is Approximate Nearest Neighbors (ANN). Searching a giant vector space naively means comparing every new query to every existing vector -- unbearably slow. ANN algorithms cut corners intelligently, trading a fractional loss in accuracy for thousandfold gains in speed.

Different strategies exist:

  • HNSW builds hierarchical "highways" that route you toward likely neighbors.
  • IVF clusters vectors into neighborhoods and only checks the relevant one.
  • LSH hashes vectors into buckets so only similar ones collide.

The details differ, but the goal is the same: lightning-fast similarity search in spaces too vast to brute force.

Which vector database provider should you choose?

The ecosystem has exploded.

  • Pinecone: Fully managed, with a developer-friendly API. You pay for the simplicity, and it gets pricey at scale.
  • Weaviate: GraphQL-native, supports multimodal data (text, images, etc.). Feature-rich but setup is heavier.
  • Qdrant: Rust-based, high-performance, filtering-friendly. Growing quickly, but community is still small.
  • Milvus: Handles massive scale, GPU acceleration, open source. But resource hungry.
  • Chroma: Lightweight, dead simple to start, ideal for prototyping. Not yet hardened for enterprise.

Each has its trade-offs. Pinecone is Apple like in ease. Qdrant is the hacker's choice. Milvus is the enterprise warhorse. Chroma is the scrappy starter kit.

| DB Provider | Free / Starter Option | Base Monthly Cost (Paid Tier) | What You Get at Entry | When Costs Increase Most |
| ----------- | ----------------------------------------- | ----------------------------------------------------------------------- | --------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
| Pinecone | Yes Free / Starter tier | ~$50/mo minimum under Standard plan | Basic serverless vector search, limited indices, basic SLA | As you increase vectors, query volume, or want higher performance / SLA |
| Weaviate | Yes Free trial / sandbox | ~$25/mo (Serverless) for standard tier | Basic performance & availability, vector-storage proportional to dimensions | Dimensions, vector-count, and moving to business-critical SLA |
| Qdrant | Yes Low-tier / open-source / small clusters | ~$200/mo for mid-scale (~10M vectors) | Good performance, filtering, Rust-based efficiency | Large vector counts, heavy query loads, managed services vs self-hosting |
| Milvus | Yes Open source self-hosted | Varies (cloud-managed higher) | High scalability, many index types | Infrastructure cost (GPU, clusters, distributed mode) dominates |
| Chroma | Yes Open source starter / prototyping | Dependent on deployment | Fast to set up for prototyping / RAG experiments | Production usage, large datasizes, need for durability / many queries |

How do you store and retrieve memory with Chroma?

With Chroma, you can spin up your own semantic memory in minutes:

from chromadb import Client
client = Client()
collection = client.create_collection("memories")
collection.add(
    documents=["I love puppies", "Dogs are loyal"],
    ids=["1", "2"]
)
results = collection.query(
    query_texts=["cute animals"],
    n_results=2
)
print(results)
# -> ["I love puppies", "Dogs are loyal"]

Congratulations, you've built a memory system smarter than most chatbots today.

Where is vector memory already working?

If this still feels abstract, look at where it's already working.

ChatGPT's retrieval plugin is powered by vector databases. It lets the model search your files in real time. Ask "What did the Q3 report say?" and instead of guessing, it pulls the actual document.

Spotify doesn't just recommend music by genre. Every song is embedded as a vector of "vibes": tempo, mood, timbre. When you hit "More like this," you're triggering a nearest-neighbor search in a giant embedding space of tracks.

GitHub Copilot isn't memorizing snippets. It builds embeddings of code patterns, which lets it suggest solutions based on meaning, not just syntax. It knows you want a debounce function even if you didn't say "debounce."

Building agents that retain knowledge across long-running conversations takes this a step further. Read AI Agent Memory That Actually Works for a deeper look at persistent memory architectures.

Every AI breakthrough of the last two years has leaned on vector memory. The pattern is so consistent it's almost boring. Without it, models remain goldfish professors. With it, they become something closer to partners.

How do you build your own AI memory system?

Here's the simplest possible quickstart with Chroma:

pip install chromadb
import chromadb
chroma = chromadb.Client()
brain = chroma.create_collection("my_brain")
brain.add(
    documents=[
        "I prefer React over Vue",
        "TypeScript saves debugging time",
        "Tailwind makes CSS bearable"
    ],
    ids=["pref1", "pref2", "pref3"]
)
answer = brain.query(
    query_texts=["What's my frontend stack?"],
    n_results=1
)
print(answer)
# -> "I prefer React over Vue"

In under ten lines, you've given your AI persistent memory. Now expand it. Upload your notes. Index your research papers. Add your email archives. You've built your own semantic search engine.

Next steps:

  • Run the snippet above (two minutes)
  • Feed it personal notes or docs (five minutes)
  • Spend a weekend turning it into your own knowledge assistant

A persistent memory layer is the single highest-leverage improvement you can make to any AI system you build.

Image Suggestions

  1. Cover image alt text: Diagram showing a 2D vector space with semantic clusters labeled "king," "queen," "puppy," "dog" positioned by meaning proximity, with dotted lines connecting related concepts to illustrate embeddings memory and vector search.
  2. Context window vs vector memory comparison: Side-by-side illustration of a chatbot with a small fixed context window and a vector database storing unlimited historical embeddings. Shows how vector databases provide scalable AI memory systems.
  3. ANN algorithm visualization: Abstract 3D space with HNSW hierarchical layers connecting neighboring vector clusters, demonstrating how approximate nearest neighbor search navigates high-dimensional spaces for semantic retrieval.

Internal Links

My Take

I used to think AI progress was mostly about bigger models. More data, more compute, more parameters. Watching vector databases rise has humbled me. The real leap isn't just raw intelligence; it's memory. The future of AI isn't about models that know everything. It's about models that remember the right things, in the right way, at the right time.

Subscribe to my newsletter at decod3d.beehiiv.com for early access to new articles and technical breakdowns on AI systems.


Character Counts:

  • Subtitle: 145 characters (max 160)
Back to all posts

Surface. Thanks for diving.

Shark model by alextkd2003, CC BY, converted to ASCII.