Dynamic Memory Retrieval

Pre-requisites:

Introduction:

Dynamic Memory Retrieval

The DMR module built by us is one of the core components of why DrDroid is able to understand production issues and queries well. The agent retrieves accurate and most recent information.

1b. The Unified Index

An index is defined uniquely for every client on DrDroid platform — it's a measure of the records.

The index size could vary depending on

(a) The expanse & scale of the infrastructure & applications

(b) The integrations added to DrDroid

(c) The duration of time for which your account is using DrDroid & the retention policy (short term memory is continuously evolving based on relevance)

Some sample index sizes:

Type	Long term records (Stateless, long term retention, can be regenerated)	Short term records (stateful, ephemeral, built over time, limited retention)
Startup with 1-20 engineers, early stage	100-2000 records	10-100 records/day
Enterprise with 500+ engineers, multi-tenant, multi-cloud infrastructure	50k+ records	2000+ records/day

What do these indexes look like?

These indexes comprise of different types of records depending on the kind of application or tooling your company uses. Here are a few examples of record types that we have:

1. Service:

A typical example here could be an asset extracted from an APM tool or Kubernetes cluster. This record would have context about the kind of data that the tool has, about that specific entity.

2. Repository readme:

A code repository (e.g. Github) would typically have a record type of "repository readme" — This record is generated by a LLM agent who's proficiency lies in generating a feature + use-case scoping of any given repository.

3. Dashboards:

A tool like Grafana, Datadog or Signoz would often have a dashboard record type. This record would have granular+nested details, from title to variable to panel queries and descriptions.

4. Log Patterns:

Any logging tool would typically have a record type called log pattern. This record would contain the overarching structure of the logs in that tool, the different services/labels that often occur in that log set.

5. Users:

A collaboration or ticketing tool like JIRA or Slack could have a record type Users indexed for it. This would comprise of user information from that tool and often, their unique ID mappings.

6. Table schema:

For a database, table schema/document structure could be a record type, comprising of information of the columns in a table, the kind of data each column stores and (potentially) some sample data.

…

We have 200+ record types across 80+ integrations that we support.

1c. Agentic Retrieval: Extracting the most relevant information

A high quality collection of information that's not searchable easily is like a dictionary where the words are randomly ordered instead of being alphabetically ordered.

While for smaller use-cases just dumping everything works fine (like uploading a group of PDFs to chatGPT or Claude), but at production scale, this could pretty much cross 1M tokens with just a part of the records information. We have seen some records to be as large as 50k tokens, although a typical record would be much much smaller.

To solve relevant search results for the agent, we designed a multi-layered search engine that the agent could reliably use, to explore and extract the EoI. Here were some practices that we followed to drive the max RoI here:

1. 1.
   Agentic Search: We tried doing RAG/embedding based search but soon realised that for engineering search, where keywords/jargons have no semantic meaning and a single character (e.g. us-east-1 vs us-east-2) could mean completely unrelated things, embeddings don't do much of a justice. In fact, keyword based search often made it easier for the agent to initiate discovery as atleast one EoI or scope is often exposed in the question/prompt/alert.
2. 2.
   Hierarchically organised information: Different EoIs have different weightage and priority. A relevant dashboard needs to be of much higher importance than a panel, and so on.
3. 3.
   Multi-attempts: The agent does multiple retrievals as it has more than a handful different ways to filter, sort and query the information, enabling nested discovery of information.
4. 4.
   Segregation of short term & long term records: Short term vs long term memory needs to be treated differently as they have very different objectives/intents, token consumption patterns, refresh rates.

Benefits of using DMR

1. 1. High Quality zero-setup investigations:

   Users are able to run high quality root cause investigations of production issues on day 0 without requiring any "training data" or "user inputs" or "custom configurations" or "professional services"

2. 2. Automated discovery:

   1. a. Issues and queries related to production systems can have varying degrees of vagueness as well as predictability. When trying to answer such a query, a right starting point becomes a key essential. If a user does not give a very specific tool + starting point combination (which most users very often do not), agent could end up getting lost in a maze full of unrelated information. This wouldn't be hallucination, it would just be non-useful information.
   2. b. The unified index helps agent get the most likely contexts in which the user or issue is occurring. From our experience, within 2-3 searches (100-200ms) the agent is often getting towards looking in the right direction.

3. 3. Smarter hypothesis building:

   1. a. When debugging, even the right direction could be the dead-end. It's colloquially called hypothesis validation / deduction by elimination (and technically called disjunctive syllogism).
   2. b. When the agent comes across a dead-end (or sees an hypothesis go flat), it needs to have parallel ideas in mind or else it's going to be a moot investigation.

4. 4. Deeper investigations:

   1. a. While beginning an investigation, one often does not know how many "Why"s are needed to identify what specific change caused the issue. As often discussed mathematically, debugging production issues is an NP-Hard problem (also why a blameless culture in engineering is essential).
   2. b. Iterating on the go often requires re-searching of context to get to the root cause accurately.

5. 5. Reduced agent investigation time/cost:

   1. a. In simpler situations, the number of attempts it takes for the agent to get to the right data becomes paramount from a commercial aspect.
   2. b. In production incidents, the difference between 2 minutes and 6 minutes is too big to be ignored.

Scope for improvement in DMR:

• •
  Pattern identification: Historical pattern of certain EoIs (e.g. alerts) can be beneficial in getting bird's eye view to the agent.
• •
  Hybrid search with embeddings: Some EoIs like customer support tickets or internal knowledge bases often perform well with a mix of fuzzy and semantic search. Introducing embeddings for certain record types would be beneficial and further improve contextual extraction.

Challenges while building & designing DMR:

1. 1. Garbage-in, garbage-out:

   To generate high quality DMR, using domain knowledge (in our case, production operations knowledge) and tool behaviours/patterns while defining EoIs and record structures was important. Doing this generically at scale could lead to relevance dilution, and eventually, sub-par DMR performance.

2. 2. Continuous updation and maintenance:

   1. a. While building the records could still happen well, SoRs are continuously evolving their data structures, EoIs and APIs. Keeping the data in sync across 80+ SoRs is essential to ensure high quality results.
   2. b. Record updations and additions on an ongoing basis require attention to detail to avoid chaotic data structures in the long term.

3. 3. Information hierarchy, scoring & search patterns:

   Nested record structuring with unequal weightage for record types, parameters and SoRs was essential to avoid noisy search results from DMR.

4. 4. Semantic search vs keyword searches:

   For parts of the index that might have more semantic relevance than keyword relevance, embeddings need to be appropriately designed for the engine to return good results.

5. 5. Maintaining tight coupling between tool/MCP parameters and records:

   The intent of the DMR is to drive accelerated + accurate tool calls. Having mismatch in the patterns of records vis-a-vis corresponding tools would lead to dilution of context for the agent and sometimes, confusion (e.g. dashboards as a record but not having a tool called fetch_dashboard_data_with_variables)

Appendix

Challenges in building a production operations agent beyond DMR:

1. 1. Context window management:

   A single agentic investigation could go for as long as 150 tool calls. While you could restrict the number of tool calls, exploratory investigations often lead to need for larger than usual context windows.

2. 2. Log & time series data analytics:

   At scale, teams could be managing TBs of hourly log volumes / Billions of timeseries rows. Getting the agent to smartly query and scan through the data efficiently can be challenging and require advanced probabilistic methodologies.

3. 3. Remote Integrations:

   SoRs used in production could often be isolated from each other and require specialised access to query, especially within a single investigation

4. 4. Enabling editing of memory by user

   Users often have insightful context that could drive the relevance of DMR even higher. Supporting users with easy and accurate ways of injecting/editing context is crucial, and often, non-negotiable.

5. 5. Unknown Unknowns

   In production systems, tribal knowledge and isolated information is often tough to escape. Command executions and config changes without easy to spot trails, selective accesses, multiple ways to do the same thing are some of the blockers in getting to an "ideal" state but there are deterministic ways to reduce each of the bottleneck.

6. 6. Tool / MCP Design for integrations

   While fundamentally, any given API spec can easily be transitioned to an MCP server, not all official MCP servers or APIs are equally good:

   • - They could be designed for editing/creating dashboards but your intent might be to query data.
   • - They could be designed for raw queries but you might want to query complex data (querying dashboards vs simple promql queries).
   • - There could be limitations at API layer itself for a SoR, restricting the types of actions / data fetches that can be done on it.