Packages
fnord
0.4.26
0.9.40
0.9.39
0.9.38
0.9.37
0.9.36
0.9.35
0.9.34
0.9.33
0.9.32
0.9.31
0.9.30
0.9.29
0.9.28
0.9.27
0.9.26
0.9.25
0.9.24
0.9.23
0.9.22
0.9.21
0.9.20
0.9.19
0.9.18
0.9.17
0.9.16
0.9.15
0.9.14
0.9.13
0.9.12
0.9.11
0.9.10
0.9.9
0.9.8
0.9.7
0.9.6
0.9.5
0.9.4
0.9.3
0.9.2
0.9.1
0.9.0
0.8.99
0.8.98
0.8.97
0.8.96
0.8.95
0.8.94
0.8.93
0.8.92
0.8.91
0.8.90
0.8.89
0.8.88
0.8.87
0.8.86
0.8.85
0.8.84
0.8.83
0.8.82
0.8.81
0.8.80
0.8.79
0.8.78
0.8.77
0.8.76
0.8.75
0.8.74
0.8.73
0.8.72
0.8.71
0.8.70
0.8.69
0.8.68
0.8.67
0.8.66
0.8.65
0.8.64
0.8.63
0.8.62
0.8.61
0.8.60
0.8.59
0.8.58
0.8.57
0.8.56
0.8.55
0.8.54
0.8.53
0.8.52
0.8.51
0.8.50
0.8.49
0.8.48
0.8.47
0.8.46
0.8.45
0.8.44
0.8.43
0.8.42
0.8.41
0.8.40
0.8.39
0.8.38
0.8.37
0.8.36
0.8.35
0.8.34
0.8.33
0.8.32
0.8.31
0.8.30
0.8.29
0.8.27
0.8.26
0.8.25
0.8.24
0.8.23
0.8.22
0.8.21
0.8.20
0.8.19
0.8.18
0.8.17
0.8.16
0.8.15
0.8.14
0.8.13
0.8.12
0.8.11
0.8.1
0.8.0
0.7.24
0.7.23
0.7.22
0.7.21
0.7.20
0.7.19
0.7.18
0.7.17
0.7.16
0.7.15
0.7.14
0.7.13
0.7.12
0.7.11
0.7.10
0.7.9
0.7.8
0.7.7
0.7.6
0.7.5
0.7.3
0.7.2
0.7.1
0.7.0
0.6.9
0.6.8
0.6.7
0.6.6
0.6.5
0.6.4
0.6.3
0.6.1
0.6.0
0.5.9
0.5.8
0.5.7
0.5.6
0.5.5
0.5.4
0.5.3
0.5.2
0.5.1
0.5.0
0.4.44
0.4.43
0.4.42
0.4.41
0.4.40
0.4.39
0.4.38
0.4.37
0.4.36
0.4.35
0.4.34
0.4.33
0.4.32
0.4.30
0.4.29
0.4.28
0.4.27
0.4.26
0.4.25
0.4.24
0.4.23
0.4.22
0.4.21
0.4.20
0.4.19
0.4.18
0.4.17
0.4.16
0.4.15
0.4.14
0.4.13
0.4.12
0.4.11
0.4.10
0.4.9
0.4.8
0.4.7
0.4.6
0.4.5
0.4.4
0.4.3
0.4.2
0.4.1
0.4.0
0.3.0
0.2.0
0.1.0
AI code archaeology
Current section
Files
Jump to
Current section
Files
lib/ai/agent/file_summary.ex
defmodule AI.Agent.FileSummary do
@moduledoc """
This module provides an agent that summarizes files' contents in order to
generate embeddings for the database and summaries for the user.
"""
defstruct [
:ai,
:file,
:splitter,
:summary
]
@model "gpt-4o-mini"
# It's actually 128k for this model, but this gives us a little bit of
# wiggle room in case the tokenizer we are using falls behind.
@max_tokens 100_000
@chunk_prompt """
You are an AI agent that summarizes the content of a file, whether it is code or documentation, like an intelligent `ctags`.
You will process the file in chunks, each paired with an "accumulator" string to update. Each input will contain:
```
# Accumulated Summary
$accumulated_summary
-----
$current_file_chunk
```
Your task is to build on the existing summary with each chunk, adhering to these guidelines:
# General Guidelines:
- Add Relevant Content: Update the accumulated summary by adding notes relevant to the current chunk.
- Structure and Continuity: Preserve the structure of the summary, maintaining a coherent flow for cross-referencing across chunks.
- Mark Incomplete Sections: If a section seems incomplete, label it `<partial>` to complete later.
- Avoid Redundancy: Avoid duplicating details already present unless it clarifies key linkages or structure.
Based on the file type, update the accumulated summary as follows:
## For Code Files:
- Synopsis: Briefly summarize the purpose of the code file.
- Languages Present: List programming languages used.
- Business Logic and Behaviors: Summarize main functions, classes, and business logic. Highlight any patterns or distinctive behaviors.
- Note any oddball configuration or unexpected aspects of the code or how it is organized.
- List of Symbols: List important classes, functions, variables, and constants, with brief descriptions.
- Call Graph: Use the format `FunctionA -> ModuleX:MethodY` to document inter-module calls, highlighting links relevant for understanding dependencies and functionality. Note that there should be an entry for every call a function makes to another function.
## For Documentation Files (e.g., README, Wiki Pages, General Documentation):
- Synopsis: Summarize the document's primary purpose.
- Topics and Sections: List main topics or sections in the document.
- Definitions and Key Terms: Note specialized terms and definitions.
- Links and References: Include any links or references, especially to related files, modules, or external resources.
- Key Points and Highlights: Summarize main points, noting insights that would aid semantic search.
Only use information from the file itself to ensure accurate summaries without false positives from external sources.
# Response Format
Respond ONLY with the updated `Accumulated Summary` section in markdown format, adding content from the current chunk as needed.
"""
@final_prompt """
You have processed a file in chunks, using an accumulated summary to track relevant details. Now, review and consolidate this accumulated summary into a cohesive, final summary of the file's contents.
Your input will contain:
```
# Accumulated Summary
$accumulated_summary
```
Please follow these steps as you create the final summary:
# General Guidelines:
- Reorganize for Clarity: Organize content logically, ensuring all details flow smoothly and follow the structure below.
- Eliminate Redundancy: Avoid duplicate information or unnecessary repetition from the accumulated notes.
- Ensure Completeness: Ensure all sections and details are covered, completing any previously marked `<partial>` sections.
- Optimize for Semantic Matching: Select details that best support semantic search, especially key linkages between code or document topics.
# Final Summary Structure
Based on the file type (Code or Documentation), structure your final summary as follows:
## For Code Files:
- Synopsis: Briefly summarize the purpose of the code file.
- Languages Present: List programming languages used.
- Business Logic and Behaviors: Summarize main functions, classes, and business logic. Highlight any patterns or distinctive behaviors.
- Note any oddball configuration or unexpected aspects of the code or how it is organized.
- List of Symbols: List important classes, functions, variables, and constants, with brief descriptions.
- Call Graph: Use the format `FunctionA -> ModuleX:MethodY` to document inter-module calls, highlighting links relevant for understanding dependencies and functionality. Note that there should be an entry for every call a function makes to another function.
Do your best to adjust each section to avoid redundancy in case the accumulated summary contains duplicate information.
## For Documentation Files (e.g., README, Wiki Pages, General Documentation):
- Synopsis: Summarize the document's primary purpose.
- Topics and Sections: List main topics or sections in the document.
- Definitions and Key Terms: Note specialized terms and definitions.
- Links and References: Include any links or references, especially to related files, modules, or external resources.
- Key Points and Highlights: Summarize main points, noting insights that would aid semantic search.
Do your best to adjust each section to avoid redundancy in case the accumulated summary contains duplicate information.
# Response Format
Respond ONLY with the final, organized summary in markdown format, excluding the "Accumulated Summary" header, following the structure for either code or documentation files. Ensure the summary is clear, concise, and includes only the relevant details from the file itself.
"""
def get_summary(ai, file, file_content) do
%__MODULE__{
ai: ai,
file: file,
splitter: AI.TokenSplitter.new(file_content, @max_tokens),
summary: ""
}
|> reduce()
end
defp reduce(%{splitter: %{done: true}} = agent) do
finish(agent)
end
defp reduce(%{splitter: %{done: false}} = agent) do
with {:ok, agent} <- process_chunk(agent) do
reduce(agent)
end
end
defp finish(agent) do
AI.get_completion(agent.ai,
model: @model,
system_prompt: @final_prompt,
user_prompt: get_prompt(agent)
)
|> case do
{:ok, %{"message" => %{"content" => summary}}} -> {:ok, summary}
{:error, reason} -> {:error, reason}
end
end
defp process_chunk(agent) do
prompt = get_prompt(agent)
{chunk, splitter} = AI.TokenSplitter.next_chunk(agent.splitter, prompt)
agent = %{agent | splitter: splitter}
message = prompt <> chunk
AI.get_completion(agent.ai,
model: @model,
system_prompt: @chunk_prompt,
user_prompt: message
)
|> case do
{:ok, %{"message" => %{"content" => summary}}} ->
{:ok, %__MODULE__{agent | splitter: splitter, summary: summary}}
{:error, reason} ->
{:error, inspect(reason)}
end
end
defp get_prompt(agent) do
"""
# Accumulated Summary
#{agent.summary}
-----
"""
end
end