Several months ago we had a talk within the team about ways we could improve our code reviews, and naturally the possibility of using AI to help with them came up. I was a bit skeptical, because doing a code review requires detailed knowledge of the code base as well as business decisions, which means that the context the AI must have is huge. However, it seemed like something we should investigate, so I decided to give it a go and create a Laravel package that performs code reviews with the help of AI. And thus, Socraites was born.

Initial plan and implementation

The basic workflow of Socraites is very simple:

1. take the output of git diff --staged,
2. build a context from it,
3. send it to OpenAI's API for a code review,
4. and display the results.

Steps 1, 3, and 4 are simple and straightforward. They are easily implemented with plain PHP, and I did not have any problems with them.

Step 2, however, is the tricky one. It requires building a context around the changed code that will tell the whole story needed for the code to be understood, without exceeding the API limit. The problem is that one changed file can use code from many different files, and it is not possible to simply take all files and send them to the API.

For the initial implementation I went with a mathematical approach to solving this problem. First I took all files from git diff --staged and parsed them using the nikic/php-parser package. After gathering metadata of changed files, I would calculate the importance of each class that is used in those files based on how many times it is used and inheritance and import detection. That led me to the following method:

private function calculateRelevanceScore(string $filePath, string $sourceFile, array $parseResult): int
{
    $score = 0;

    if (in_array($filePath, $parseResult['imports'])) {
        $score += 5;
    }

    if (in_array($filePath, $parseResult['extends'])) {
        $score += 10;
    }

    if (in_array($filePath, array_keys($parseResult['usageCounts'][$sourceFile]))) {
        // Logarithmic scale to avoid classes with many references dominating completely
        $usageCount = $parseResult['usageCounts'][$sourceFile][$filePath];
        if ($usageCount > 0) {
            $score += 5 + min(20, (int)(5 * log($usageCount + 1, 2)));
        }
    }

    return $score;
}

This formula was used to calculate the relevance score of every changed file, and in the $context array I would include all the most important files until the limit was reached.

This approach had several obvious limitations. It was possible for a controller class in Laravel to use one very important function from a model class, and also use Exception class in three different places. In that case, this formula would decide that the Exception class is more important and would add it to the context first. Also, if we are using only one method from a model class, there is a good chance that we do not need the whole model as context, but we should probably include some info about it—like names of the columns, Eloquent relations, and similar. With this approach, it was really hard to cherry-pick which part of the class we need and which part is not necessary.

Improvements in building context

Based on advice from my colleagues Matija Bojanic and Jaksa Vlahovic, I decided to abandon this approach and let the AI build the context for itself. So the new workflow looks like this:

1. take the output from git diff --staged,
2. send it to the AI,
3. AI asks about the code it wants to know more about,
4. send that code to the AI,
5. repeat until AI knows enough to perform the code review.

Letting AI decide what context is missing is a good way to save tokens. Only relevant parts of the code are sent to it, and business decisions can be added as part of the prompt, so that the business side is covered as well.

I have three tools that are used when interacting with AI, two of which are used for AI requesting the code, and the third one is for making sure that AI validates that the deleted code is properly reviewed and ensures it can be safely deleted. AI can ask for code in two ways: by asking for a specific code from a specific class (e.g. isActive() from the User model) or using semantic search (e.g. "I need a method that explains the relation between user and posts"). This means that we need a fast and reliable way to query our code and send the necessary info. I solve this by vectorizing the codebase.

As part of the setup, the user must run the following command:

php artisan socraites:vectorize

This command takes the codebase and vectorizes each method, and saves it in the code_chunks table, together with metadata like class name, method name, etc. This way, when AI asks for a "method that explains the relation between user and post," I can perform a vector search, find it, and send it as context.

It is important to update the code_chunks table regularly, which is handled by a git hook on the main branch. When the local main/master branch is updated, the hook automatically vectorizes new code and deletes the vectors of the deleted code.

This method proved to be more reliable and requires fewer tokens.

Using Socraites

Using Socraites is straightforward. After staging the changes they want reviewed, the user should run the command:

php artisan socraites:code-review

This will return a nicely designed review from the AI, together with a Socrates quote and a suggested commit message. If the user has any questions about the review, they can continue the conversation and ask for clarification. An example of the output:

Analyzing your code...

"Wisdom begins in wonder."
- Socrates

  
  SOCRAITES CODE REVIEW


  ✅ Overall Summary:
  ────────────────────────────────────────────────────────────
     The changes in 'src/Entity/User.php' aim to define a new User class that implements the UserInterface from Symfony's Security component. However, the methods required by the interface are declared but not implemented, which will lead to issues during runtime if the class is used.
  ────────────────────────────────────────────────────────────


  📁 Files from context:
  ────────────────────────────────────────────────────────────
     src/Entity/User.php
  ────────────────────────────────────────────────────────────


  🔍 Reviewing: src/Entity/User.php
  ────────────────────────────────────────────────────────────
  ✅ Summary:
     Implementation of the User class which implements the UserInterface from Symfony Security.
  ❌ Major Issues:
     Unimplemented methods required by the UserInterface.
  ⚠️  Minor Issues:
     Missing newline at end of file.
  💡 Suggestions:
     Implement the methods 'getRoles', 'eraseCredentials', and 'getUserIdentifier' to fulfill the contract of the UserInterface.
     Add a newline at the end of the file to follow good coding standards.
  ────────────────────────────────────────────────────────────

  💬 Suggested Commit Message:
  ────────────────────────────────────────────────────────────
     Add User class skeleton implementing UserInterface
  ────────────────────────────────────────────────────────────
   Do you have a question about the code review? [no]:
   > 

Future plans

Currently, Socraites works only with the OpenAI API. That must be expanded to allow users to use any LLM they want.

The code_chunks table stores only code from the main/master branch, which is not ideal when working on bigger features that introduce a lot of new code. All non-main code should be stored somewhere and queried separately.

Conclusion

Socraites is open-source and can be found on my github page. All ideas, reviews and comments are welcome.

So, after all the work, the question from the beginning — can AI be helpful with code reviews — can be answered. In my opinion, we are still not there yet. Socraites was an excellent exercise and I have learned a lot about how AI works, vector searches, git hooks, etc.

With more work it will get even better, but for the time being, I think that AI can be used only as a way to catch any obvious oversights, like typos, N+1 queries, and similar. This applies when using Claude Code and all other LLM tools as well.