Local AI Coding Assistant with Qwen 2.5

Qwen is a large language model (LLM) developed by the Alibaba group. This model is quite well-known for its capabilities that compete with GPT-4 (OpenAI), Llama-3 (Meta), and Mistral (Mistral AI). Qwen is released under the Apache 2.0 license, so we can use it for both non-commercial and commercial purposes, such as coding local assistants for both personal and professional use. The latest version of Qwen is Qwen-2.5, and for coding, we will use Qwen-2.5-Coder.

There are seven variants of Qwen-2.5 available, ranging from 0.5B, or 0.5 billion parameters, to 72B. Meanwhile, at the time of this writing, there are only six variants of Qwen-2.5-Coder, with the 32B variant being the flagship. The size of these parameters is related to the accuracy of the AI ​​model in handling complex coding tasks.

The easiest way to use Qwen-2.5-Coder offline is to run it with Ollama. Ollama is a desktop application that makes it easy for users to run LLM on PCs or laptops running Windows, Mac, or Linux. This article will outline how to run Qwen-2.5-Coder with Ollama and connect it to the Continue AI assistant extension in Visual Studio Code.

First, we need to download and install the Ollama app from the ollama.com homepage . Once the Ollama app is installed, download the Qwen-2.5-Coder model by typing the following command:

ollama pull qwen2.5-coder:7b

I used Qwen-2.5-Coder:7B parameters because models with 3B parameters or less cannot solve the coding task well, so a model with higher capabilities is needed that can run on a mid-range GPU with 8GB of VRAM, such as an RTX 3070 or equivalent.

Once Ollama has finished downloading the desired model, to connect Qwen-2.5-Coder with the Continue extension in the Visual Studio Code editor, we need a local AI server. There are two ways to run an AI server with Ollama: by running the desktop application or by running the following command::

ollama serve 

Next, we'll add the Continue extension from Continue.Dev to the Visual Studio Code editor. (This extension is also available for JetBrains IDEs like Intellij IDEA, but I haven't tried it yet.) Once the Continue extension is installed, we'll need to edit the extension's config.json file. To do this, use the keyboard shortcut Ctrl+Shift+P and type "continue: config," then select "Continue: Open Continue Config" from the resulting options.

Fill in or change the config for "models" and "tabAutoCompleteModel" according to the instructions from the Continue extension here, as shown by this sample below:

{
  "models": [
    {
      "model": "qwen2.5-coder:7b",
      "provider": "ollama",
      "title": "Qwen 2.5 Coder 7B",
      "systemMessage": "You are an expert software developer. You give helpful and concise responses."
    }
  ],
  "tabAutocompleteModel": {
    "title": "Qwen 2.5 Coder 7B",
    "provider": "ollama",
    "model": "qwen2.5-coder:7b"
  },
  "contextProviders": [
    {

Don't forget to save this file so the Continue extension can connect to Qwen-2.5-Coder in Ollama. If you still don't see any changes, try opening a new VS Code window or restarting both Ollama and VS Code. Sometimes, restarting your PC or laptop is necessary so Ollama can use all available graphics memory to run Qwen-2.5-Coder smoothly.

Happy coding~!