Skip to main content
AI
  • Get started with AI

    • Red Hat AI

      Accelerate the development and deployment of enterprise AI solutions.

    • AI learning hub

      Explore learning materials and tools, organized by task.

    • AI interactive demos

      Click through scenarios with Red Hat AI, including training LLMs and more.

    • AI/ML learning paths

      Expand your Red Hat OpenShift AI knowledge using these learning resources.

    • AI quickstarts

      Focused AI use cases designed for fast deployment on Red Hat AI platforms.

    • No-cost AI training

      Foundational Red Hat AI training.

  • Featured resources

    • Red Hat OpenShift AI
    • Open source AI for developers
    • AI product application development
    • Open source-powered AI/ML for hybrid cloud
    • AI and Node.js cheat sheet

Red Hat AI Factory with NVIDIA

Red Hat AI Factory with NVIDIA is a co-engineered, enterprise-grade AI solution for building, deploying, and managing AI at scale across hybrid cloud environments.

Learn
  • Self-guided

    • Documentation

      Find answers, get step-by-step guidance, and learn how to use Red Hat products.

    • Learning paths

      Explore curated walkthroughs for common development tasks.

    • Guided learning

      Receive custom learning paths powered by our AI assistant.

    • See all learning
  • Hands-on

    • Developer Sandbox

      Spin up Red Hat's products and technologies without setup or configuration.

    • Interactive labs

      Learn by doing in these hands-on, browser-based experiences.

    • Interactive demos

      Click through product features in these guided tours.

  • Browse by topic

    • AI/ML
    • Automation
    • Java
    • Kubernetes
    • Linux
    • See all topics
  • Training & certifications

    • Courses and exams
    • Certifications
    • Skills assessments
    • Red Hat Academy
    • Learning subscription
    • Explore training
Build
  • Get started

    • Red Hat build of Podman Desktop

      A downloadable, local development hub to experiment with our products and builds.

    • Developer Sandbox

      Spin up Red Hat's products and technologies without setup or configuration.

  • Download products

    • Red Hat Enterprise Linux
    • Red Hat AI
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform
    • See all products
  • Featured

    • Red Hat build of OpenJDK
    • Red Hat JBoss Enterprise Application Platform
    • Red Hat OpenShift Dev Spaces
    • Red Hat Developer Toolset
  • References

    • E-books
    • Documentation
    • Cheat sheets
    • Architecture center
Community
  • Get involved

    • Events
    • Live AI events
    • Red Hat Summit
    • Red Hat Accelerators
    • Community discussions
  • Follow along

    • Articles & blogs
    • Developer newsletter
    • Videos
    • Github
  • Get help

    • Customer service
    • Customer support
    • Regional contacts
    • Find a partner

Join the Red Hat Developer program

Download Red Hat products and project builds, access support documentation, learning content, and more.

ConsoleDocsSupport Search
New For you

Recommended

We'll recommend resources you may like as you browse. Try these suggestions for now.

  • Product downloads
  • Cheat sheets
  • Red Hat Developer learning portal
  • Developer blog
  • E-books
  • Developer Sandbox
Log in

Get more with a Red Hat Developer account

  • Product downloads
  • Console access
  • Event registration
  • Training & trials
Explore more account benefitsLog in or register

A subscription may be required for some services.

Contact us

New tools for automating end-to-end tests for VS Code extensions

<p>&nbsp;</p> <quillbot-extension-portal></quillbot-extension-portal>

November 18, 2019
Jan Richter
Related topics:
IDEs
Related products:
Developer Toolset

    It is a common practice to test software from the user's perspective before releasing it. With this assumption, I have set out on a quest to find a VS Code extension with automated end-to-end tests. My quest ended in failure. Naturally, a lazy person like me then asked: "Why would nobody try to automate this?" It turns out that automating this was, in fact, quite difficult.

    My quest then became finding a solution that would enable developers to do just the thing. It is my pleasure to announce that no more hours need be wasted on this menial, manual activity. Enter the aptly named vscode-extension-tester: A framework that lets you create automated tests for your VS Code extensions and launch them with ease. All you need is an npm package.

    How it works

    VS Code is based on Electron, so it is a web application. Hence, the idea was to automate the tests using the Selenium WebDriver. For that purpose, we needed to:

      1. Download the appropriate version of ChromeDriver, which meant knowing the version of Chromium packaged inside the Electron browser our VS Code uses.
      2. Add ChromeDriver to our PATH.
      3. Choose the appropriate VS Code binary (which is different in every OS).
      4. Set up our VS Code to run the tests properly. We cannot, for instance, use the native title bar.
      5. Download another instance of VS Code just for testing. (We do not want to mess up the instance of VS Code we actually use.)
      6. Build our extension.
      7. Install the extension into the new instance.

      Finally, we were all set to begin writing our tests, but Figure 1 shows what we would have had to sift through in order to push a button and open a view:

      Mass of VS Code DOM contents we would have to sift through
      Figure 1: The mass of VS Code DOM contents we would have to sift through.

      That’s 15 layers of block elements just to find an icon representing a view container, which is quite a tall order to find a simple element. You can imagine what the rest of the DOM looks like.But enough scare tactics, we are here to make testing exciting. Almost as exciting as coding itself, because we are turning testing into coding. Let’s see how easy all of this becomes once we employ the vscode-extension-tester framework.

      Making it simple

      To demonstrate, we will take an extension and create end-to-end tests for it using our framework. As a first step, I like to use something simple, like the helloworld sample extension from Microsoft’s extension samples repo. This extension contributes a new command called Hello World that shows a notification saying Hello World! Now we need to write tests to verify that the command works properly.

      Hello world command enlarge to play
      Figure 2: The Hello World command: Enlarge to play.

      Getting the dependencies

      First, we need to get the necessary dependencies. To start, we need the Extension Tester itself, along with the test framework it integrates into: Mocha. We can get both from the npm registry:

      $ npm install --save-dev vscode-extension-tester mocha @types/mocha

      I will also use Chai for assertions. You can use whichever assertion library you like:

      $ npm install --save-dev chai @types/chai

      Setting up the test

      Now that we have our dependencies installed, we can start putting all the pieces together. Let us start by creating a test file. Our test files will rest in the src/ui-test folder, but you can use any path that is covered by your tsconfig, because we will write our tests in TypeScript just like the rest of the extension. Let’s go ahead and create the folder we chose and create a test file inside. I will call mine helloworld-test.ts. Our file structure should now look like Figure 3:

      Our beginning file structure
      Figure 3: Our beginning file structure.

      Next, we need a way to launch our tests. For that purpose, we create a new script in our package.json file. Let’s call our new script ui-test, and use the CLI that comes with the Extension Tester, called extest.  For this demo, we want to use the default configuration with the latest version of VS Code, the default settings, and the default storage location (which we will come back to momentarily).

      We also want to perform all of the setup and then run our tests within a single command. For that purpose, we can use the setup-and-run command that takes the path to our test files as an argument in the form of a glob. Note that we cannot use the original .ts files to launch the tests. Instead, we need to use the compiled .js files, which in this case are located in the out/ folder. The script will then look something like this:

      "ui-test": "extest setup-and-run out/ui-test/*.js"

      It is also important to compile our tests before attempting to run them, which we can do along with the rest of the code. For that purpose, this extension has a compile script we can use. The final script will then look like this:

      "ui-test": "npm run compile && extest setup-and-run out/ui-test/*.js"

      Setting up the build

      Now is the time to talk about the importance of the storage folder I mentioned earlier. This is where the framework stores everything it needs for the tests to run, including a fresh instance of VS Code, the ChromeDriver binary, and potentially screenshots from failed tests. It is imperative to exclude this folder from compilation and vsce packaging. Otherwise, you are bound to run into build errors. We also recommend adding the storage folder into your .gitignore file. By default, this folder is called test-resources, and is created in the root of your extension repository.

      First, let us exclude the folder from compilation. We need to open the tsconfig.json file and add the storage folder into the "exclude" array. This is what my tsconfig now looks like:

      {
      	"compilerOptions": {
      		"module": "commonjs",
      		"target": "es6",
      		"outDir": "out",
      		"sourceMap": true,
      		"strict": true,
      		"rootDir": "src"
      	},
      	"exclude": ["node_modules", ".vscode-test", "test-resources"]
      }

      With that code, our extension should not run into build errors with the folder present. Next, we need to make sure the folder is not included in the final .vsix file when we package the extension. For that purpose, we can utilize the .vscodeignore file. Let's go ahead and create one in the root of our repository if it doesn’t already exist. Then, put the folder into it just like we would with .gitignore, as shown in Figure 4:

      Excluding the test resources directory from packaging
      Figure 4: Excluding the test-resources directory from packaging.

      With these three simple steps completed, we are ready to dive into writing our tests. If you wish to get additional information about the test setup, check out the framework’s wiki.

      Writing the tests

      Remember that dreadful screenshot from the VS Code DOM? If you are familiar with WebDriver testing, you know how tedious it can become when the element structure is that complex.

      Introducing page objects

      Luckily, we do not need to bother ourselves with the DOM now. The Extension Tester framework brings us a comprehensive Page Object API.

      Each type of component in VS Code is represented by a particular typescript class and can be manipulated by a set of easy-to-understand methods. We recommend going through the page objects quick guide to get an understanding of what each object represents in the browser. Additionally, each object extends the vanilla WebDriver’s WebElement, so you can use plain WebDriver code to your heart’s desire.

      Back to the test at hand

      First, we need to create a test suite and a test case using the Mocha BDD format. The first step of our test case is to execute the command Hello World. For that purpose, we can use the Workbench class and its executeCommand method. Our test file now looks a bit like this:

      import { Workbench } from 'vscode-extension-tester';
       
      describe('Hello World Example UI Tests', () => {
          it('Command shows a notification with the correct text', async () => {
              const workbench = new Workbench();
              await workbench.executeCommand('Hello World');
          });
      });
      

      Simple, isn't it? Now, we need to assert that the correct notification has appeared. This command will take time to execute and display the result, so we cannot do this assertion straight away. Therefore, we use WebDriver to wait for the notification to appear. For that, we need a suitable wait condition.

      Our wait condition needs to view the currently displayed notifications and return the notification that matches our needs. In this case, the notification would be one that contains, say, the text Hello. If no such condition is found, do not return anything (return undefined). This way, the wait will terminate once the first truthy value is returned:

      async function notificationExists(text: string): Promise<Notification | undefined> {
          const notifications = await new Workbench().getNotifications();
          for (const notification of notifications) {
              const message = await notification.getMessage();
              if (message.indexOf(text) >= 0) {
                  return notification;
              }
          }
      }

      With this condition set up, we now start waiting. To do this, we need a reference to the underlying WebDriver instance. We can get that reference from the VSBrowser object, which is the entry point to the Extension Tester API. We will use the before function to initialize the WebDriver instance before the tests run by adding the following lines to the beginning of our suite:

          let driver: WebDriver;
      
          before(() => {
              driver = VSBrowser.instance.driver;
          });

      Initiating the wait is now as simple as this:

      const notification = await driver.wait(() => { return notificationExists('Hello'); }, 2000) as Notification;

      Note the cast at the end. Our wait condition may return undefined, and we need to work with a Notification object.

      The last step is to assert that our notification has the correct attributes by checking if the notification has the correct text, and is of an info type. Using Chai's expect to accomplish this task looks like this:

              expect(await notification.getMessage()).equals('Hello World!');
              expect(await notification.getType()).equals(NotificationType.Info);

      At this point, our first test is finished. The whole test file should look as follows:

      import { Workbench, Notification, WebDriver, VSBrowser, NotificationType } from 'vscode-extension-tester';
      import { expect } from 'chai';
      
      describe('Hello World Example UI Tests', () => {
          let driver: WebDriver;
      
          before(() => {
              driver = VSBrowser.instance.driver;
          });
      
          it('Command shows a notification with the correct text', async () => {
              const workbench = new Workbench();
              await workbench.executeCommand('Hello World');
              const notification = await driver.wait(() => { return notificationExists('Hello'); }, 2000) as Notification;
      
              expect(await notification.getMessage()).equals('Hello World!');
              expect(await notification.getType()).equals(NotificationType.Info);
          });
      });
      
      async function notificationExists(text: string): Promise<Notification | undefined> {
          const notifications = await new Workbench().getNotifications();
          for (const notification of notifications) {
              const message = await notification.getMessage();
              if (message.indexOf(text) >= 0) {
                  return notification;
              }
          }
      }

      Launching the tests

      All that is left now is to launch our tests. To do that, we can head to our favorite terminal and launch the script we created during the setup phase:

      $ npm run ui-test

      Now we can watch as the tooling runs the setup for us automatically:

      [video width="1920" height="1080" webm="https://developers.redhat.com/blog/wp-content/uploads/2019/11/extest_screencast.webm"][/video]

      Our test run was a success: We verified our extension's feature works. And best of all, we do not need to do all of this work manually anymore.

      Learning more

      If you wish to learn more about using the Extension Tester, be sure to visit the GitHub repository or the npm registry page. The wiki, in particular, might be of interest.

      To find detailed descriptions of all the steps we have gone through in this article, see the links below:

      • Test setup
      • Example test case
      • Page Object API quick guide

      Interested in the sample project we used in this article? Check out its code in the sample projects section, complete with commented tests.

      We also already have a few working test suites for real extensions (not just example ones). Feel free to take a look for inspiration:

      • The Apache Camel extension test suite.
      • Fuse tooling's UI test tooling, extending the Extension Tester.
      • Extension Tester's own test suite, which covers almost every available page object.

      If you would like to get involved, check out the Contributor's guide. We are always happy to see your feedback and suggestions, or indeed your code contributions.

    Last updated: February 13, 2024

    Recent Posts

    • A day in the life of an NVIDIA H100 on Red Hat OpenShift 5

    • Migrate BuildConfigs to builds for Red Hat OpenShift with one plug-in install

    • Run Red Hat Enterprise Linux CoreOS on NVIDIA BlueField DPUs

    • Agent harness engineering: Separating model capabilities from enforcement

    • What's new for developers in Red Hat Enterprise Linux 10.3 and 9.9

    Red Hat Developers logo

    Platforms

    • Red Hat AI
    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform
    • See all products

    Build

    • Developer Sandbox
    • Developer tools
    • Interactive tutorials
    • API catalog

    Quicklinks

    • Learning resources
    • E-books
    • Cheat sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site status dashboard
    • Report a security problem

    Red Hat Developer

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit
    © 2026 Red Hat

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility
    Ask AI