Extracting information from Python source code

What library symbols does a Python source code file use? And what symbols does it provide to its users? A simple tool called invectio can provide this information based on static source code analysis. Invectio means “import” in Latin. As the name suggests, this small tool can extract information about imports as well as information about what users can import from Python modules.

Making Python symbols visible

The Python standard library contains a module called Abstract Syntax Trees (ast), with routines to parse Python source code and access it within a program. This capability, along with other parts of the standard library, allows you to examine Python source code and extract information about symbols used or provided. Because this task is widely applicable in many situations, the Project Thoth team has extracted the logic into a simple application that displays the symbols used or exported by source code.

invectio whatuses

The first command available in invectio extracts information about symbols used in the source files. Let's consider a simple Flask application:

import flask

app = flask.Flask(__name__)

@app.route("/")
def hello_world():
    return flask.jsonify({})

Running invectio whatuses extracts information about symbols used from other modules, the use of built-in symbols, and Python standard library symbols. The output is in JSON format:

$ invectio whatuses test.py
{
  "report": {
    "test.py": {
      "__builtins__": [
        "__builtins__.__name__"
      ],
      "flask": [
        "flask.Flask",
        "flask.jsonify"
      ]
    }
  },
  "version": "0.2.0"
}

The tool can also take a directory as an argument and display the symbols from all the Python source files in that directory.

invectio whatprovides

Another invectio subcommand extracts information about functions, classes, and constants available in the source code. Consider the following Python source code:

CONST = 42

def is_palindrome(s: str) -> bool:
    return s == s[::-1]  # TODO: optimize

Running invectio whatprovides gives insights about symbols provided by the module:

$ invectio whatprovides module.py
{
  "report": {
    "module.py": [
      "module.CONST",
      "module.is_palindrome"
    ]
  },
  "version": "0.2.0"
}

Like the whatuses subcommand, the whatprovides subcommand can work on directories to inspect all the Python source code present in a directory.

How to install and use invectio

Invectio is available as a Python package on PyPI. To install it, just enter the following:

$ pip install invectio

The project source files are hosted on GitHub in Project Thoth's invectio repository.

About Project Thoth

This tool was developed by Project Thoth, part of the Artificial Intelligence Center of Excellence group (AICoE). As part of Project Thoth, we are accumulating knowledge to help Python developers create healthy applications. The introduced tool, invectio, is integrated into Thoth's client applications that aggregate information for Thoth's resolver to give better guidance to developers on dependencies they use. If you would like to follow updates in Project Thoth, feel free to subscribe to our YouTube channel or follow us on the @ThothStation Twitter handle.