CodeQue

Get startedDocsRoadmapMissionPlayground

CodeQue basic documentation

Overview

CodeQue is structural code search tool for JavaScript and TypeScript projects.

CodeQue can be used to search for any code, from simple symbol search to complex multiline patterns.

It can be used as a linter by asserting search results to be found or not found in the codebase.

It's designed to be used on your device and help you during code development.

It's available as:

  • Handy CLI tool for code search
  • Visual Studio Code extension for code search and refactoring
  • ESLint plugin for creating custom rules in zero time

Looking for CLI docs? Check GitHub readme. This is general CodeQue documentation.

How it works

In syntax aware search modes the query has to be a valid TypeScript code.

Once you type your query, it's then parsed by babel to AST.

The code pattern represented by AST is then matched within all TypeScript or JavaScript source files (excluding git-ignored).

So it's simply(?) trying to find matching sub tree (query) in given file AST.

CodeQue also offers text search mode which is generating a regular expression based on the query. Text search mode is useful for initial discovery, but it is not that accurate as syntax aware search

Search modes

CodeQue contains a few search modes which can be used to achieve different search results.

include
search mode

Include is a default search mode. It's the less strict and allows to match code which contains more properties than query.

It also ignores order of the enumerable properties like keys in objects, properties in react components or statements in blocks of code.

It also matches code that has missing optional properties from query

Ok, I need an example...

A hypothetical query:

function myFunction(param) => {
if (param.length > 10) {
console.log(param)
}
console.log(param + param)
}

Can match this code:

function myFunction(param: string) => {
// statement 1
console.log(param + param)
// statement 2
if (param.length > 10) {
console.log(param)
}
// statement 3
return 'done'
}

Explanation:

  • console.log()
    call expression and
    if
    statement are enumerable elements of block statement, so they can be in different order,
  • function body contains return statement which is missing in query, but include search mode allows for additional nodes in the matched code,
  • parameters declaration is missing type annotation in query, but types annotations are optional in TypeScript, so the code is matched.

If you wonder which other elements of the language are optional, you can explore

import { NODE_FIELDS } from '@babel/types'
in @babel/types

include-wβ€Žith-order
search mode

It works in the same way as include mode, but checks for the order of the statements.

However forced order doesn't mean there can't be any additional property in between! Previous query will also match this code:

function myFunction(param) => {
if (param.length > 10) {
console.log(param)
}
param = param + param // I'm not in query, but that's not a problem
console.log(param + param) // I have to be after if statement to preserve order
}

exact
search mode

As the name suggest, exact mode matches only exactly the same piece of code.

text
search mode

What makes conventional search useless for me are:

  • lack of easy way of adding wildcards
  • matching whitespaces exactly as they are defined

CodeQue address these two problems by generating a regular expression out of query. RegExp is then used to match patterns across source files and - unlike in the syntax aware modes - treat code as string.

Such a RegExp matches any whitespace in place of a specific whitespace from a query and matches optional whitespaces between keywords and punctuators in the code.

What ?

It basically means that code can have optional whitespaces in places where they are not affecting the code logic.

So i.e you can look for a given variable declaration and find it on different levels of indentation. Unlike in normal search, where indentation matters.

If you type a query like

const a = {
key: value
}

it will be transformed into regexp (pseudo code):

const__ANY-WHITE-SPACE__a__ANY-WHITE-SPACE__=__ANY-WHITE-SPACE__{__ANY-WHITE-SPACE__ __ANY-WHITE-SPACE__key__ANY-WHITE-SPACE__:__ANY-WHITE-SPACE__value__ANY-WHITE-SPACE__ __ANY-WHITE-SPACE__}

So both

const a = { key: value }

and

const a = {
key: value
}

will be found!

But not the following, as it includes comma

const a = {
key: value,
}

Writing a Query

The simplest way to start writing query is to copy some existing piece of code.

Then replacing some identifiers or statements with wildcards, which makes "gaps" in the query, so more similar code patterns can be found.

const a = () => {
console.log("test")
}

Wildcards in syntax aware search modes

Wildcards is what makes CodeQue search so powerful!

They acts differently depending on whether they are used outside or inside strings.

Non string wildcards are:

  • a identifier wildcard represented by
    $$
  • a statement/expression wildcard represented by
    $$$

String wildcards are:

  • optional string content which can be used as
    "$$"
  • required string content (one or more chars) which is
    "$$$"

For number wildcards use

0x0

Example

Query

const $$ = "$$$"

will match any variable declaration initialized with non-empty string, so say:

  • const someVar = "value"
  • const someNewVar = "new value"
  • const x = "another value"

Example 2

Wildcards can be combined with other characters.

Query

const some$$ = "$$value"

will match the following code from previous example:

  • const someVar = "value"
  • const someNewVar = "new value"

but will not match the last one, due to mismatch in variable identifier.

You've probably noticed that each wildcard in code snippets is red and bold.

Example 3

Let's say you found out that developers in your project often assign a result of

notifyUsers
function to a variable. This function accepts callback as an argument, and always return undefined. So there is no point is assigning the returned value. As a good dev, you want to find and refactor all of these.

The query might be

const $$ = notifyUsers($$$)

Regardless of how the provided callback would look like, this query would match all of them:

  • callback might be a variable identifier
  • callback might be anonymous function

The statement wildcard

$$$
is matching any kind of AST node and it's children.

Example 4

Wildcards are also working for TypeScript types.

You can look for some duplicated types with such queries:

type $$ = "sm" | "md" | "lg"
interface $$ {
key: string
value: $$$
}

Some exceptions while writing specific queries

Due to the fact a query is a valid code, it might be confusing to prepare some of them.

Looking for objects

While looking for objects with some keys, you might try to write such a query:

{
key: "value"
}

Which is a valid code, but it's not an object. Parser interprets that as block of code!

Paste that in AST Explorer and check the output

To look for object, you have to add additional brackets:

({
key: "value",
})
Looking for strings

Remember

'use-strict'
? JavaScript has support for directives, which are strings in the top of the source file.

Most of us, including me, do not actually remember what it does πŸ˜€.

If you write query with just apostrophes and content, parser would interpret that as a directive instead of a string.

So the desired query has to have brackets - like in the object example:

("my string") // now it's looking for a string

Wildcards in text search mode

I've come up with these four patterns to act as "match anything" expressions:

  • $$
    - 0 or more chars in the same line,
  • $$m
    - 0 or more chars multiline,
  • $$$
    - 1 or more chars in the same line,
  • $$$m
    - 1 or more chars multiline.

Wildcards in text mode can match more than you would expect. They are useful for initial discovery, but once you need more accuracy, it's better to use syntax aware search modes.

Avoid Using very generic text queries, eg. just

$$m
, because it will result in huge performance degradation. It's also not very useful πŸ€·β€β™‚οΈ.

Case insensitivity

For each search you can enable case insensitivity.

It will affect matching both identifiers and contents inside the strings.

Caveats

CodeQue is in early stage and not all codebases might be supported.

I had to choose some settings for the parser to support most of the projects.

These language features (according to babel settings) are currently not supported:

  • flow & flowComments (flow would probably never be supported, I hate flow)
  • v8intrinsic
  • asyncDoExpressions
  • decimal
  • doExpressions
  • exportDefaultFrom
  • functionBind
  • importAssertions
  • moduleBlocks
  • partialApplication
  • pipelineOperator
  • recordAndTuple
  • throwExpressions

You probably don't use these features anyway.

In future it's planned to infer parser configuration out of project config files like

babel.config.js
or
tsconfig.json
!

Features coming soon

For the time being CodeQue offers quite comprehensive search API

In nearest future I plan to implement features like

Visual Studio Code extension

For me It's a must have. CLI interface is usable, but not ergonomic at all.

Editor integration is the first priority just after initial release

Perhaps some other code editors will be supported in the future

ESlint plugin

CodeQue ESlint plugin will be super powerful tool for enforcing code standards.

Implementing custom eslint plugins it super time consuming and usually it's not worth doing for temporal issues.

I want to simplify writing ESLint plugins to providing code pattern to match - a CodeQue query!

It requires some refactoring though, ESlint is using different parser and AST nodes than babel.

Replace / Refactor

Each search needs a replace functionality.

Replacing part of trees is quite tricky task.

I hope I will manage to implement that.

It would allow to make ad-hoc codemods without needs to manual implementations which usually takes more time than doing the replace manually.

Query extensions

I plan to add several query extensions like:

  • linking wildcards in query so you can indicate the same identifier
  • allowing for looking for statements in nested structures
  • changing search modes on different levels
  • adding some kind of boolean conditions like not, and, or

I'm not sure about exact API yet, but I have use cases for them.

Stay tuned

I'm super excited for all those features!

Check project roadmap for more context.

Hey πŸ‘‹

Something not clear?

Have a question or doubt?