Skip to main content

Queries

The main way to use crul is through authoring and running queries. These queries are composed of one or more stages, "piped" together using the double pipe || syntax.

Let's look at an example to better understand how queries are constructed.

Example query​

api get https://... --useragent "custom" /* stage 1 */
|| find "keyword" /* stage 2

Stage 1​

In the above query, stage 1 is described by a command, and any arguments and flags that determine how to execute the stage. In this example, the command is api, the first argument is get (a REST method), the second argument is https://... (the endpoint to request), and the --useragent is a flag that in this case will set the useragent of the API request.

Stage 2​

Once stage 1 has completed, the results are "piped" (||) to stage 2. Stage 2 is described by a command (find), and a single argument ("keyword"). This stage will search for matches to the provided key word.

Additional stages​

We can add additional stages to this query by adding || [command] [args] [flags]. See the commands reference for a full list of available commands. For example, from here, we could sort the results, find the min or max, add a timestamp, freeze our results to write them to a 3rd party destination, and much more!

Expanding stages​

Many commands are simple "reducing" type commands, for example, sort or find will take the data that is piped to it from a previous stage and transform it in some way. However other commands can be used in expanding stages, which take the data piped to it from a previous stage and operate on each row in an expanding fashion. Let's look at a more concrete example.

api get https://... /* stage 1 */
|| api get $url$ /* stage 2 */

Stage 1​

In the above query, stage 1 will make a GET request to the provided address (https://).

Let's say the result set after stage 1 completes looks like this:

{
  "account": "0987654321",
  "url": "https://.../asdf-1234-qwer"
},
{
  "account": "1234567890",
  "url": "https://.../zxcv-5678-tyui"
}

Stage 2​

Stage 2 is an example of an expanding stage. The $url$ represents a token that will be replaced based off of the matching key value(s) from the results "piped" in from stage 1. The query processor will recognize that stage 2 includes a token, and will run through each result in the "piped" in results to replace $url$ with the url value from the previous results.

Stage 2 will go from:

api get $url$

to

Stage 2 Command 1:

api get https://.../asdf-1234-qwer /* see url value in the first result from stage 1 */

Stage 2 Command 2:

api get https://.../zxcv-5678-tyui /* see url value in the first result from stage 1 */

These commands will run in parallel, and when both complete, the results of each command will be merged and stage 2 will be considered complete.