TRY ME

Try Valo for free

We want to show you something amazing.

We'll send you a link to download a fully functional Valo copy to play with.



Great! Check your email and enjoy Valo



Apologies, we seem to be having a problem processing your input, please try again

Execution session

An execution session is a logical grouping of queries which shares variables, functions, and is executed within an optional domain. The session is created and modified via the execution api.

Domains

A domain can be applied to a session to filter the query result. The queries will only include data from the contributors which are included in the domain.

{
  "domain" : "/domain/demo/support/my_servers",
  "queries": [
    {
        "id": "q1",
        "body": "from /streams/demo/infrastructure/log1 where host == `ABC`"
    }
  ]
}

If the domain is not valid an INVALID_DOMAIN issue will be returned.

Dependent queries

Queries within an execution session can use the output of other queries using the Into keyword.

In the following example, queries q1 and q2 feeds a logical stream x which is then referenced by queries q3 and q4

{
  "queries": [
    {
      "id": "q1",
      "body": "from /streams/demo/infrastructure/log1 where host == `ABC` into x"
    },
    {
      "id": "q2",
      "body": "from /streams/demo/infrastructure/log2 where host == `ABC` into x"
    },
    {
      "id": "q3",
      "body": "from x select host, msg"
    },
    {
      "id": "q4",
      "body": "from x where contains(msg, `error`) select host, msg"
    }
  ]
}

The streams feeding the same logical stream within an execution session must have the same schema. If not, a LOGICAL_STREAM_CONFLICT issue will be returned. If the logical stream hasn’t been defined in the session issue INVALID_STREAM_REFERENCE is returned.

Variables

A session supports the notion of user-defined variables which can be referenced within a query. A variable is referenced using the @ sign followed by the variable name as shown by the following example.

{
  "queries": [
    {
      "id": "q1",
      "body": "from /streams/demo/infrastructure/log1 where contains(msg, @tradeId) into x",
      "variables": [
         { "id": "tradeId", "value": "ABC123", "type": "string" }
      ]
    }
  ]
}

Variables can either be defined as local or global with local variables overriding global variables. In the following example, the tradeId variable is defined as a global variable.

{
  "queries": [
    {
      "id": "q1",
      "body": "from /streams/demo/infrastructure/log1 where contains(msg, @tradeId) into x"
    }
  ],
  "variables": [
      { "id": "tradeId", "value": "ABC123", "type": "string" }
  ]
}

If a query references a variable which is not defined in the session MISSING_VARIABLE is returned. If the variable type has an incompatible schema type then INVALID_VARIABLE_TYPE is returned.

The variable type can be any of the simple types defined in Data Types. Complex types like records and arrays are not supported.

Query output

The result of a query can be sent to an output channel for consumption by the client, or it can be declared as a logical stream, or sent to another stream in Valo. The user defined this behaviour via the query language by using the Into keyword.

Output channel

An output channel is an endpoint the client can connect to in order to retrieve the data from a query.

{
    "type": "OUTPUT_CHANNEL",
    "id"  : "ac3c7fad-a858-48fc-b431-27775397700f",
    "schema": {
        "typeDef": {
            "type": "record",
                "properties": {
                    "host": { "type": "string" },
                    "user": { "type": "double" }
                }
            }
    },
    "alias" : "result",
    "outputUri": "/output/demo/6daccb6739b02bacda2d4c37428263d3/5264726f-e17b-4bb4-bc44-f45be8fdb44e",
    "outputType": "BOUNDED",
    "isHistorical": true
}
field description
type OUTPUT_CHANNEL
id the id of the template node which generates this output.
schema the output schema.
alias an optional alias given to this output channel.
outputUri the uri the client should connect to when receiving the data. Might be NOT_AVAILABLE if the node has not been assigned a physical node.
outputType BOUNDED or UNBOUNDED query output.
isHistorical true if the output was generated by a historical query.

The result of a query is sent to the client using the following message types;

Type Description
sos Start of stream.
increment Batched results from the query.
eos End of stream.

Sos

The first message to be sent is an sos message.

Increment

Increment messages are sent as the query returns results. A message is sent when a max batch size is reached (default batch size is 100), or on a timeout (default timeout is 1 second).

Eos

The last message to be sent is an eos message.

For a historical query, the eos message is sent as soon the repository queries have completed. For a real-time query, the eos message is sent when the query is stopped by the user.

Example

{
  "type": "increment",
  "items": [
    {
      "action": "add",
      "key": "a",
      "data": {
        "host": "192.0.10.10",
        "user": 15
      }
    },
    {
      "action": "update",
      "key": "a",
      "data": {
        "host": "192.0.10.11",
        "user": 15
      }
    },
    {
      "action": "add",
      "key": "b",
      "data": {
        "host": "192.0.10.17",
        "user": 15
      }
    },
    {
      "action": "remove",
      "key": "a",
      "data": {
        "host": "192.0.10.11",
        "user": 15
      }
    }
  ]
}

{
  "type": "eos"
}

Logical stream

A logical query output which can be used as an input to other queries.

{
    "type": "LOGICAL_STREAM",
    "id"  : "ac3c7fad-a858-48fc-b431-27775397700f",
    "schema": {
        "topDef": {
            "type": "record",
            "properties": {
                "host": { "type": "string" },
                "user": { "type": "double" }
            }
        }
    },
    "name" : "result"
}
field description
type LOGICAL_STREAM
id the id of the template node which generates this output.
schema the output schema.
name the name of the logical stream.

Physical stream

A query output which sends the data to an existing Valo stream.

{
    "type": "PHYSICAL_STREAM",
    "id"  : "ac3c7fad-a858-48fc-b431-27775397700f",
    "schema": {
        "topDef": {
            "type": "record",
            "properties": {
                "host": { "type": "string" },
                "user": { "type": "double" }
            }
        }
    },
    "uri" : "/streams/demo/derived/cpu_and_mem"
}
field description
type PHYSICAL_STREAM
id the id of the template node which generates this output.
schema the output schema.
uri the physical stream uri.

Error handing

A query within a session can only be executed if it has no issues. The issues have to be fixed and the query re-executed.

ALIAS_NOT_UNIQUE

The same alias is defined multiple times.

{ "type": "ALIAS_NOT_UNIQUE", "alias": "ts" }

In the following query the field name ts is used more than once

from /streams/demo/infra/cpu
where user as ts, kernel as ts

CIRCULAR_DEPENDENCY

A circular dependency was discovered.

{
  "type": "CIRCULAR_DEPENDENCY",
  "uri": "x"
}

The following is a circular dependency

from x
into x

ILLEGAL_EXPRESSION

An illegal query expression.

{
  "type": "ILLEGAL_EXPRESSION",
  "msg": "Expression is not valid"
}

In the following query the filter expression is invalid as it does not evaluate to a boolean expression.

from /streams/demo/infra/cpu
where user + 2

INVALID_DOMAIN

Unable to resolve the domain uri.

{ "type": "INVALID_DOMAIN", "uri": "/domains/demo/does_not_exist" }

INVALID_EXPRESSION_TYPE

Generated if an expression in a query evaluates to the wrong type. For example, a filter predicate which does not evaluate to a boolean expression.

{ "type": "INVALID_EXPRESSION_TYPE", "expectedType": "boolean" }

INVALID_FIELD_REFERENCE

The query references fields which cannot be resolved using the source schema.

{ "type": "INVALID_FIELD_REFERENCE", "name": "os.name" }

INVALID_FUNCTION

The query references a function which cannot be resolved.

{ "type": "INVALID_FUNCTION", "name": "toLower" }

INVALID_FUNCTION_INIT_ARGS

The init arguments for a function is invalid.

{
  "type": "INVALID_FUNCTION_INIT_ARGS",
  "msg": "Argument 'b' for function 'OAWithInitParams' is defined multiple times.",
  "functionName": "OAWithInitParams",
  "args": [
    {
      "name": "b",
      "type": "int",
      "isArray": false
    },
    {
      "name": "i",
      "type": "int",
      "isArray": false,
      "default": "10",
      "description": "A description"
    },
    {
      "name": "k",
      "type": "double",
      "isArray": true,
      "default": "[10.0,15.0]"
    }
  ]
}

The functionName field is the function referenced by the query and the args are the actual init arguments required by that function.

INVALID_FUNCTION_SIGNATURE

The query references a function which exists but the function signature is invalid.

{ "type": "INVALID_FUNCTION_SIGNATURE", "name": "funcX", "arguments": ["int","double"], "candidates": [
    { "arguments": ["int","int"] },
    { "arguments": ["double","double"] }
  ]
}

INVALID_LITERAL_CONVERSION

Unable to convert a literal expression.

{"type": "INVALID_LITERAL_CONVERSION", "from": "\"A\"", "to": "double" }

INVALID_MEMBER_REFERENCE

An expression refers to a member which is not in scope.

{"type": "INVALID_MEMBER_REFERENCE", "member":"left"}

INVALID_OPERATOR_TYPES

Unable to apply the specified operator to the left and right schema types.

{"type": "INVALID_OPERATOR_TYPES", "operator": "-", "leftType": "double", "rightType": "string" }

In the following example, the user is a double type and the host is a string type and no subtraction operator is supported for these types.

from /streams/demo/infra/cpu
select user - host

INVALID_STREAM_REFERENCE

The query references an invalid stream (physical or logical).

{ "type": "INVALID_STREAM_REFERENCE", "name": "/streams/demo/infra/cpu" }

INVALID_TENANT_REFERENCE

Valo does not support accessing resource in a different tenant. This issue is raised if a query within a session is referencing a different tenant than the session belongs to.

{ "type": "INVALID_TENANT_REFERENCE", "expected": "A", "actual": "B" }

INVALID_VARIABLE_TYPE

The variable does not evaluate to the correct type and cannot be applied to the query.

{ "type": "INVALID_VARIABLE_TYPE", "name": "myVar", "actualType": "date", "expectedType": "int" }

LOGICAL_STREAM_CONFLICT

A session declares several queries which references the same logical stream, but with different schemas.

{
   "type": "LOGICAL_STREAM_CONFLICT",
   "uri": "x"
}

The following query

from /streams/demo/infra/cpu
where user, kernel
into x

and this

from /streams/demo/infra/cpu
where user, host
into x

would resolve into a conflict as the schemas are not the same.

MISSING_VARIABLE

The variable referenced by the query was not found in the execution context.

{ "type": "MISSING_VARIABLE", "name": "myVar", "expectedType": "int" }

The issue is generated if the variable level in the following example hasn’t been defined in the session.

from /streams/demo/infra/cpu
where user > @level

NO_REPLICAS_AVAILABLE

Unable to access repository data. This can happen if the cluster node(s) holding the data are unavailable.

{
    "type": "NO_REPLICAS_AVAILABLE",
    "uri": "/streams/demo/infra/cpu"
}

NOT_ALIASED

The query is not fully aliased.

{ "type": "NOT_ALIASED" }

STREAM_NOT_REPOSITORY_MAPPED

An historical query references a stream which is not repository mapped.

{ "type": "STREAM_NOT_REPOSITORY_MAPPED", "name": "/streams/demo/infra/cpu" }

TARGET_STREAM_DOES_NOT_EXIST

The target stream defined in a query does not exist.

{
  "type": "TARGET_STREAM_DOES_NOT_EXIST",
  "uri": "/streams/demo/infra/cpu_derived",
  "expectedSchema": {
    "type": "record",
    "properties": {
      "host": {
        "type": "string"
      },
      "user": {
        "type": "double"
      }
    }
  }
}

The expectedSchema field is the schema which is derived from the query output and can be used as a basis for creating the stream in Valo.

This issue would be generated if the stream /streams/demo/infra/cpu_derived in the following example does not exist

from /streams/demo/infra/cpu
where user, host
into /streams/demo/infra/cpu_derived

TARGET_STREAM_SCHEMA_CONFLICT

The target stream exists but the schema for the query does not match the stream schema.

{
  "type": "TARGET_STREAM_SCHEMA_CONFLICT",
  "uri": "/streams/demo/infra/cpu_derived",
  "actualSchema": {
    "type": "record",
    "properties": {
      "host": {
        "type": "string"
      }
    }
  },
  "expectedSchema": {
    "type": "record",
    "properties": {
      "host": {
        "type": "string"
      },
      "user": {
        "type": "double"
      }
    }
  }
}

The actualSchema field is the schema which is derived from the query and the expectedSchema is the schema of the target stream.

UNABLE_TO_PARSE_QUERY

Failed to parse the query.

{"type": "UNABLE_TO_PARSE_QUERY", "line": 10, "column": 5, "description": "whoa"}

AMBIGUOUS_FIELD_REFERENCE

A field if referenced which cannot be resolved as it is defined in multiple inputs (a join for example).

{ "type": "AMBIGUOUS_FIELD_REFERENCE", "path": "user" }

UNKNOWN

An unknown/unhandled exception occurred. This error should not happen and if it does, please let the Valo team know so it can be fixed.

{"type": "UNKNOWN", "msg":"Failed to rewrite expression"}

VALIDATION_ERROR

A generic validation error related to the query topology.

{
  "type": "VALIDATION_ERROR",
  "msg": "Join node should have two inputs. Only one specified."
}