pglast.parser — The interface with libpg_query

This module is a C extension written in Cython that exposes a few functions from the underlying libpg_query library it links against.

pglast.parser.LONG_MAX

The highest integer that can be stored in a C long variable: it is used as a marker, for example in PG’s FetchStmt.howMany, that uses the constant FETCH_ALL.

exception pglast.parser.ParseError

Exception representing the error state returned by the parser.

exception pglast.parser.DeparseError

Exception representing the error state returned by the deparser.

class pglast.parser.Displacements(string)

Helper class used to find the index of Unicode character from its offset in the corresponding UTF-8 encoded array.

Example:

>>> from pglast.parser import Displacements
>>> unicode = '€ 0.01'
>>> utf8 = unicode.encode('utf-8')
>>> d = Displacements(unicode)
>>> for offset in range(len(utf8)):
...   idx = d(offset)
...   print(f'{offset} [{utf8[offset]:2x}] -> {idx} [{unicode[idx]}]')
...
0 [e2] -> 0 [€]
1 [82] -> 0 [€]
2 [ac] -> 0 [€]
3 [20] -> 1 [ ]
4 [30] -> 2 [0]
5 [2e] -> 3 [.]
6 [30] -> 4 [0]
7 [31] -> 5 [1]

The underlying libpg_parse library operates on UTF-8 strings: its parser functions emit tokens with a location, that is actually the offset within the UTF-8 representation of the statement. With this class you can fixup those offsets, like in the following example:

>>> import json
>>> from pglast.parser import parse_sql_json
>>> stmt = 'select alias.bar as alìbàbà from foo as alias'
>>> parsed = json.loads(parse_sql_json(stmt))
>>> select = parsed['stmts'][0]['stmt']['SelectStmt']
>>> rangevar = select['fromClause'][0]['RangeVar']
>>> loc = rangevar['location']
>>> print(stmt[loc:loc+3])
 as
>>> d = Displacements(stmt)
>>> adjloc = d(loc)
>>> print(stmt[adjloc:adjloc+3])
foo
pglast.parser.deparse_protobuf(buffer)
Parameters:

buffer (bytes) – a Protobuf buffer

Returns:

str

Return the SQL statement from the given buffer argument, something generated by parse_sql_protobuf().

pglast.parser.fingerprint(query)
Parameters:

query (str) – The SQL statement

Returns:

str

Fingerprint the given query, a string with the SQL statement(s), and return a hash digest that can identify similar queries. For similar queries that are different only because of the queried object or formatting, the returned digest will be the same.

pglast.parser.get_postgresql_version()
Returns:

a tuple

Return the PostgreSQL version as a tuple (major, minor, patch).

pglast.parser.parse_sql(query)
Parameters:

query (str) – The SQL statement

Returns:

tuple

Parse the given query, a string with the SQL statement(s), and return the corresponding parse tree as a tuple of pglast.ast.RawStmt instances.

pglast.parser.parse_sql_json(query)
Parameters:

query (str) – The SQL statement

Returns:

str

Parse the given query, a string with the SQL statement(s), and return the libpg_query‘s JSON-serialized parse tree.

pglast.parser.parse_sql_protobuf(query)
Parameters:

query (str) – The SQL statement

Returns:

bytes

Parse the given query, a string with the SQL statement(s), and return the libpg_query‘s Protobuf-serialized parse tree.

pglast.parser.parse_plpgsql_json(query)
Parameters:

query (str) – The PLpgSQL statement

Returns:

str

Parse the given query, a string with the plpgsql statement(s), and return the libpg_query‘s JSON-serialized parse tree.

pglast.parser.scan(query)
Parameters:

query (str) – The SQL statement

Returns:

sequence of tuples

Split the given query into its tokens. Each token is a namedtuple with the following slots:

startint

the index of the start of the token

endint

the index of the end of the token

namestr

the name of the token

kindstr

the kind of the token

Example:

>>> from pglast.parser import scan
>>> stmt = 'select bar as alìbàbà from foo'
>>> tokens = scan(stmt)
>>> print(tokens[0])
Token(start=0, end=5, name='SELECT', kind='RESERVED_KEYWORD')
>>> print([stmt[t.start:t.end+1] for t in tokens])
['select', 'bar', 'as', 'alìbàbà', 'from', 'foo']
pglast.parser.split(query, with_parser=True, only_slices=False)
Parameters:

  • query (str) – The SQL statement

  • with_parser (bool) – Whether to use the parser or the scanner

  • only_slices (bool) – Return slices instead of statement’s text

Returns:

tuple

Split the given stmts string into a sequence of the single SQL statements.

By default this uses the parser to perform the job; when with_parser is False the scanner variant is used, indicated when the statements may contain parse errors.

When only_slices is True, return a sequence of slice instances, one for each statement, instead of statements text.

Note

Leading and trailing whitespace are removed from the statements.

Example:

>>> from pglast.parser import split
>>> split('select 1 for; select 2')
Traceback (most recent call last):
  ...
pglast.parser.ParseError: syntax error at or near ";", at index 12
>>> split('select 1 for; select 2', with_parser=False)
('select 1 for', 'select 2')
>>> stmts = "select 'fòò'; select 'bàr'"
>>> print([stmts[r] for r in split(stmts, only_slices=True)])
["select 'fòò'", "select 'bàr'"]