OwlCyberSecurity - MANAGER
Edit File: lexer.cpython-39.pyc
a gH�fg � @ s\ d Z ddlZddlmZ ddlmZ ddlmZmZ ddl m Z G dd� d�Z d d d �Z dS ) z SQL Lexer� N)�Lock)� TextIOBase)�tokens�keywords)�consumec @ sX e Zd ZdZdZe� Zedd� �Zdd� Z dd� Z d d � Z d d � Z d d� Z ddd�ZdS )�LexerzrThe Lexer supports configurable syntax. To add support for additional keywords, use the `add_keywords` method.Nc C sH | j �, | jdu r$| � | _| j�� W d � n1 s80 Y | jS )zRReturns the lexer instance used internally by the sqlparse core functions.N)�_lock�_default_instance�default_initialization)�cls� r �P/home/gouroczh/virtualenv/test/3.9/lib/python3.9/site-packages/sqlparse/lexer.py�get_default_instance0 s (zLexer.get_default_instancec C s� | � � | �tj� | �tj� | �tj� | �tj� | �tj� | �tj � | �tj � | �tj � | �tj � | �tj � dS )zlInitialize the lexer with default dictionaries. Useful if you need to revert custom syntax settings.N)�clear� set_SQL_REGEXr � SQL_REGEX� add_keywordsZKEYWORDS_COMMONZKEYWORDS_ORACLEZKEYWORDS_MYSQLZKEYWORDS_PLPGSQLZ KEYWORDS_HQLZKEYWORDS_MSACCESSZKEYWORDS_SNOWFLAKEZKEYWORDS_BIGQUERY�KEYWORDS��selfr r r r : s z Lexer.default_initializationc C s g | _ g | _dS )z�Clear all syntax configurations. Useful if you want to load a reduced set of syntax configurations. After this call, regexps and keyword dictionaries need to be loaded to make the lexer functional again.N)� _SQL_REGEX� _keywordsr r r r r I s z Lexer.clearc s$ t jt jB � � fdd�|D �| _dS )z.Set the list of regex that will parse the SQL.c s" g | ]\}}t �|� �j|f�qS r )�re�compile�match)�.0�rx�tt��FLAGSr r � <listcomp>T s �z'Lexer.set_SQL_REGEX.<locals>.<listcomp>N)r � IGNORECASE�UNICODEr )r r r r r r Q s �zLexer.set_SQL_REGEXc C s | j �|� dS )zhAdd keyword dictionaries. Keywords are looked up in the same order that dictionaries were added.N)r �append)r r r r r r Y s zLexer.add_keywordsc C s6 |� � }| jD ] }||v r|| |f S qtj|fS )z�Checks for a keyword. If the given value is in one of the KEYWORDS_* dictionary it's considered a keyword. Otherwise, tokens.Name is returned. N)�upperr r �Name)r �value�valZkwdictr r r � is_keyword^ s zLexer.is_keywordc c s t |t�r|�� }t |t�r nZt |t�rf|r8|�|�}qxz|�d�}W qx tyb |�d�}Y qx0 ntd�t |����t |�}|D ]�\}}| j D ]j\}}|||�}|s�q�n6t |t j �r�||�� fV n|tju r�| �|�� �V t||�� | d � q�q�t j|fV q�dS )a� Return an iterable of (tokentype, value) pairs generated from `text`. If `unfiltered` is set to `True`, the filtering mechanism is bypassed even if filters are defined. Also preprocess the text, i.e. expand tabs and strip it if wanted and applies registered filters. Split ``text`` into (tokentype, text) pairs. ``stack`` is the initial stack (default: ``['root']``) zutf-8zunicode-escapez+Expected text or file-like object, got {!r}� N)� isinstancer �read�str�bytes�decode�UnicodeDecodeError� TypeError�format�type� enumerater r � _TokenType�groupr ZPROCESS_AS_KEYWORDr( r �end�Error) r �text�encoding�iterable�pos�char�rexmatch�action�mr r r � get_tokensk s6 � zLexer.get_tokens)N)�__name__� __module__� __qualname__�__doc__r r r � classmethodr r r r r r( r@ r r r r r s r c C s t �� �| |�S )z�Tokenize sql. Tokenize *sql* using the :class:`Lexer` and return a 2-tuple stream of ``(token type, value)`` items. )r r r@ )�sqlr9 r r r �tokenize� s rG )N) rD r � threadingr �ior Zsqlparser r Zsqlparse.utilsr r rG r r r r �<module> s