OwlCyberSecurity - MANAGER
Edit File: lexer.cpython-39.pyc
a ���e� � @ sP d Z ddlZddlmZ ddlmZmZ ddlmZ G dd� d�Z d dd �Z dS ) z SQL Lexer� N)� TextIOBase)�tokens�keywords)�consumec @ sR e Zd ZdZdZedd� �Zdd� Zdd� Zd d � Z d d � Z d d� Z ddd�Z dS )�LexerzrThe Lexer supports configurable syntax. To add support for additional keywords, use the `add_keywords` method.Nc C s" | j du r | � | _ | j �� | j S )zRReturns the lexer instance used internally by the sqlparse core functions.N)�_default_intance�default_initialization)�cls� r �O/home/gouroczh/virtualenv/pat/3.9/lib/python3.9/site-packages/sqlparse/lexer.py�get_default_instance. s zLexer.get_default_instancec C s` | � � | �tj� | �tj� | �tj� | �tj� | �tj� | �tj � | �tj � dS )zlInitialize the lexer with default dictionaries. Useful if you need to revert custom syntax settings.N) �clear� set_SQL_REGEXr � SQL_REGEX� add_keywordsZKEYWORDS_COMMONZKEYWORDS_ORACLEZKEYWORDS_PLPGSQLZ KEYWORDS_HQLZKEYWORDS_MSACCESS�KEYWORDS��selfr r r r 7 s z Lexer.default_initializationc C s g | _ g | _dS )z�Clear all syntax configurations. Useful if you want to load a reduced set of syntax configurations. After this call, regexps and keyword dictionaries need to be loaded to make the lexer functional again.N)� _SQL_REGEX� _keywordsr r r r r C s z Lexer.clearc s$ t jt jB � � fdd�|D �| _dS )z.Set the list of regex that will parse the SQL.c s" g | ]\}}t �|� �j|f�qS r )�re�compile�match)�.0�rx�tt��FLAGSr r � <listcomp>N s �z'Lexer.set_SQL_REGEX.<locals>.<listcomp>N)r � IGNORECASE�UNICODEr )r r r r r r K s �zLexer.set_SQL_REGEXc C s | j �|� dS )zhAdd keyword dictionaries. Keywords are looked up in the same order that dictionaries were added.N)r �append)r r r r r r S s zLexer.add_keywordsc C s6 |� � }| jD ] }||v r|| |f S qtj|fS )z�Checks for a keyword. If the given value is in one of the KEYWORDS_* dictionary it's considered a keyword. Otherwise, tokens.Name is returned. N)�upperr r �Name)r �value�valZkwdictr r r � is_keywordX s zLexer.is_keywordc c s t |t�r|�� }t |t�r nZt |t�rf|r8|�|�}qxz|�d�}W qx tyb |�d�}Y qx0 ntd�t |����t |�}|D ]�\}}| j D ]j\}}|||�}|s�q�n6t |t j �r�||�� fV n|tju r�| �|�� �V t||�� | d � q�q�t j|fV q�dS )a� Return an iterable of (tokentype, value) pairs generated from `text`. If `unfiltered` is set to `True`, the filtering mechanism is bypassed even if filters are defined. Also preprocess the text, i.e. expand tabs and strip it if wanted and applies registered filters. Split ``text`` into (tokentype, text) pairs. ``stack`` is the initial stack (default: ``['root']``) zutf-8zunicode-escapez+Expected text or file-like object, got {!r}� N)� isinstancer �read�str�bytes�decode�UnicodeDecodeError� TypeError�format�type� enumerater r � _TokenType�groupr ZPROCESS_AS_KEYWORDr& r �end�Error) r �text�encoding�iterable�pos�char�rexmatch�action�mr r r � get_tokense s6 � zLexer.get_tokens)N) �__name__� __module__� __qualname__�__doc__r � classmethodr r r r r r& r> r r r r r s r c C s t �� �| |�S )z�Tokenize sql. Tokenize *sql* using the :class:`Lexer` and return a 2-tuple stream of ``(token type, value)`` items. )r r r> )�sqlr7 r r r �tokenize� s rE )N) rB r �ior Zsqlparser r Zsqlparse.utilsr r rE r r r r �<module> s