OwlCyberSecurity - MANAGER
Edit File: lexer.cpython-39.pyc
a )��e} � @ s� d Z ddlZddlZddlZddlmZmZ ddlmZ ddl m Z m Z m Z m Z ddlmZmZmZmZmZmZ ddlmZ g d�Zg d �Zed d � �ZG d d � d e�Z G dd� de d�Z G dd� de �Z G dd� de �Z G dd� d�Z!e!� Z"G dd� de#�Z$G dd� d�Z%dd � Z&G d d � d �Z'e'� Z(d d � Z)G d!d"� d"�Z*G d#d$� d$e�Z+G d%d&� d&e �Z,G d'd(� d(e e,d�Z-G d)d*� d*�Z.G d+d,� d,e-�Z/d-d.� Z0G d/d0� d0e,�Z1G d1d2� d2e-e1d�Z2dS )3z� pygments.lexer ~~~~~~~~~~~~~~ Base lexer classes. :copyright: Copyright 2006-2022 by the Pygments team, see AUTHORS. :license: BSD, see LICENSE for details. � N)� apply_filters�Filter)�get_filter_by_name)�Error�Text�Other� _TokenType)� get_bool_opt� get_int_opt� get_list_opt�make_analysator�Future� guess_decode)� regex_opt) �Lexer� RegexLexer�ExtendedRegexLexer�DelegatingLexer� LexerContext�include�inherit�bygroups�using�this�default�words))s �utf-8)s �� zutf-32)s ��zutf-32be)s ��zutf-16)s ��zutf-16bec C s dS )N� � ��xr r �[/home/gouroczh/virtualenv/pat/3.9/lib/python3.9/site-packages/pip/_vendor/pygments/lexer.py�<lambda>! � r" c @ s e Zd ZdZdd� ZdS )� LexerMetaz� This metaclass automagically converts ``analyse_text`` methods into static methods which always return float values. c C s( d|v rt |d �|d< t�| |||�S )N� analyse_text)r �type�__new__)Zmcs�name�bases�dr r r! r' * s zLexerMeta.__new__N)�__name__� __module__� __qualname__�__doc__r' r r r r! r$ $ s r$ c @ s^ e Zd ZdZdZdZg Zg Zg Zg Z dZ dd� Z dd� Z dd � Z d d � Zdd d�Zdd� ZdS )r a� Lexer for a specific language. Basic options recognized: ``stripnl`` Strip leading and trailing newlines from the input (default: True). ``stripall`` Strip all leading and trailing whitespace from the input (default: False). ``ensurenl`` Make sure that the input ends with a newline (default: True). This is required for some lexers that consume input linewise. .. versionadded:: 1.3 ``tabsize`` If given and greater than 0, expand tabs in the input (default: 0). ``encoding`` If given, must be an encoding name. This encoding will be used to convert the input string to Unicode, if it is not already a Unicode string (default: ``'guess'``, which uses a simple UTF-8 / Locale / Latin1 detection. Can also be ``'chardet'`` to use the chardet library, if it is installed. ``inencoding`` Overrides the ``encoding`` if given. Nr c K s� || _ t|dd�| _t|dd�| _t|dd�| _t|dd�| _|�dd �| _|�d �pZ| j| _g | _ t |d d �D ]}| � |� qpd S ) N�stripnlT�stripallF�ensurenl�tabsizer �encoding�guess� inencoding�filtersr ) �optionsr r/ r0 r1 r r2 �getr3 r6 r � add_filter)�selfr7 �filter_r r r! �__init__a s zLexer.__init__c C s( | j rd| jj| j f S d| jj S d S )Nz <pygments.lexers.%s with %r>z<pygments.lexers.%s>)r7 � __class__r+ �r: r r r! �__repr__m s �zLexer.__repr__c K s* t |t�st|fi |��}| j�|� dS )z8 Add a new stream filter to this lexer. N)� isinstancer r r6 �append)r: r; r7 r r r! r9 t s zLexer.add_filterc C s dS )a~ Has to return a float between ``0`` and ``1`` that indicates if a lexer wants to highlight this text. Used by ``guess_lexer``. If this method returns ``0`` it won't highlight it in any case, if it returns ``1`` highlighting with this lexer is guaranteed. The `LexerMeta` metaclass automatically wraps this function so that it works like a static method (no ``self`` or ``cls`` parameter) and the return value is automatically converted to `float`. If the return value is an object that is boolean `False` it's the same as if the return values was ``0.0``. Nr )�textr r r! r% | s zLexer.analyse_textFc s� t �t��s� jdkr$t��\�}n� jdkr�zddlm} W n. tyl } ztd�|�W Y d}~n d}~0 0 d}tD ].\}}��|�rv�t |�d� � |d�} q�qv|du r�|� �dd� �} �� | � d �p�d d�}|�n(�� � j����d ��r"�t d �d� �n ��d ��r"�t d �d� ��� d d ���� dd ��� j�rL��� �n� j�r^��d ��� jdk�rv��� j��� j�r���d ��s��d 7 �� �fdd�} | � } |�s�t| � j� �} | S )a= Return an iterable of (tokentype, value) pairs generated from `text`. If `unfiltered` is set to `True`, the filtering mechanism is bypassed even if filters are defined. Also preprocess the text, i.e. expand tabs and strip it if wanted and applies registered filters. r4 �chardetr )rC zkTo enable chardet encoding guessing, please install the chardet library from http://chardet.feedparser.org/N�replacei r3 r u z � � c 3 s$ � � ��D ]\} }}||fV q d S �N)�get_tokens_unprocessed)�_�t�v�r: rB r r! �streamer� s z"Lexer.get_tokens.<locals>.streamer)r@ �strr3 r Z pip._vendorrC � ImportError� _encoding_map� startswith�len�decodeZdetectr8 rD r0 �stripr/ r2 � expandtabsr1 �endswithr r6 ) r: rB Z unfilteredrI rC �eZdecodedZbomr3 �encrM �streamr rL r! � get_tokens� sR � � zLexer.get_tokensc C s t �dS )z� Return an iterable of (index, tokentype, value) pairs where "index" is the starting position of the token within the input text. In subclasses, implement this method as a generator to maximize effectiveness. N)�NotImplementedErrorrL r r r! rH � s z Lexer.get_tokens_unprocessed)F)r+ r, r- r. r( �url�aliases� filenames�alias_filenames� mimetypes�priorityr< r? r9 r% rZ rH r r r r! r 0 s ;r )� metaclassc @ s$ e Zd ZdZefdd�Zdd� ZdS )r a This lexer takes two lexer as arguments. A root lexer and a language lexer. First everything is scanned using the language lexer, afterwards all ``Other`` tokens are lexed using the root lexer. The lexers from the ``template`` lexer package use this base lexer. c K s< |f i |��| _ |f i |��| _|| _tj| fi |�� d S rG )� root_lexer�language_lexer�needler r< )r: Z _root_lexerZ_language_lexerZ_needler7 r r r! r< � s zDelegatingLexer.__init__c C s� d}g }g }| j �|�D ]H\}}}|| ju rP|rF|�t|�|f� g }||7 }q|�|||f� q|rx|�t|�|f� t|| j�|��S )N� )rd rH re rA rR � do_insertionsrc )r: rB Zbuffered� insertionsZ lng_buffer�irJ rK r r r! rH � s �z&DelegatingLexer.get_tokens_unprocessedN)r+ r, r- r. r r< rH r r r r! r � s r c @ s e Zd ZdZdS )r zI Indicates that a state should include rules from another state. N�r+ r, r- r. r r r r! r � s r c @ s e Zd ZdZdd� ZdS )�_inheritzC Indicates the a state should inherit from its superclass. c C s dS )Nr r r> r r r! r? s z_inherit.__repr__N)r+ r, r- r. r? r r r r! rk � s rk c @ s e Zd ZdZdd� Zdd� ZdS )�combinedz: Indicates a state combined from multiple states. c G s t �| |�S rG )�tupler' )�cls�argsr r r! r' s zcombined.__new__c G s d S rG r )r: ro r r r! r< s zcombined.__init__N)r+ r, r- r. r' r< r r r r! rl s rl c @ sF e Zd ZdZdd� Zddd�Zddd�Zdd d �Zd d � Zd d� Z dS )� _PseudoMatchz: A pseudo match object constructed from a string. c C s || _ || _d S rG )�_text�_start)r: �startrB r r r! r< s z_PseudoMatch.__init__Nc C s | j S rG )rr �r: �argr r r! rs s z_PseudoMatch.startc C s | j t| j� S rG )rr rR rq rt r r r! �end! s z_PseudoMatch.endc C s |r t d��| jS )Nz No such group)� IndexErrorrq rt r r r! �group$ s z_PseudoMatch.groupc C s | j fS rG )rq r> r r r! �groups) s z_PseudoMatch.groupsc C s i S rG r r> r r r! � groupdict, s z_PseudoMatch.groupdict)N)N)N) r+ r, r- r. r< rs rv rx ry rz r r r r! rp s rp c s d� fdd� }|S )zL Callback that yields multiple actions for each group in the match. Nc 3 s� t � �D ]�\}}|d u r qqt|�tu rR|�|d �}|r�|�|d �||fV q|�|d �}|d ur|r||�|d �|_|| t|�|d �|�|�D ]}|r�|V q�q|r�|�� |_d S )N� )� enumerater& r rx rs �posrp rv )�lexer�match�ctxri �action�data�item�ro r r! �callback4 s$ � zbygroups.<locals>.callback)Nr )ro r� r r� r! r 0 s r c @ s e Zd ZdZdS )�_ThiszX Special singleton used for indicating the caller class. Used by ``using``. Nrj r r r r! r� J s r� c sj i �d�v r:�� d�}t|ttf�r.|�d< n d|f�d< � tu rTd��fdd� }nd � ��fdd� }|S ) a� Callback that processes the match with a different lexer. The keyword arguments are forwarded to the lexer, except `state` which is handled separately. `state` specifies the state that the new lexer will start in, and can be an enumerable such as ('root', 'inline', 'string') or a simple string which is assumed to be on top of the root state. Note: For that to work, `_other` must not be an `ExtendedRegexLexer`. �state�stack�rootNc 3 sr �r"�� | j� | jf i ���}n| }|�� }|j|�� fi � ��D ]\}}}|| ||fV qD|rn|�� |_d S rG )�updater7 r= rs rH rx rv r} �r~ r r� Zlx�sri rJ rK )� gt_kwargs�kwargsr r! r� i s zusing.<locals>.callbackc 3 sf �� | j� � f i ���}|�� }|j|�� fi ���D ]\}}}|| ||fV q8|rb|�� |_d S rG )r� r7 rs rH rx rv r} r� ��_otherr� r� r r! r� x s )N)N)�popr@ �listrm r )r� r� r� r� r r� r! r S s r c @ s e Zd ZdZdd� ZdS )r z� Indicates a state or state action (e.g. #pop) to apply. For example default('#pop') is equivalent to ('', Token, '#pop') Note that state tuples may be used as well. .. versionadded:: 2.0 c C s || _ d S rG )r� )r: r� r r r! r< � s zdefault.__init__N)r+ r, r- r. r< r r r r! r � s r c @ s" e Zd ZdZddd�Zdd� ZdS ) r z� Indicates a list of literal words that is transformed into an optimized regex that matches any of the words. .. versionadded:: 2.0 rf c C s || _ || _|| _d S rG )r �prefix�suffix)r: r r� r� r r r! r< � s zwords.__init__c C s t | j| j| jd�S )N�r� r� )r r r� r� r> r r r! r8 � s z words.getN)rf rf )r+ r, r- r. r< r8 r r r r! r � s r c @ sJ e Zd ZdZdd� Zdd� Zdd� Zdd � Zdd d �Zd d� Z dd� Z d S )�RegexLexerMetazw Metaclass for RegexLexer, creates the self._tokens attribute from self.tokens on the first instantiation. c C s t |t�r|�� }t�||�jS )zBPreprocess the regular expression component of a token definition.)r@ r r8 �re�compiler )rn �regex�rflagsr� r r r! �_process_regex� s z RegexLexerMeta._process_regexc C s& t |�tu s"t|�s"J d|f ��|S )z5Preprocess the token component of a token definition.z2token type must be simple type or callable, not %r)r& r �callable)rn �tokenr r r! �_process_token� s �z RegexLexerMeta._process_tokenc C s t |t�rd|dkrdS ||v r$|fS |dkr0|S |dd� dkrRt|dd� � S dsbJ d| ��n�t |t�r�d | j }| jd 7 _g }|D ],}||ks�J d | ��|�| �|||�� q�|||< |fS t |t��r |D ] }||v s�|d v s�J d | ��q�|S d�sJ d| ��dS )z=Preprocess the state transition action of a token definition.�#pop����#pushN� z#pop:Fzunknown new state %rz_tmp_%dr{ zcircular state ref %r)r� r� zunknown new state zunknown new state def %r)r@ rN �intrl �_tmpname�extend�_process_staterm )rn � new_state� unprocessed� processedZ tmp_state�itokensZistater r r! �_process_new_state� s< � ��z!RegexLexerMeta._process_new_statec C s� t |�tu sJ d| ��|d dks0J d| ��||v r@|| S g }||< | j}|| D �]0}t|t�r�||ks~J d| ��|�| �||t|��� qZt|t�r�qZt|t�r�| � |j ||�}|� t � d�jd|f� qZt |�tu s�J d| ��z| �|d ||�}W n@ t�yF } z&td |d || | f �| �W Y d} ~ n d} ~ 0 0 | �|d �} t|�d k�rjd}n| � |d ||�}|� || |f� qZ|S ) z%Preprocess a single state definition.zwrong state name %rr �#zinvalid state name %rzcircular state reference %rrf Nzwrong rule def %rz+uncompilable regex %r in state %r of %r: %sr{ � )r& rN �flagsr@ r r� r� rk r r� r� rA r� r� r rm r� � Exception� ValueErrorr� rR ) rn r� r� r� �tokensr� Ztdefr� �rex�errr� r r r! r� � sH � �� �z RegexLexerMeta._process_stateNc C s< i }| j |< |p| j| }t|�D ]}| �|||� q$|S )z-Preprocess a dictionary of token definitions.)� _all_tokensr� r� r� )rn r( � tokendefsr� r� r r r! �process_tokendef s z RegexLexerMeta.process_tokendefc C s� i }i }| j D ]�}|j�di �}|�� D ]�\}}|�|�}|du rz|||< z|�t�}W n tyn Y q(Y n0 |||< q(|�|d�}|du r�q(||||d �< z|�t�} W n ty� Y q(0 || ||< q(q|S )a Merge tokens from superclasses in MRO order, returning a single tokendef dictionary. Any state that is not defined by a subclass will be inherited automatically. States that *are* defined by subclasses will, by default, override that state in the superclass. If a subclass wishes to inherit definitions from a superclass, it can use the special value "inherit", which will cause the superclass' state definition to be included at that point in the state. r� Nr{ )�__mro__�__dict__r8 �items�indexr r� r� ) rn r� � inheritable�c�toksr� r� ZcuritemsZ inherit_ndxZ new_inh_ndxr r r! � get_tokendefs s0 z RegexLexerMeta.get_tokendefsc O sR d| j vr:i | _d| _t| d�r(| jr(n| �d| �� �| _tj | g|�R i |��S )z:Instantiate cls after preprocessing its token definitions.�_tokensr �token_variantsrf ) r� r� r� �hasattrr� r� r� r� r& �__call__)rn ro �kwdsr r r! r� ; s zRegexLexerMeta.__call__)N) r+ r, r- r. r� r� r� r� r� r� r� r r r r! r� � s #, 1r� c @ s$ e Zd ZdZejZi Zddd�ZdS )r z� Base for simple stateful regular expression-based lexers. Simplifies the lexing process so that you need only provide a list of states and regular expressions. �r� c c s� d}| j }t|�}||d }|D �](\}}} |||�} | r"|durrt|�tu rb||| �� fV n|| | �E dH | �� }| du�rHt| t�r�| D ]D} | dkr�t|�dkr�|� � q�| dkr�|� |d � q�|� | � q�nbt| t ��rt | �t|�k�r|dd�= n || d�= n,| dk�r*|� |d � nd�s<J d| ��||d } q q"zP|| d k�r�d g}|d }|t d fV |d7 }W q |t|| fV |d7 }W q t�y� Y �q�Y q 0 q dS ) z~ Split ``text`` into (tokentype, text) pairs. ``stack`` is the initial stack (default: ``['root']``) r r� Nr� r{ r� F�wrong state def: %rrE r� )r� r� r& r rx rv r@ rm rR r� rA r� �absr r rw ) r: rB r� r} r� Z statestack� statetokens�rexmatchr� r� �mr� r r r! rH l sR z!RegexLexer.get_tokens_unprocessedN)r� ) r+ r, r- r. r� � MULTILINEr� r� rH r r r r! r I s r c @ s" e Zd ZdZddd�Zdd� ZdS )r z9 A helper object that holds lexer position data. Nc C s* || _ || _|pt|�| _|p"dg| _d S )Nr� )rB r} rR rv r� )r: rB r} r� rv r r r! r< � s zLexerContext.__init__c C s d| j | j| jf S )NzLexerContext(%r, %r, %r))rB r} r� r> r r r! r? � s �zLexerContext.__repr__)NN)r+ r, r- r. r< r? r r r r! r � s r c @ s e Zd ZdZddd�ZdS )r zE A RegexLexer that uses a context object to store its state. Nc c s: | j }|s t|d�}|d }n|}||jd }|j}|D �]`\}}}|||j|j�} | r:|dur�t|�tu r�|j|| �� fV | �� |_n$|| | |�E dH |s�||jd }|du�r�t |t ��r |D ]P} | dkr�t |j�dkr�|j� � q�| dk�r|j� |jd � q�|j� | � q�nlt |t��rZt|�t |j�k�rL|jdd�= n |j|d�= n0|dk�rx|j� |jd � nd�s�J d | ��||jd } q6q:zz|j|jk�r�W �q6||j d k�r�dg|_|d }|jtd fV | jd7 _W q6|jt||j fV | jd7 _W q6 t�y2 Y �q6Y q60 q6dS ) z Split ``text`` into (tokentype, text) pairs. If ``context`` is given, use this lexer context instead. r r� r� Nr� r{ r� Fr� rE )r� r r� rB r} rv r& r rx r@ rm rR r� rA r� r� r r rw ) r: rB �contextr� r� r� r� r� r� r� r� r r r! rH � s` z)ExtendedRegexLexer.get_tokens_unprocessed)NN)r+ r, r- r. rH r r r r! r � s r c c s� t | �} zt| �\}}W n ty6 |E dH Y dS 0 d}d}|D ]�\}}}|du rZ|}d} |�r|t|� |k�r|| || � } | r�||| fV |t| �7 }|D ]"\} } } || | fV |t| �7 }q�|| } zt| �\}}W q^ t�y d}Y �qY q^0 q^| t|�k rD|||| d� fV |t|�| 7 }qD|�r�|�pDd}|D ]$\}}}|||fV |t|�7 }�qJzt| �\}}W n t�y� d}Y �q�Y n0 �q6dS )ag Helper for lexers which must combine the results of several sublexers. ``insertions`` is a list of ``(index, itokens)`` pairs. Each ``itokens`` iterable should be inserted at position ``index`` into the token stream given by the ``tokens`` argument. The result is a combined token stream. TODO: clean up the code here. NTr F)�iter�next� StopIterationrR )rh r� r� r� ZrealposZinsleftri rJ rK ZoldiZtmpvalZit_indexZit_tokenZit_value�pr r r! rg sN rg c @ s e Zd ZdZdd� ZdS )�ProfilingRegexLexerMetaz>Metaclass for ProfilingRegexLexer, collects regex timing info.c sL t |t�r t|j|j|jd��n|�t��|��tjf� ���fdd� }|S )Nr� c s` � j d ���fddg�}t�� }��| ||�}t�� }|d d7 < |d || 7 < |S )Nr� r r r{ )� _prof_data� setdefault�timer )rB r} �endpos�info�t0�res�t1�rn �compiledr� r� r r! � match_funcM s z:ProfilingRegexLexerMeta._process_regex.<locals>.match_func) r@ r r r� r� r� r� �sys�maxsize)rn r� r� r� r� r r� r! r� E s � z&ProfilingRegexLexerMeta._process_regexN)r+ r, r- r. r� r r r r! r� B s r� c @ s" e Zd ZdZg ZdZddd�ZdS )�ProfilingRegexLexerzFDrop-in replacement for RegexLexer that does profiling of its regexes.� r� c # s� � j j�i � t�� ||�E d H � j j�� }tdd� |�� D �� fdd�dd�}tdd� |D ��}t � t d� j j t |�|f � t d � t d d � t d � |D ]}t d | � q�t d � d S )Nc s sN | ]F\\}}\}}|t |��d ��dd�dd� |d| d| | fV qdS )zu'z\\�\N�A i� )�reprrT rD )�.0r� �r�nrJ r r r! � <genexpr>c s � �z=ProfilingRegexLexer.get_tokens_unprocessed.<locals>.<genexpr>c s | � j S rG )�_prof_sort_indexr r> r r! r" f r# z<ProfilingRegexLexer.get_tokens_unprocessed.<locals>.<lambda>T)�key�reversec s s | ]}|d V qdS )� Nr )r� r r r r! r� h r# z2Profiling result for %s lexing %d chars in %.3f mszn==============================================================================================================z$%-20s %-64s ncalls tottime percall)r� r� zn--------------------------------------------------------------------------------------------------------------z%-20s %-65s %5d %8.4f %8.4f) r= r� rA r rH r� �sortedr� �sum�printr+ rR )r: rB r� Zrawdatar� Z sum_totalr* r r> r! rH ^ s( � �� z*ProfilingRegexLexer.get_tokens_unprocessedN)r� )r+ r, r- r. r� r� rH r r r r! r� X s r� )3r. r� r� r� Zpip._vendor.pygments.filterr r Z pip._vendor.pygments.filtersr �pip._vendor.pygments.tokenr r r r �pip._vendor.pygments.utilr r r r r r Z pip._vendor.pygments.regexoptr �__all__rP � staticmethodZ_default_analyser& r$ r r rN r rk r rm rl rp r r� r r r r r� r r r rg r� r� r r r r! �<module> sD !' 2 )aH@