OwlCyberSecurity - MANAGER
Edit File: lexer.cpython-39.pyc
a �F�fԉ � @ s� d Z ddlZddlZddlZddlmZmZ ddlmZ ddl m Z m Z m Z m Z mZ ddlmZmZmZmZmZmZ ddlmZ g d�Ze�d �Zg d �Ze d d � �Z G d d� de �Z G dd� de d�Z G dd� de �Z!G dd� de"�Z#G dd� d�Z$e$� Z%G dd� de&�Z'G dd� d�Z(d d � Z)G d d � d �Z*e*� Z+d d!� Z,G d"d#� d#�Z-G d$d%� d%e�Z.G d&d'� d'e �Z/G d(d)� d)e e/d�Z0G d*d+� d+�Z1G d,d-� d-e0�Z2d.d/� Z3G d0d1� d1e/�Z4G d2d3� d3e0e4d�Z5dS )4z� pygments.lexer ~~~~~~~~~~~~~~ Base lexer classes. :copyright: Copyright 2006-2023 by the Pygments team, see AUTHORS. :license: BSD, see LICENSE for details. � N)� apply_filters�Filter)�get_filter_by_name)�Error�Text�Other� Whitespace� _TokenType)� get_bool_opt� get_int_opt� get_list_opt�make_analysator�Future� guess_decode)� regex_opt) �Lexer� RegexLexer�ExtendedRegexLexer�DelegatingLexer� LexerContext�include�inherit�bygroups�using�this�default�words�line_rez.*? ))s �utf-8)s �� zutf-32)s ��zutf-32be)s ��zutf-16)s ��zutf-16bec C s dS )N� � ��xr r �\/home/gouroczh/virtualenv/test/3.9/lib/python3.9/site-packages/pip/_vendor/pygments/lexer.py�<lambda>" � r$ c @ s e Zd ZdZdd� ZdS )� LexerMetaz� This metaclass automagically converts ``analyse_text`` methods into static methods which always return float values. c C s( d|v rt |d �|d< t�| |||�S )N� analyse_text)r �type�__new__)Zmcs�name�bases�dr r r# r) + s zLexerMeta.__new__N)�__name__� __module__� __qualname__�__doc__r) r r r r# r& % s r& c @ sj e Zd ZdZdZg Zg Zg Zg ZdZ dZ dZ dd� Z dd� Z dd � Zd d � Zd d � Zddd�Zdd� ZdS )r a� Lexer for a specific language. See also :doc:`lexerdevelopment`, a high-level guide to writing lexers. Lexer classes have attributes used for choosing the most appropriate lexer based on various criteria. .. autoattribute:: name :no-value: .. autoattribute:: aliases :no-value: .. autoattribute:: filenames :no-value: .. autoattribute:: alias_filenames .. autoattribute:: mimetypes :no-value: .. autoattribute:: priority Lexers included in Pygments should have an additional attribute: .. autoattribute:: url :no-value: Lexers included in Pygments may have additional attributes: .. autoattribute:: _example :no-value: You can pass options to the constructor. The basic options recognized by all lexers and processed by the base `Lexer` class are: ``stripnl`` Strip leading and trailing newlines from the input (default: True). ``stripall`` Strip all leading and trailing whitespace from the input (default: False). ``ensurenl`` Make sure that the input ends with a newline (default: True). This is required for some lexers that consume input linewise. .. versionadded:: 1.3 ``tabsize`` If given and greater than 0, expand tabs in the input (default: 0). ``encoding`` If given, must be an encoding name. This encoding will be used to convert the input string to Unicode, if it is not already a Unicode string (default: ``'guess'``, which uses a simple UTF-8 / Locale / Latin1 detection. Can also be ``'chardet'`` to use the chardet library, if it is installed. ``inencoding`` Overrides the ``encoding`` if given. Nr c K s� || _ t|dd�| _t|dd�| _t|dd�| _t|dd�| _|�dd �| _|�d �pZ| j| _g | _ t |d d �D ]}| � |� qpd S )a� This constructor takes arbitrary options as keyword arguments. Every subclass must first process its own options and then call the `Lexer` constructor, since it processes the basic options like `stripnl`. An example looks like this: .. sourcecode:: python def __init__(self, **options): self.compress = options.get('compress', '') Lexer.__init__(self, **options) As these options must all be specifiable as strings (due to the command line usage), there are various utility functions available to help with that, see `Utilities`_. �stripnlT�stripallF�ensurenl�tabsizer �encoding�guess� inencoding�filtersr N) �optionsr r1 r2 r3 r r4 �getr5 r8 r � add_filter)�selfr9 �filter_r r r# �__init__� s zLexer.__init__c C s( | j rd| jj| j f S d| jj S d S )Nz <pygments.lexers.%s with %r>z<pygments.lexers.%s>)r9 � __class__r- �r< r r r# �__repr__� s �zLexer.__repr__c K s* t |t�st|fi |��}| j�|� dS )z8 Add a new stream filter to this lexer. N)� isinstancer r r8 �append)r< r= r9 r r r# r; � s zLexer.add_filterc C s dS )a� A static method which is called for lexer guessing. It should analyse the text and return a float in the range from ``0.0`` to ``1.0``. If it returns ``0.0``, the lexer will not be selected as the most probable one, if it returns ``1.0``, it will be selected immediately. This is used by `guess_lexer`. The `LexerMeta` metaclass automatically wraps this function so that it works like a static method (no ``self`` or ``cls`` parameter) and the return value is automatically converted to `float`. If the return value is an object that is boolean `False` it's the same as if the return values was ``0.0``. Nr )�textr r r# r' � s zLexer.analyse_textc C s� t |t�s�| jdkr"t|�\}}n�| jdkr�z td��W n. tyf } ztd�|�W Y d}~n d}~0 0 d}tD ].\}}|�|�rp|t|�d� �|d�} q�qp|du r�t � |dd� �}|�|� d�p�d d�}|}n&|�| j�}|�d �r�|td �d� }n |�d ��r|td �d� }|� d d �}|� d d �}| j �rD|�� }n| j�rV|�d �}| jdk�rn|�| j�}| j�r�|�d ��s�|d 7 }|S )zVApply preprocessing such as decoding the input, removing BOM and normalizing newlines.r6 �chardetz chardet is not vendored by pipzkTo enable chardet encoding guessing, please install the chardet library from http://chardet.feedparser.org/N�replacei r5 r u z � � r )rB �strr5 r � ImportError� _encoding_map� startswith�len�decoderE Zdetectr: rF r2 �stripr1 r4 � expandtabsr3 �endswith)r< rD �_�eZdecodedZbomr5 �encr r r# �_preprocess_lexer_input� sJ � � z Lexer._preprocess_lexer_inputFc s4 � � ���� �fdd�}|� }|s0t|� j� �}|S )ae This method is the basic interface of a lexer. It is called by the `highlight()` function. It must process the text and return an iterable of ``(tokentype, value)`` pairs from `text`. Normally, you don't need to override this method. The default implementation processes the options recognized by all lexers (`stripnl`, `stripall` and so on), and then yields all tokens from `get_tokens_unprocessed()`, with the ``index`` dropped. If `unfiltered` is set to `True`, the filtering mechanism is bypassed even if filters are defined. c 3 s$ � � ��D ]\} }}||fV q d S �N)�get_tokens_unprocessed)rR �t�v�r< rD r r# �streamer s z"Lexer.get_tokens.<locals>.streamer)rU r r8 )r< rD Z unfilteredr[ �streamr rZ r# � get_tokens� s zLexer.get_tokensc C s t �dS )aS This method should process the text and return an iterable of ``(index, tokentype, value)`` tuples where ``index`` is the starting position of the token within the input text. It must be overridden by subclasses. It is recommended to implement it as a generator to maximize effectiveness. N)�NotImplementedErrorrZ r r r# rW s z Lexer.get_tokens_unprocessed)F)r- r. r/ r0 r* �aliases� filenames�alias_filenames� mimetypes�priority�urlZ_exampler> rA r; r' rU r] rW r r r r# r 1 s 9 1 r )� metaclassc @ s$ e Zd ZdZefdd�Zdd� ZdS )r a This lexer takes two lexer as arguments. A root lexer and a language lexer. First everything is scanned using the language lexer, afterwards all ``Other`` tokens are lexed using the root lexer. The lexers from the ``template`` lexer package use this base lexer. c K s< |f i |��| _ |f i |��| _|| _tj| fi |�� d S rV )� root_lexer�language_lexer�needler r> )r< Z _root_lexerZ_language_lexerZ_needler9 r r r# r> ) s zDelegatingLexer.__init__c C s� d}g }g }| j �|�D ]H\}}}|| ju rP|rF|�t|�|f� g }||7 }q|�|||f� q|rx|�t|�|f� t|| j�|��S )N� )rg rW rh rC rM � do_insertionsrf )r< rD Zbuffered� insertionsZ lng_buffer�irX rY r r r# rW / s �z&DelegatingLexer.get_tokens_unprocessedN)r- r. r/ r0 r r> rW r r r r# r s r c @ s e Zd ZdZdS )r zI Indicates that a state should include rules from another state. N�r- r. r/ r0 r r r r# r F s r c @ s e Zd ZdZdd� ZdS )�_inheritzC Indicates the a state should inherit from its superclass. c C s dS )Nr r r@ r r r# rA Q s z_inherit.__repr__N)r- r. r/ r0 rA r r r r# rn M s rn c @ s e Zd ZdZdd� Zdd� ZdS )�combinedz: Indicates a state combined from multiple states. c G s t �| |�S rV )�tupler) )�cls�argsr r r# r) \ s zcombined.__new__c G s d S rV r )r< rr r r r# r> _ s zcombined.__init__N)r- r. r/ r0 r) r> r r r r# ro W s ro c @ sF e Zd ZdZdd� Zddd�Zddd�Zdd d �Zd d � Zd d� Z dS )� _PseudoMatchz: A pseudo match object constructed from a string. c C s || _ || _d S rV )�_text�_start)r< �startrD r r r# r> i s z_PseudoMatch.__init__Nc C s | j S rV )ru �r< �argr r r# rv m s z_PseudoMatch.startc C s | j t| j� S rV )ru rM rt rw r r r# �endp s z_PseudoMatch.endc C s |r t d��| jS )Nz No such group)� IndexErrorrt rw r r r# �groups s z_PseudoMatch.groupc C s | j fS rV )rt r@ r r r# �groupsx s z_PseudoMatch.groupsc C s i S rV r r@ r r r# � groupdict{ s z_PseudoMatch.groupdict)N)N)N) r- r. r/ r0 r> rv ry r{ r| r} r r r r# rs d s rs c s d� fdd� }|S )zL Callback that yields multiple actions for each group in the match. Nc 3 s� t � �D ]�\}}|d u r qqt|�tu rR|�|d �}|r�|�|d �||fV q|�|d �}|d ur|r||�|d �|_|| t|�|d �|�|�D ]}|r�|V q�q|r�|�� |_d S )N� )� enumerater( r r{ rv �posrs ry )�lexer�match�ctxrl �action�data�item�rr r r# �callback� s$ � zbygroups.<locals>.callback)Nr )rr r� r r� r# r s r c @ s e Zd ZdZdS )�_ThiszX Special singleton used for indicating the caller class. Used by ``using``. Nrm r r r r# r� � s r� c sj i �d�v r:�� d�}t|ttf�r.|�d< n d|f�d< � tu rTd��fdd� }nd � ��fdd� }|S ) a� Callback that processes the match with a different lexer. The keyword arguments are forwarded to the lexer, except `state` which is handled separately. `state` specifies the state that the new lexer will start in, and can be an enumerable such as ('root', 'inline', 'string') or a simple string which is assumed to be on top of the root state. Note: For that to work, `_other` must not be an `ExtendedRegexLexer`. �state�stack�rootNc 3 sr �r"�� | j� | jf i ���}n| }|�� }|j|�� fi � ��D ]\}}}|| ||fV qD|rn|�� |_d S rV )�updater9 r? rv rW r{ ry r� �r� r� r� Zlx�srl rX rY )� gt_kwargs�kwargsr r# r� � s zusing.<locals>.callbackc 3 sf �� | j� � f i ���}|�� }|j|�� fi ���D ]\}}}|| ||fV q8|rb|�� |_d S rV )r� r9 rv rW r{ ry r� r� ��_otherr� r� r r# r� � s )N)N)�poprB �listrp r )r� r� r� r� r r� r# r � s r c @ s e Zd ZdZdd� ZdS )r z� Indicates a state or state action (e.g. #pop) to apply. For example default('#pop') is equivalent to ('', Token, '#pop') Note that state tuples may be used as well. .. versionadded:: 2.0 c C s || _ d S rV )r� )r< r� r r r# r> � s zdefault.__init__N)r- r. r/ r0 r> r r r r# r � s r c @ s" e Zd ZdZddd�Zdd� ZdS ) r z� Indicates a list of literal words that is transformed into an optimized regex that matches any of the words. .. versionadded:: 2.0 ri c C s || _ || _|| _d S rV )r �prefix�suffix)r< r r� r� r r r# r> � s zwords.__init__c C s t | j| j| jd�S )N�r� r� )r r r� r� r@ r r r# r: � s z words.getN)ri ri )r- r. r/ r0 r> r: r r r r# r � s r c @ sJ e Zd ZdZdd� Zdd� Zdd� Zdd � Zdd d �Zd d� Z dd� Z d S )�RegexLexerMetazw Metaclass for RegexLexer, creates the self._tokens attribute from self.tokens on the first instantiation. c C s t |t�r|�� }t�||�jS )zBPreprocess the regular expression component of a token definition.)rB r r: �re�compiler� )rq �regex�rflagsr� r r r# �_process_regex� s z RegexLexerMeta._process_regexc C s& t |�tu s"t|�s"J d|f ��|S )z5Preprocess the token component of a token definition.z2token type must be simple type or callable, not %r)r( r �callable)rq �tokenr r r# �_process_token� s �z RegexLexerMeta._process_tokenc C s t |t�rd|dkrdS ||v r$|fS |dkr0|S |dd� dkrRt|dd� � S dsbJ d| ��n�t |t�r�d | j }| jd 7 _g }|D ],}||ks�J d | ��|�| �|||�� q�|||< |fS t |t��r |D ] }||v s�|d v s�J d | ��q�|S d�sJ d| ��dS )z=Preprocess the state transition action of a token definition.�#pop����#pushN� z#pop:Fzunknown new state %rz_tmp_%dr~ zcircular state ref %r)r� r� zunknown new state zunknown new state def %r)rB rI �intro �_tmpname�extend�_process_staterp )rq � new_state� unprocessed� processedZ tmp_state�itokensZistater r r# �_process_new_state s< � ��z!RegexLexerMeta._process_new_statec C s� t |�tu sJ d| ��|d dks0J d| ��||v r@|| S g }||< | j}|| D �]0}t|t�r�||ks~J d| ��|�| �||t|��� qZt|t�r�qZt|t�r�| � |j ||�}|� t � d�jd|f� qZt |�tu s�J d| ��z| �|d ||�}W n@ t�yF } z&td |d || | f �| �W Y d} ~ n d} ~ 0 0 | �|d �} t|�d k�rjd}n| � |d ||�}|� || |f� qZ|S ) z%Preprocess a single state definition.zwrong state name %rr �#zinvalid state name %rzcircular state reference %rri Nzwrong rule def %rz+uncompilable regex %r in state %r of %r: %sr~ � )r( rI �flagsrB r r� r� rn r r� r� rC r� r� r� rp r� � Exception� ValueErrorr� rM ) rq r� r� r� �tokensr� Ztdefr� �rex�errr� r r r# r� % sH � �� �z RegexLexerMeta._process_stateNc C s< i }| j |< |p| j| }t|�D ]}| �|||� q$|S )z-Preprocess a dictionary of token definitions.)� _all_tokensr� r� r� )rq r* � tokendefsr� r� r r r# �process_tokendefQ s z RegexLexerMeta.process_tokendefc C s� i }i }| j D ]�}|j�di �}|�� D ]�\}}|�|�}|du rz|||< z|�t�}W n tyn Y q(Y n0 |||< q(|�|d�}|du r�q(||||d �< z|�t�} W n ty� Y q(0 || ||< q(q|S )a Merge tokens from superclasses in MRO order, returning a single tokendef dictionary. Any state that is not defined by a subclass will be inherited automatically. States that *are* defined by subclasses will, by default, override that state in the superclass. If a subclass wishes to inherit definitions from a superclass, it can use the special value "inherit", which will cause the superclass' state definition to be included at that point in the state. r� Nr~ )�__mro__�__dict__r: �items�indexr r� r� ) rq r� � inheritable�cZtoksr� r� ZcuritemsZ inherit_ndxZ new_inh_ndxr r r# � get_tokendefsY s0 z RegexLexerMeta.get_tokendefsc O sR d| j vr:i | _d| _t| d�r(| jr(n| �d| �� �| _tj | g|�R i |��S )z:Instantiate cls after preprocessing its token definitions.�_tokensr �token_variantsri ) r� r� r� �hasattrr� r� r� r� r( �__call__)rq rr �kwdsr r r# r� � s zRegexLexerMeta.__call__)N) r- r. r/ r0 r� r� r� r� r� r� r� r r r r# r� � s #, 1r� c @ s$ e Zd ZdZejZi Zddd�ZdS )r z� Base for simple stateful regular expression-based lexers. Simplifies the lexing process so that you need only provide a list of states and regular expressions. �r� c c s� d}| j }t|�}||d }|D �](\}}} |||�} | r"|durrt|�tu rb||| �� fV n|| | �E dH | �� }| du�rHt| t�r�| D ]D} | dkr�t|�dkr�|� � q�| dkr�|� |d � q�|� | � q�nbt| t ��rt | �t|�k�r|dd�= n || d�= n,| dk�r*|� |d � nd�s<J d| ��||d } q q"zP|| d k�r�d g}|d }|t d fV |d7 }W q |t|| fV |d7 }W q t�y� Y �q�Y q 0 q dS ) z~ Split ``text`` into (tokentype, text) pairs. ``stack`` is the initial stack (default: ``['root']``) r r� Nr� r~ r� F�wrong state def: %rrG r� )r� r� r( r r{ ry rB rp rM r� rC r� �absr r rz ) r< rD r� r� r� Z statestack� statetokens�rexmatchr� r� �mr� r r r# rW � sR z!RegexLexer.get_tokens_unprocessedN)r� ) r- r. r/ r0 r� � MULTILINEr� r� rW r r r r# r � s r c @ s" e Zd ZdZddd�Zdd� ZdS )r z9 A helper object that holds lexer position data. Nc C s* || _ || _|pt|�| _|p"dg| _d S )Nr� )rD r� rM ry r� )r< rD r� r� ry r r r# r> � s zLexerContext.__init__c C s d| j | j| jf S )NzLexerContext(%r, %r, %r))rD r� r� r@ r r r# rA s �zLexerContext.__repr__)NN)r- r. r/ r0 r> rA r r r r# r � s r c @ s e Zd ZdZddd�ZdS )r zE A RegexLexer that uses a context object to store its state. Nc c s: | j }|s t|d�}|d }n|}||jd }|j}|D �]`\}}}|||j|j�} | r:|dur�t|�tu r�|j|| �� fV | �� |_n$|| | |�E dH |s�||jd }|du�r�t |t ��r |D ]P} | dkr�t |j�dkr�|j� � q�| dk�r|j� |jd � q�|j� | � q�nlt |t��rZt|�t |j�k�rL|jdd�= n |j|d�= n0|dk�rx|j� |jd � nd�s�J d | ��||jd } q6q:zz|j|jk�r�W �q6||j d k�r�dg|_|d }|jtd fV | jd7 _W q6|jt||j fV | jd7 _W q6 t�y2 Y �q6Y q60 q6dS ) z Split ``text`` into (tokentype, text) pairs. If ``context`` is given, use this lexer context instead. r r� r� Nr� r~ r� Fr� rG )r� r r� rD r� ry r( r r{ rB rp rM r� rC r� r� r r rz ) r< rD �contextr� r� r� r� r� r� r� r� r r r# rW s` z)ExtendedRegexLexer.get_tokens_unprocessed)NN)r- r. r/ r0 rW r r r r# r s r c c s� t | �} zt| �\}}W n ty6 |E dH Y dS 0 d}d}|D ]�\}}}|du rZ|}d} |�r|t|� |k�r|| || � } | r�||| fV |t| �7 }|D ]"\} } } || | fV |t| �7 }q�|| } zt| �\}}W q^ t�y d}Y �qY q^0 q^| t|�k rD|||| d� fV |t|�| 7 }qD|�r�|�pDd}|D ]$\}}}|||fV |t|�7 }�qJzt| �\}}W n t�y� d}Y �q�Y n0 �q6dS )ag Helper for lexers which must combine the results of several sublexers. ``insertions`` is a list of ``(index, itokens)`` pairs. Each ``itokens`` iterable should be inserted at position ``index`` into the token stream given by the ``tokens`` argument. The result is a combined token stream. TODO: clean up the code here. NTr F)�iter�next� StopIterationrM )rk r� r� r� ZrealposZinsleftrl rX rY ZoldiZtmpvalZit_indexZit_tokenZit_value�pr r r# rj Q sN rj c @ s e Zd ZdZdd� ZdS )�ProfilingRegexLexerMetaz>Metaclass for ProfilingRegexLexer, collects regex timing info.c sL t |t�r t|j|j|jd��n|�t��|��tjf� ���fdd� }|S )Nr� c s` � j d ���fddg�}t�� }��| ||�}t�� }|d d7 < |d || 7 < |S )Nr� r r r~ )� _prof_data� setdefault�timer� )rD r� �endpos�info�t0�res�t1�rq �compiledr� r� r r# � match_func� s z:ProfilingRegexLexerMeta._process_regex.<locals>.match_func) rB r r r� r� r� r� �sys�maxsize)rq r� r� r� r� r r� r# r� � s � z&ProfilingRegexLexerMeta._process_regexN)r- r. r/ r0 r� r r r r# r� � s r� c @ s" e Zd ZdZg ZdZddd�ZdS )�ProfilingRegexLexerzFDrop-in replacement for RegexLexer that does profiling of its regexes.� r� c # s� � j j�i � t�� ||�E d H � j j�� }tdd� |�� D �� fdd�dd�}tdd� |D ��}t � t d� j j t |�|f � t d � t d d � t d � |D ]}t d | � q�t d � d S )Nc s sN | ]F\\}}\}}|t |��d ��dd�dd� |d| d| | fV qdS )zu'z\\�\N�A i� )�reprrO rF )�.0r� �r�nrX r r r# � <genexpr>� s � �z=ProfilingRegexLexer.get_tokens_unprocessed.<locals>.<genexpr>c s | � j S rV )�_prof_sort_indexr! r@ r r# r$ � r% z<ProfilingRegexLexer.get_tokens_unprocessed.<locals>.<lambda>T)�key�reversec s s | ]}|d V qdS )� Nr )r� r" r r r# r� � r% z2Profiling result for %s lexing %d chars in %.3f mszn==============================================================================================================z$%-20s %-64s ncalls tottime percall)r� r� zn--------------------------------------------------------------------------------------------------------------z%-20s %-65s %5d %8.4f %8.4f) r? r� rC r rW r� �sortedr� �sum�printr- rM )r< rD r� Zrawdatar� Z sum_totalr, r r@ r# rW � s( � �� z*ProfilingRegexLexer.get_tokens_unprocessedN)r� )r- r. r/ r0 r� r� rW r r r r# r� � s r� )6r0 r� r� r� Zpip._vendor.pygments.filterr r Z pip._vendor.pygments.filtersr �pip._vendor.pygments.tokenr r r r r �pip._vendor.pygments.utilr r r r r r Z pip._vendor.pygments.regexoptr �__all__r� r rK � staticmethodZ_default_analyser( r& r r rI r rn r rp ro rs r r� r r r r r� r r r rj r� r� r r r r# �<module> sF o' 2 )aH@