OwlCyberSecurity - MANAGER
Edit File: collector.cpython-39.pyc
a )��ex@ � @ s� d Z ddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl m Z ddlmZ ddlmZmZmZmZmZmZmZmZmZmZmZ ddl m Z ddl m Z ddl m!Z!m"Z" dd l#m$Z$ dd l%m&Z& dd l'm(Z( dd l)m*Z* dd l+m,Z, ddl-m.Z. ddl/m0Z0 ddl1m2Z2 ddl3m4Z4m5Z5m6Z6 e�rHddlm7Z7 ne8Z7e�9e:�Z;ee<e<f Z=e<ee< d�dd�Z>G dd� de?�Z@e dd�dd�ZAG d d � d e?�ZBe<e*dd �d d �ZCe<e*e d �d!d"�ZDe=ee< d#�d$d%�ZEG d&d'� d'�ZFG d(d)� d)e7�ZGeGeGd*�d+d,�ZHeHd-ee& d.�d/d0��ZIG d1d-� d-�ZJG d2d3� d3e �ZKdCe&ee<e?f eed4 dd5�d6d7�ZLdDe eMeJd9�d:d;�ZNe&e*ed- d<�d=d>�ZOG d?d@� d@e�ZPG dAdB� dB�ZQdS )EzO The main purpose of this module is to expose LinkCollector.collect_sources(). � N)� HTMLParser)�Values) � TYPE_CHECKING�Callable�Dict�Iterable�List�MutableMapping� NamedTuple�Optional�Sequence�Tuple�Union)�requests)�Response)� RetryError�SSLError)�NetworkConnectionError)�Link)� SearchScope)� PipSession)�raise_for_status)�is_archive_file��redact_auth_from_url)�vcs� )�CandidatesFromPage� LinkSource� build_source)�Protocol��url�returnc C s6 t jD ]*}| �� �|�r| t|� dv r| S qdS )zgLook for VCS schemes in the URL. Returns the matched VCS scheme, or None if there's no match. z+:N)r Zschemes�lower� startswith�len)r" �scheme� r( �^/home/gouroczh/virtualenv/pat/3.9/lib/python3.9/site-packages/pip/_internal/index/collector.py�_match_vcs_scheme7 s r* c s& e Zd Zeedd�� fdd� Z� ZS )�_NotAPIContentN)� content_type� request_descr# c s t � �||� || _|| _d S �N)�super�__init__r, r- )�selfr, r- �� __class__r( r) r0 C s z_NotAPIContent.__init__)�__name__� __module__� __qualname__�strr0 � __classcell__r( r( r2 r) r+ B s r+ )�responser# c C s6 | j �dd�}|�� }|�d�r$dS t|| jj��dS )z� Check the Content-Type header to ensure the response contains a Simple API Response. Raises `_NotAPIContent` if the content type is not a valid content-type. � Content-Type�Unknown)z text/htmlz#application/vnd.pypi.simple.v1+html�#application/vnd.pypi.simple.v1+jsonN)�headers�getr$ r% r+ �request�method)r9 r, �content_type_lr( r( r) �_ensure_api_headerI s �rB c @ s e Zd ZdS )�_NotHTTPN)r4 r5 r6 r( r( r( r) rC _ s rC )r" �sessionr# c C sF t j�| �\}}}}}|dvr$t� �|j| dd�}t|� t|� dS )z� Send a HEAD request to the URL, and ensure the response contains a simple API Response. Raises `_NotHTTP` if the URL is not available for a HEAD request, or `_NotAPIContent` if the content type is not a valid content type. > �https�httpT)�allow_redirectsN)�urllib�parse�urlsplitrC �headr rB )r" rD r' �netloc�path�query�fragment�respr( r( r) �_ensure_api_responsec s rQ c C sx t t| �j�rt| |d� t�dt| �� |j| d�g d��dd�d�}t |� t |� t�dt| �|j �d d �� |S ) aY Access an Simple API response with GET, and return the response. This consists of three parts: 1. If the URL looks suspiciously like an archive, send a HEAD first to check the Content-Type is HTML or Simple API, to avoid downloading a large file. Raise `_NotHTTP` if the content type cannot be determined, or `_NotAPIContent` if it is not HTML or a Simple API. 2. Actually perform the request. Raise HTTP exceptions on network failures. 3. Check the Content-Type header to make sure we got a Simple API response, and raise `_NotAPIContent` otherwise. �rD zGetting page %sz, )r< z*application/vnd.pypi.simple.v1+html; q=0.1ztext/html; q=0.01z max-age=0)�Acceptz Cache-Control)r= zFetched page %s as %sr: r; ) r r �filenamerQ �logger�debugr r> �joinr rB r= )r" rD rP r( r( r) �_get_simple_responseu s& ��� �rX )r= r# c C s< | r8d| v r8t j�� }| d |d< |�d�}|r8t|�S dS )z=Determine if we have any encoding information in our headers.r: z content-type�charsetN)�email�message�Message� get_paramr7 )r= �mrY r( r( r) �_get_encoding_from_headers� s r_ c @ s: e Zd Zddd�dd�Zeed�dd�Zed �d d �ZdS ) �CacheablePageContent� IndexContentN��pager# c C s |j s J �|| _d S r. )�cache_link_parsingrc �r1 rc r( r( r) r0 � s z CacheablePageContent.__init__)�otherr# c C s t |t| ��o | jj|jjkS r. )� isinstance�typerc r" )r1 rf r( r( r) �__eq__� s zCacheablePageContent.__eq__�r# c C s t | jj�S r. )�hashrc r" �r1 r( r( r) �__hash__� s z CacheablePageContent.__hash__) r4 r5 r6 r0 �object�boolri �intrm r( r( r( r) r` � s r` c @ s e Zd Zdee d�dd�ZdS )� ParseLinksra rb c C s d S r. r( re r( r( r) �__call__� s zParseLinks.__call__N)r4 r5 r6 r r rr r( r( r( r) rq � s rq )�fnr# c sL t jdd�ttt d�� fdd� ��t �� �dtt d�� �fdd � �}|S ) z� Given a function that parses an Iterable[Link] from an IndexContent, cache the function's result (keyed by CacheablePageContent), unless the IndexContent `page` has `page.cache_link_parsing == False`. N)�maxsize)�cacheable_pager# c s t � | j��S r. )�listrc )ru )rs r( r) �wrapper� s z*with_cached_index_content.<locals>.wrapperra rb c s | j r�t| ��S t� | ��S r. )rd r` rv )rc �rs rw r( r) �wrapper_wrapper� s z2with_cached_index_content.<locals>.wrapper_wrapper)� functools� lru_cacher` r r �wraps)rs ry r( rx r) �with_cached_index_content� s r} ra rb c c s� | j �� }|�d�rTt�| j�}|�dg �D ]"}t�|| j �}|du rHq,|V q,dS t | j �}| j pfd}|� | j� |�� | j }|jp�|}|jD ]$} tj| ||d�}|du r�q�|V q�dS )z\ Parse a Simple API's Index Content, and yield its anchor elements as Link objects. r< �filesNzutf-8)�page_url�base_url)r, r$ r% �json�loads�contentr> r � from_jsonr" �HTMLLinkParser�encoding�feed�decoder� �anchors� from_element) rc rA �data�file�link�parserr� r" r� �anchorr( r( r) � parse_links� s& r� c @ s< e Zd ZdZd eeee eedd�dd�Zed�dd �Z dS ) ra z5Represents one response (or page), along with its URLTN)r� r, r� r" rd r# c C s" || _ || _|| _|| _|| _dS )am :param encoding: the encoding to decode the given content. :param url: the URL from which the HTML was downloaded. :param cache_link_parsing: whether links parsed from this page's url should be cached. PyPI index urls should have this set to False, for example. N)r� r, r� r" rd )r1 r� r, r� r" rd r( r( r) r0 s zIndexContent.__init__rj c C s t | j�S r. )r r" rl r( r( r) �__str__ s zIndexContent.__str__)T) r4 r5 r6 �__doc__�bytesr7 r ro r0 r� r( r( r( r) ra s �� c sn e Zd ZdZedd�� fdd� Zeeeeee f dd�dd�Z eeeee f ee d �d d �Z � Z S ) r� zf HTMLParser that keeps the first base HREF and a list of all anchor elements' attributes. Nr! c s$ t � jdd� || _d | _g | _d S )NT)Zconvert_charrefs)r/ r0 r" r� r� )r1 r" r2 r( r) r0 # s zHTMLLinkParser.__init__)�tag�attrsr# c C sH |dkr,| j d u r,| �|�}|d urD|| _ n|dkrD| j�t|�� d S )N�base�a)r� �get_hrefr� �append�dict)r1 r� r� �hrefr( r( r) �handle_starttag* s z HTMLLinkParser.handle_starttag)r� r# c C s" |D ]\}}|dkr| S qd S )Nr� r( )r1 r� �name�valuer( r( r) r� 2 s zHTMLLinkParser.get_href) r4 r5 r6 r� r7 r0 r r r r� r� r8 r( r( r2 r) r� s "r� ).N)r� �reason�methr# c C s |d u rt j}|d| |� d S )Nz%Could not fetch URL %s: %s - skipping)rU rV )r� r� r� r( r( r) �_handle_get_simple_fail9 s r� T)r9 rd r# c C s& t | j�}t| j| jd || j|d�S )Nr: )r� r" rd )r_ r= ra r� r" )r9 rd r� r( r( r) �_make_index_contentC s �r� )r� rD r# c C s | j �dd�d }t|�}|r0t�d|| � d S tj�|�\}}}}}}|dkr�tj � tj � |��r�|� d�sv|d7 }tj�|d�}t�d|� zt||d �}W �nL ty� t�d | � Y �n< ty� } z"t�d | |j|j� W Y d }~�n d }~0 t�y( } zt| |� W Y d }~n�d }~0 t�yV } zt| |� W Y d }~n�d }~0 t�y� } z,d } | t|�7 } t| | tjd � W Y d }~nld }~0 tj �y� } z t| d|� �� W Y d }~n6d }~0 tj �y� t| d� Y n0 t || j d�S d S )N�#r r zICannot look at %s URL %s because it does not support lookup as web pages.r� �/z index.htmlz# file: URL is directory, getting %srR z`Skipping page %s because it looks like an archive, and cannot be checked by a HTTP HEAD request.z�Skipping page %s because the %s request got Content-Type: %s. The only supported Content-Types are application/vnd.pypi.simple.v1+json, application/vnd.pypi.simple.v1+html, and text/htmlz4There was a problem confirming the ssl certificate: )r� zconnection error: z timed out)rd ) r" �splitr* rU �warningrH rI �urlparse�osrM �isdirr? � url2pathname�endswith�urljoinrV rX rC r+ r- r, r r� r r r7 �infor �ConnectionError�Timeoutr� rd ) r� rD r" Z vcs_schemer' �_rM rP �excr� r( r( r) �_get_index_contentP sV � � � $$r� c @ s. e Zd ZU eee ed< eee ed< dS )�CollectedSources� find_links� index_urlsN)r4 r5 r6 r r r �__annotations__r( r( r( r) r� � s r� c @ sx e Zd ZdZeedd�dd�Zedeee d d�dd ��Z e e e d �d d ��Zeee d �dd�Ze eed�dd�ZdS )� LinkCollectorz� Responsible for collecting Link objects from all configured locations, making network requests as needed. The class's main method is its collect_sources() method. N)rD � search_scoper# c C s || _ || _d S r. )r� rD )r1 rD r� r( r( r) r0 � s zLinkCollector.__init__F)rD �options�suppress_no_indexr# c C sd |j g|j }|jr8|s8t�dd�dd� |D ��� g }|jp@g }tj|||jd�}t ||d�}|S )z� :param session: The Session to use to make requests. :param suppress_no_index: Whether to ignore the --no-index option when constructing the SearchScope object. zIgnoring indexes: %s�,c s s | ]}t |�V qd S r. r )�.0r" r( r( r) � <genexpr>� � z'LinkCollector.create.<locals>.<genexpr>)r� r� �no_index)rD r� ) � index_url�extra_index_urlsr� rU rV rW r� r �creater� )�clsrD r� r� r� r� r� �link_collectorr( r( r) r� � s$ � ��zLinkCollector.createrj c C s | j jS r. )r� r� rl r( r( r) r� � s zLinkCollector.find_links)�locationr# c C s t || jd�S )z> Fetch an HTML page containing package links. rR )r� rD )r1 r� r( r( r) �fetch_response� s z LinkCollector.fetch_response)� project_name�candidates_from_pager# c s� t �� �fdd��j�|�D ���� }t �� �fdd��jD ���� }t�tj �r�dd� t � ||�D �}t |�� d|� d�g| }t� d�|�� tt|�t|�d �S ) Nc 3 s$ | ] }t |� �jjd d d�V qdS )F�r� Zpage_validatorZ expand_dirrd N�r rD Zis_secure_origin�r� �loc�r� r1 r( r) r� � s ��z0LinkCollector.collect_sources.<locals>.<genexpr>c 3 s$ | ] }t |� �jjd d d�V qdS )Tr� Nr� r� r� r( r) r� � s ��c S s* g | ]"}|d ur|j d urd|j � ��qS )Nz* )r� )r� �sr( r( r) � <listcomp>� s �z1LinkCollector.collect_sources.<locals>.<listcomp>z' location(s) to search for versions of �:� )r� r� )� collections� OrderedDictr� Zget_index_urls_locations�valuesr� rU � isEnabledFor�logging�DEBUG� itertools�chainr& rV rW r� rv )r1 r� r� Zindex_url_sourcesZfind_links_sources�linesr( r� r) �collect_sources� s* � � � ����z LinkCollector.collect_sources)F)r4 r5 r6 r� r r r0 � classmethodr ro r� �propertyr r7 r� r r ra r� r r� r� r( r( r( r) r� � s( � ��!�r� )N)T)Rr� r� � email.messagerZ rz r� r� r� r� � urllib.parserH �urllib.requestZ html.parserr �optparser �typingr r r r r r r r r r r � pip._vendorr Zpip._vendor.requestsr Z pip._vendor.requests.exceptionsr r �pip._internal.exceptionsr �pip._internal.models.linkr Z!pip._internal.models.search_scoper � pip._internal.network.sessionr Zpip._internal.network.utilsr � pip._internal.utils.filetypesr �pip._internal.utils.miscr �pip._internal.vcsr �sourcesr r r r rn � getLoggerr4 rU r7 ZResponseHeadersr* � Exceptionr+ rB rC rQ rX r_ r` rq r} r� ra r� r� ro r� r� r� r� r( r( r( r) �<module> st 4 ? � � �� =