⚝
One Hat Cyber Team
⚝
Your IP:
216.73.216.67
Server IP:
50.6.168.112
Server:
Linux server-617809.webnetzimbabwe.com 5.14.0-570.25.1.el9_6.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Jul 9 04:57:09 EDT 2025 x86_64
Server Software:
Apache
PHP Version:
8.4.10
Buat File
|
Buat Folder
Eksekusi
Dir :
~
/
usr
/
lib64
/
python3.9
/
urllib
/
__pycache__
/
View File Name :
robotparser.cpython-39.pyc
a ÕDOgÐ$ ã @ s\ d Z ddlZddlZddlZdgZe dd¡ZG dd„ dƒZG dd„ dƒZ G d d „ d ƒZ dS ) a% robotparser.py Copyright (C) 2000 Bastian Kleineidam You can choose between two licenses when using this package: 1) GNU GPLv2 2) PSF license for Python 2.2 The robots.txt Exclusion Protocol is implemented as specified in http://www.robotstxt.org/norobots-rfc.txt é NÚRobotFileParserÚ RequestRatezrequests secondsc @ sr e Zd ZdZd dd„Zdd„ Zdd„ Zd d „ Zd d „ Zd d„ Z dd„ Z dd„ Z dd„ Z dd„ Z dd„ Zdd„ ZdS ) r zs This class provides a set of methods to read, parse and answer questions about a single robots.txt file. Ú c C s2 g | _ g | _d | _d| _d| _| |¡ d| _d S )NFr )ÚentriesÚsitemapsÚ default_entryÚ disallow_allÚ allow_allÚset_urlÚ last_checked©ÚselfÚurl© r ú*/usr/lib64/python3.9/urllib/robotparser.pyÚ__init__ s zRobotFileParser.__init__c C s | j S )z·Returns the time the robots.txt file was last fetched. This is useful for long-running web spiders that need to check for new robots.txt files periodically. )r ©r r r r Úmtime% s zRobotFileParser.mtimec C s ddl }| ¡ | _dS )zYSets the time the robots.txt file was last fetched to the current time. r N)Útimer )r r r r r Úmodified. s zRobotFileParser.modifiedc C s& || _ tj |¡dd… \| _| _dS )z,Sets the URL referring to a robots.txt file.é é N)r ÚurllibÚparseÚurlparseÚhostÚpathr r r r r 6 s zRobotFileParser.set_urlc C sˆ zt j | j¡}W nT t jjyf } z8|jdv r8d| _n|jdkrR|jdk rRd| _W Y d}~n&d}~0 0 | ¡ }| | d¡ ¡ ¡ dS )z4Reads the robots.txt URL and feeds it to the parser.)i‘ i“ Ti iô Nzutf-8) r ZrequestÚurlopenr ÚerrorÚ HTTPErrorÚcoder r Úreadr ÚdecodeÚ splitlines)r ÚfÚerrÚrawr r r r! ; s zRobotFileParser.readc C s, d|j v r | jd u r(|| _n | j |¡ d S ©NÚ*)Ú useragentsr r Úappend)r Úentryr r r Ú _add_entryH s zRobotFileParser._add_entryc C sP d}t ƒ }| ¡ |D ] }|sP|dkr4t ƒ }d}n |dkrP| |¡ t ƒ }d}| d¡}|dkrn|d|… }| ¡ }|s|q| dd¡}t|ƒdkr|d ¡ ¡ |d< tj |d ¡ ¡|d< |d dkrú|dkrä| |¡ t ƒ }|j |d ¡ d}q|d dkr.|dkr6|j t|d d ƒ¡ d}q|d d krb|dkr6|j t|d d ƒ¡ d}q|d d kr |dkr6|d ¡ ¡ ršt|d ƒ|_d}q|d d kr|dkr6|d d¡}t|ƒdkr|d ¡ ¡ r|d ¡ ¡ rtt|d ƒt|d ƒƒ|_d}q|d dkr| j |d ¡ q|dkrL| |¡ dS )z”Parse the input lines from a robots.txt file. We allow that a user-agent: line is not preceded by one or more blank lines. r r é ú#Nú:z user-agentZdisallowFZallowTz crawl-delayz request-rateú/Zsitemap)ÚEntryr r, ÚfindÚstripÚsplitÚlenÚlowerr r Úunquoter) r* Ú rulelinesÚRuleLineÚisdigitÚintÚdelayr Úreq_rater )r ÚlinesÚstater+ ÚlineÚiZnumbersr r r r Q sj ÿ zRobotFileParser.parsec C s | j r dS | jrdS | js dS tj tj |¡¡}tj dd|j|j |j |j f¡}tj |¡}|sfd}| j D ] }| |¡rl| |¡ S ql| jrœ| j |¡S dS )z=using the parsed robots.txt decide if useragent can fetch urlFTr r0 )r r r r r r r7 Ú urlunparser ÚparamsÚqueryÚfragmentÚquoter Ú applies_toÚ allowancer )r Ú useragentr Z parsed_urlr+ r r r Ú can_fetchš s& ÿ zRobotFileParser.can_fetchc C s> | ¡ s d S | jD ]}| |¡r|j S q| jr:| jjS d S ©N)r r rG r< r ©r rI r+ r r r Ú crawl_delay· s zRobotFileParser.crawl_delayc C s> | ¡ s d S | jD ]}| |¡r|j S q| jr:| jjS d S rK )r r rG r= r rL r r r Ú request_rateÁ s z RobotFileParser.request_ratec C s | j s d S | j S rK )r r r r r Ú site_mapsË s zRobotFileParser.site_mapsc C s, | j }| jd ur || jg }d tt|ƒ¡S )Nz )r r ÚjoinÚmapÚstr)r r r r r Ú__str__Ð s zRobotFileParser.__str__N)r )Ú__name__Ú __module__Ú __qualname__Ú__doc__r r r r r! r, r rJ rM rN rO rS r r r r r s I c @ s( e Zd ZdZdd„ Zdd„ Zdd„ ZdS ) r9 zoA rule line is a single "Allow:" (allowance==True) or "Disallow:" (allowance==False) followed by a path.c C s< |dkr|sd}t j t j |¡¡}t j |¡| _|| _d S )Nr T)r r rB r rF r rH )r r rH r r r r Ú s zRuleLine.__init__c C s | j dkp| | j ¡S r' )r Ú startswith)r Úfilenamer r r rG â s zRuleLine.applies_toc C s | j r dndd | j S )NZAllowZDisallowz: )rH r r r r r rS å s zRuleLine.__str__N)rT rU rV rW r rG rS r r r r r9 × s r9 c @ s0 e Zd ZdZdd„ Zdd„ Zdd„ Zdd „ Zd S ) r1 z?An entry has one or more user-agents and zero or more rulelinesc C s g | _ g | _d | _d | _d S rK )r) r8 r< r= r r r r r ë s zEntry.__init__c C s‚ g }| j D ]}| d|› ¡ q | jd ur<| d| j› ¡ | jd urf| j}| d|j› d|j› ¡ | tt| j ƒ¡ d |¡S )Nz User-agent: z Crawl-delay: zRequest-rate: r0 Ú ) r) r* r< r= ZrequestsZsecondsÚextendrQ rR r8 rP )r ÚretÚagentZrater r r rS ñ s z Entry.__str__c C sF | d¡d ¡ }| jD ](}|dkr* dS | ¡ }||v r dS qdS )z2check if this entry applies to the specified agentr0 r r( TF)r4 r6 r) )r rI r] r r r rG ý s zEntry.applies_toc C s$ | j D ]}| |¡r|j S qdS )zZPreconditions: - our agent applies to this entry - filename is URL decodedT)r8 rG rH )r rY r@ r r r rH s zEntry.allowanceN)rT rU rV rW r rS rG rH r r r r r1 é s r1 ) rW Ú collectionsÚ urllib.parser Zurllib.requestÚ__all__Ú namedtupler r r9 r1 r r r r Ú<module> s B