z^%sz[%s]+N) charsUntilRegExKeyErrorr3 recompilerl r% rP rO endr0 r[ ) r! Z
charactersZoppositecharsZregexr4 mrr rr r r
charsUntil@ s.
z!HTMLUnicodeInputStream.charsUntilc C s@ |d k r<| j dkr.|| j | _| jd7 _n| j d8 _ d S )Nr r )rP r% rO )r! r\ r r r ungeto s
z HTMLUnicodeInputStream.unget)N)F)r5 r6 r7 r8 r_ r" rM rK rX r r\ r[ rG rH rv rw r r r r r? s
&
/r? c @ sL e Zd ZdZdddZdd Zd d
Zdd d Zd
d Zdd Z dd Z
dS )r@ zProvides a unicode stream of characters to the HTMLTokenizer.
This class takes care of character encoding and removing or replacing
incorrect byte-sequences and also provides column and line tracking.
N windows-1252Tc C s\ | j || _tj| | j d| _d| _|| _|| _|| _|| _ || _
| j || _ | j
dS )a Initialises the HTMLInputStream.
HTMLInputStream(source, [encoding]) -> Normalized stream from source
for use by html5lib.
source can be either a file-object, local filename or a string.
The optional encoding parameter must be a string that indicates
the encoding. If specified, that encoding will be used,
regardless of any BOM or later declaration (such as in a meta
element)
i d N)rK rawStreamr? r" numBytesMetanumBytesChardetoverride_encodingtransport_encodingsame_origin_parent_encodinglikely_encodingdefault_encodingdetermineEncodingrJ rM )r! rA r} r~ r r r Z
useChardetr r r r" s z HTMLBinaryInputStream.__init__c C s& | j d jj| jd| _tj| d S )Nr ra )rJ Z
codec_info streamreaderrz rL r? rM )r! r r r rM s zHTMLBinaryInputStream.resetc C sD t |dr|}nt|}y|j|j W n t|}Y nX |S )zvProduces a file object from source.
source can be either a file object, local filename or a string.
r- )r= r r) r& r )r! rA r r r r rK s
z HTMLBinaryInputStream.openStreamc C s | j df}|d d k r |S t| jdf}|d d k r:|S t| jdf}|d d k rX|S | j df}|d d k rt|S t| jdf}|d d k r|d jjd r|S t| jdf}|d d k r|S |rdyddl m
} W n t k
r Y nxX g }| }x6|j s.| j
j| j}|sP |j| |j| qW |j t|jd }| j
jd |d k rd|dfS t| jdf}|d d k r|S tddfS )NrD r Z tentativezutf-16)UniversalDetectorencodingz windows-1252) detectBOMrI r} r~ detectEncodingMetar name
startswithr Zchardet.universaldetectorr ImportErrordonerz r- r| r0 Zfeedcloseresultr) r )r! ZchardetrJ r ZbuffersZdetectorr r r r r r sP
z'HTMLBinaryInputStream.determineEncodingc C s t |}|d krd S |jdkr(t d}nT|| jd krH| jd df| _n4| jjd |df| _| j td| jd |f d S )Nutf-16beutf-16lezutf-8r rD z Encoding changed from %s to %s)r r )rI r rJ rz r) rM r )r! Z newEncodingr r r changeEncoding s
z$HTMLBinaryInputStream.changeEncodingc
C s t jdt jdt jdt jdt jdi}| jjd}|j|dd }d}|sp|j|}d}|sp|j|dd }d }|r| jj | t
|S | jj d
dS dS ) zAttempts to detect at BOM at the start of the stream. If
an encoding can be determined from the BOM return the name of the
encoding otherwise return Nonezutf-8zutf-16lezutf-16bezutf-32lezutf-32be N rf r ) codecsBOM_UTF8 BOM_UTF16_LE BOM_UTF16_BE BOM_UTF32_LE BOM_UTF32_BErz r- getr) rI )r! ZbomDictstringr r) r r r r s"
z HTMLBinaryInputStream.detectBOMc C sH | j j| j}t|}| j jd |j }|dk rD|jdkrDtd}|S )z9Report the encoding declared by the meta element
r Nutf-16beutf-16lezutf-8)r r )rz r- r{ EncodingParserr) getEncodingr rI )r! r parserr r r r r 9 s z(HTMLBinaryInputStream.detectEncodingMeta)NNNNrx T)T) r5 r6 r7 r8 r" rM rK r r r r r r r r r@ s
(
>"r@ c @ s e Zd ZdZdd Zdd Zdd Zdd Zd
d Zd d
Z dd Z
dd Z e e e
Z
dd Ze eZefddZdd Zdd Zdd Zd S )
EncodingByteszString-like object with an associated position and various extra methods
If the position is ever greater than the string length then an exception is
raisedc C s t j| |j S )N)r, __new__lower)r! valuer r r r L s zEncodingBytes.__new__c C s
d| _ d S )Nr r )rX )r! r r r r r" P s zEncodingBytes.__init__c C s | S )Nr )r! r r r __iter__T s zEncodingBytes.__iter__c C s> | j d }| _ |t| kr"tn |dk r.t| ||d S )Nr r )rX r#
StopIterationr> )r! pr r r __next__W s zEncodingBytes.__next__c C s | j S )N)r )r! r r r next_ s zEncodingBytes.nextc C sB | j }|t| krtn |dk r$t|d | _ }| ||d S )Nr r )rX r# r r> )r! r r r r previousc s zEncodingBytes.previousc C s | j t| krt|| _ d S )N)rX r# r )r! r r r r setPositionl s zEncodingBytes.setPositionc C s* | j t| krt| j dkr"| j S d S d S )Nr )rX r# r )r! r r r getPositionq s
zEncodingBytes.getPositionc C s | | j | j d S )Nr )r )r! r r r getCurrentByte{ s z EncodingBytes.getCurrentBytec C sL | j }x:|t| k r@| ||d }||kr6|| _|S |d7 }qW || _dS )z Skip past a list of charactersr N)r r# rX )r! rs r rm r r r rk s zEncodingBytes.skipc C sL | j }x:|t| k r@| ||d }||kr6|| _|S |d7 }qW || _d S )Nr )r r# rX )r! rs r rm r r r skipUntil s zEncodingBytes.skipUntilc C s> | j }| ||t| }|j|}|r:| j t|7 _ |S )zLook for a sequence of bytes at the start of a string. If the bytes
are found return True and advance the position to the byte after the
match. Otherwise return False and leave the position alone)r r# r )r! r, r r1 r4 r r r
matchBytes s
zEncodingBytes.matchBytesc C sR | | j d j|}|dkrJ| jdkr,d| _| j|t| d 7 _dS tdS )zLook for the next sequence of bytes matching a given sequence. If
a match is found advance the position to the last byte of the matchNr r Tr r )r findrX r# r )r! r, Z newPositionr r r jumpTo s
zEncodingBytes.jumpToN)r5 r6 r7 r8 r r" r r r r r r propertyr r currentBytespaceCharactersBytesrk r r r r r r r r H s
r c @ sX e Zd ZdZdd Zdd Zdd Zdd Zd
d Zd d
Z dd Z
dd Z dd Z dS )r z?Mini parser for detecting character encoding from meta elementsc C s t || _d| _dS )z3string - the data to work on for encoding detectionN)r r1 r )r! r1 r r r r" s
zEncodingParser.__init__c
C s d| j fd| jfd| jfd| jfd| jfd| jff}x^| jD ]T}d}xD|D ]<\}}| jj|rJy | }P W qJ tk
r d}P Y qJX qJW |s)r1 r )r! r r r r s z EncodingParser.handleCommentc C s | j jtkrdS d}d }x| j }|d kr.dS |d dkr^|d dk}|r|d k r|| _dS q|d dkr|d }t|}|d k r|| _dS q|d dkrtt|d }|j }|d k rt|}|d k r|r|| _dS |}qW d S ) NTFr s
http-equivr s content-types charsets content) r1 r r getAttributer rI ContentAttrParserr parse)r! Z hasPragmaZpendingEncodingattrZtentativeEncodingcodecZ
contentParserr r r r s: zEncodingParser.handleMetac C s
| j dS )NF)handlePossibleTag)r! r r r r s z%EncodingParser.handlePossibleStartTagc C s t | j | jdS )NT)r r1 r )r! r r r r s
z#EncodingParser.handlePossibleEndTagc C sf | j }|jtkr(|r$|j | j dS |jt}|dkrD|j n | j }x|d k r`| j }qNW dS )NTr )r1 r asciiLettersBytesr r r spacesAngleBracketsr )r! ZendTagr1 rm r r r r r s
z EncodingParser.handlePossibleTagc C s | j jdS )Nr )r1 r )r! r r r r s zEncodingParser.handleOtherc C s | j }|jttdgB }|dkr&dS g }g }xt|dkr@|r@P nX|tkrT|j }P nD|d krjdj|dfS |tkr|j|j n|dkrdS |j| t|}q0W |dkr|j dj|dfS t| |j }|d
kr:|}xt|}||krt| dj|dj|fS |tkr*|j|j q|j| qW nJ|dkrRdj|dfS |tkrl|j|j n|dkrzdS |j| x^t|}|t
krdj|dj|fS |tkr|j|j n|dkrdS |j| qW dS ) z_Return a name,value pair for the next attribute in the stream,
if one is found, or None /r N =r2 ' ")r N)r r )r r ) r1 rk r frozensetr3 asciiUppercaseBytesr0 r r r r )r! r1 rm ZattrNameZ attrValueZ quoteCharr r r r sf
zEncodingParser.getAttributeN)
r5 r6 r7 r8 r" r r r r r r r r r r r r r s $r c @ s e Zd Zdd Zdd ZdS )r c C s
|| _ d S )N)r1 )r! r1 r r r r" f s zContentAttrParser.__init__c C s y| j jd | j jd7 _| j j | j jdks8d S | j jd7 _| j j | j jdkr| j j}| j jd7 _| j j}| j j|r| j || j j S d S nF| j j}y | j jt | j || j j S tk
r | j |d S X W n tk
r d S X d S )Ns charsetr r r r )r r )r1 r r rk r r r r )r! Z quoteMarkZ oldPositionr r r r j s.
zContentAttrParser.parseN)r5 r6 r7 r" r r r r r r e s r c C s` t | tr.y| jd} W n tk
r, dS X | dk rXy
tj| S tk
rT dS X ndS dS )z{Return the python codec name corresponding to an encoding or None if the
string doesn't correspond to a valid encoding.r N)r; r decodeUnicodeDecodeError webencodingslookupAttributeError)r r r r rI s
rI r )1Z
__future__r r r Zsixr r Z six.movesr r r rp r Z constantsr
r r r
r rN r ior r r r r r r r Z invalid_unicode_no_surrogaterE rq evalrc setrj Zascii_punctuation_rern objectr rC r? r@ r, r r r rI r r r r sV
J g Ih 6'