tag, if the del looks like (DEL_START,
'Text
', DEL_END)r rE z']
>>> doc = ['
', 'Text', '
', '', 'More Text', '
']
>>> pre, post = doc[:3], doc[3:]
>>> pre, post
(['', 'Text', '
'], ['', 'More Text', '
'])
>>> locate_unbalanced_start(unbalanced_start, pre, post)
>>> pre, post
(['', 'Text', '
', ''], ['More Text', '
'])
As you can see, we moved the point so that the dangling that
we found will be effectively replaced by the div in the original
document. If this doesn't work out, we just throw away
unbalanced_start without doing anything.
r z<>rX r rY insN)r_ r rK r^ ra r3 )rH rV rW finding finding_namenextrZ r r r
rS i s&
rS c C s x| sP | d }|j d jd}|s(P |d }|tksD|jd rFP |j d jd}|dksh|dkrjP ||kr| j |jd|j qP qW dS )
zt like locate_unbalanced_start, except handling end tags and
possibly moving the point earlier in the document. r r z<>/z tag, which takes up visible space just like a word but
is only represented in a document by a tag. Nr c C s2 t j| dt|f |||d}||_||_||_|S )Nz%s: %s)r1 r0 r4 )r6 rl r5 r\ data html_repr)rm r\ rr rs r1 r0 r4 rn r r r
rl s ztag_token.__new__c C s d| j | j| j| j| j| jf S )NzRtag_token(%s, %s, html_repr=%s, post_tags=%r, pre_tags=%r, trailing_whitespace=%r))r\ rr rs r1 r0 r4 )rp r r r
ro s ztag_token.__repr__c C s | j S )N)rs )rp r r r
r7 s ztag_token.html)NNr )rL rM rN rQ rl ro r7 r r r r
rq s
rq c @ s e Zd ZdZdZdd ZdS )
href_tokenzh Represents the href in an anchor tag. Unlike other words, we only
show the href when it changes. Tc C s d| S )Nz Link: %sr )rp r r r
r7 s zhref_token.htmlN)rL rM rN rQ rD r7 r r r r
rt s rt Tc C s2 t j| r| }n t| dd}t|d|d}t|S )ak
Parse the given HTML and returns token objects (words with attached tags).
This parses only the content of a page; anything in the head is
ignored, and the and elements are themselves
optional. The content is then parsed by lxml, which ensures the
validity of the resulting parsed document (though lxml may make
incorrect guesses when the markup is particular bad).
and tags are also eliminated from the document, as
that gets confusing.
If include_hrefs is true, then the href attribute of tags is
included as a special kind of diffable token.T)cleanup)skip_tagr )r Z iselement
parse_html
flatten_el fixup_chunks)r7 r Zbody_elrU r r r
r s
r c C s |r t | } t| ddS )a
Parses an HTML fragment, returning an lxml element. Note that the HTML will be
wrapped in a
tag that was not in the original document.
If cleanup is true, make sure there's no or , and get
rid of any and tags.
T)Z
create_parent) cleanup_htmlr )r7 ru r r r
rw s rw z z
zc C sL t j| }|r | |j d } tj| }|r<| d|j } tjd| } | S )z This 'cleans' the HTML, meaning that any page structure is removed
(only the contents of are used, if there is any and tags are removed. Nr )_body_researchrd _end_body_rerc _ins_del_resub)r7 matchr r r
rz - s
rz z
[ \t\n\r]$c C s$ t | j }| d| | |d fS )zP
This function takes a word, such as 'test
' and returns ('test','
')
r N)rb rstrip)wordZstripped_lengthr r r
split_trailing_whitespace= s r c C s0 g }d}g }x| D ]}t |tr|d dkrf|d }t|d \}}td||||d}g }|j| q|d dkr|d }t||dd }g }|j| qt|rt|\}}t|||d }g }|j| qt|r|j| qt |r|r|j| n |j
j| qqW |s td
|d gS |d j
j | |S )
zM
This function takes a list of chunks and produces a list of tokens.
Nr imgr )rs r1 r4 hrefrE )r1 r4 r )r1 r/ )
isinstancetupler rq r3 rt is_wordr6 is_start_tag
is_end_tagr0 r? ) rU Z tag_accumZcur_wordr r] r- r\ r4 r r r r
ry E sF
ry paramr areabrbasefontinputbasemetalinkcoladdress
blockquotecenterdirdivdlfieldsetformh1h2h3h4h5h6hrisindexmenunoframesnoscriptolpr8 tableuldddtframesetlitbodytdtfootththeadtrc c s |s0| j dkr&d| jdt| fV n
t| V | j tkrX| j rXt| rX| j rXdS t| j}x|D ]}t|V qhW x(| D ] }xt ||dD ]
}|V qW qW | j dkr| jdr|rd| jdfV | st
| V t| j}x|D ]}t|V qW dS )a Takes an lxml element el, and generates all the text chunks for
that tag. Each start tag is a chunk, each word is a chunk, and each
end tag is a chunk.
If skip_tag is true, then the outermost container tag is
not returned (just its contents).r r- N)r r r ) r\ get start_tagr` r
rb tail split_wordsr rx end_tag)elr rv Z start_wordsr childitemZ end_wordsr r r
rx s&
$
rx z \S+(?:\s+|$)c C s" | s| j rg S tj| }|S )z_ Splits some text into words. Includes trailing whitespace
on each word when appropriate. )r split_words_refindall)r
Zwordsr r r
r s
r z
^[ \t\n\r]c C s$ d| j djdd | jj D f S )z=
The text representation of the start tag for a tag.
z<%s%s>r c S s" g | ]\}}d |t |df qS )z %s="%s"T)r )r rZ valuer r r
r s z start_tag..)r\ r Zattribitems)r r r r
r s r c C s* | j rtj| j rd}nd}d| j|f S )zg The text representation of an end tag for a tag. Includes
trailing whitespace when appropriate. rE r z%s)r start_whitespace_rer| r\ )r Zextrar r r
r s r c C s | j d S )NrX )r^ )r r r r
r s r c C s
| j dS )Nz or tags inside of any
block-level elements, e.g. transform , etc.
TF)r\ block_level_tagsblock_level_container_tagsr )r r r r r
r s
r c C s xV| D ]}t |rP qW ddl}tj|}| j|_d| _|jt| |g| dd< dS xxt| D ]l}t |rt|| |jrtj|}|j|_d|_| j | j
|d | qbtj|}| j || |j | qbW | jrtj|}| j|_d| _| j d| dS )zt helper for _fixup_ins_del_tags; actually takes the etc tags
and moves them inside any block-level tags. r Nr )
r sysr ZElementr
r? listr r r< re r= r3 )r r\ r r Z children_tagZtail_tagZ child_tagZtext_tagr r r
r s4
r c C s | j }| jpd}| jrXt| s,|| j7 }n,| d jrL| d j| j7 _n | j| d _|j| }|r|dkrtd}n ||d }|dkr|jr| j|7 _q||_n |jr| j|7 _n||_| j |||d < dS )z
Removes an element, but merges its contents into its place, e.g.,
given .)minrb r r difflibSequenceMatcherget_matching_blocks)rp sizeactualr )r r
r f s z.InsensitiveSequenceMatcher.get_matching_blocksN)rL rM rN rQ r r r r r r
r" ^ s r" __main__) _diffcommand)F)T)T)
r r r r r r r r r r )r r r r r r r r r r r r r r r r r r r r r r8 r r )
r r r r r r r r r r )F)F)Nr Zlxmlr Z lxml.htmlr re__all__r7 r r ImportErrorZcgiZunicoder NameErrorstrZ
basestringr r r r r$ r r2 r r r: r@ rA rK rO ExceptionrP rB rC rF rR rS rT r6 rq rt r rw compileISr{ r} r~ rz Zend_whitespace_rer ry r` r r rx Ur r r r r r r r r; r r r r r r r" rL r mainr r r r
s
(
&
'( 2)
7
!"
word
toword
F)ru T) skip_outer)rw _fixup_ins_del_tagsserialize_html_fragment)r7 r r r r r; s r; c C sL t j| dtd}|rD||jdd d }|d|jd }|j S |S dS )z Serialize a single lxml element as HTML. The serialized form includes the elements tail. If skip_outer is true, then don't serialize the outermost tag r7 )methodencoding>r NrX )r Ztostringr findrfindr )r r r7 r r r r s r c C sH xBdD ]:}x4| j d| D ]"}t|s(qt||d |j qW qW dS )z?fixup_ins_del_tags that works on an lxml document in-place rg rk zdescendant-or-self::%s)r\ N)rg rk )Zxpath_contains_block_level_tag_move_el_inside_blockZdrop_tag)r r\ r r r r r s r c C s6 | j tks| j tkrdS x| D ]}t|r dS q W dS )zPTrue if the element contains any block-level elements, like,
Hi there!
, if you remove the element you getHi there!
r r r Nr/ r/ r/ )Z getparentr r rb re Z getchildren)r parentr re Zpreviousr r r _merge_element_contents< s* r c @ s e Zd ZdZdZdd ZdS )r" zt Acts like SequenceMatcher, but tries not to find very small equal blocks amidst large spans of changes r c sD t t| jt| j}t | j|d tjj| } fdd|D S )N c s& g | ] }|d ks |d r|qS )r r )r r ) thresholdr r r j s zBInsensitiveSequenceMatcher.get_matching_blocks.