BXQc@sdZddlmZdZdZdZdZdZddlm Z m Z dd l Z dd l Z dd lZ d efd YZejeZd fd YZdefdYZdeefdYZdeefdYZdefdYZdZdZdee fdYZdefdYZdefdYZdefdYZd efd!YZd"efd#YZ d$efd%YZ!d&efd'YZ"e#d(krdd l$Z$ee$j%j&Z'e'j(GHnd S()s/Beautiful Soup Elixir and Tonic "The Screen-Scraper's Friend" v2.1.1 http://www.crummy.com/software/BeautifulSoup/ Beautiful Soup parses arbitrarily invalid XML- or HTML-like substance into a tree representation. It provides methods and Pythonic idioms that make it easy to search and modify the tree. A well-formed XML/HTML document will yield a well-formed data structure. An ill-formed XML/HTML document will yield a correspondingly ill-formed data structure. If your document is only locally well-formed, you can use this library to find and process the well-formed part of it. The BeautifulSoup class has heuristics for obtaining a sensible parse tree in the face of common HTML errors. Beautiful Soup has no external dependencies. It works with Python 2.2 and up. Beautiful Soup defines classes for four different parsing strategies: * BeautifulStoneSoup, for parsing XML, SGML, or your domain-specific language that kind of looks like XML. * BeautifulSoup, for parsing run-of-the-mill HTML code, be it valid or invalid. * ICantBelieveItsBeautifulSoup, for parsing valid but bizarre HTML that trips up BeautifulSoup. * BeautifulSOAP, for making it easier to parse XML documents that use lots of subelements containing a single string, where you'd prefer they put that string into an attribute (such as SOAP messages). You can subclass BeautifulStoneSoup or BeautifulSoup to create a parsing strategy specific to an XML schema or a particular bizarre HTML document. Typically your subclass would just override SELF_CLOSING_TAGS and/or NESTABLE_TAGS. i(t generatorss*Leonard Richardson (leonardr@segfault.org)s2.1.1s$Date: 2004/10/18 00:14:20 $s*Copyright (c) 2004-2005 Leonard RichardsontPSF(t SGMLParsertSGMLParseErrorNtNullTypecBsheZdZdZdZdZdZdZdZdZ dZ d Z d Z RS( sSimilar to NoneType with a corresponding singleton instance 'Null' that, unlike None, accepts any message and returns itself. Examples: >>> Null("send", "a", "message")("and one more", ... "and what you get still") is Null True cCstS(N(tNull(tcls((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyt__new__AscOstS(N(R(tselftargstkwargs((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyt__call__BscCstS(N(R(Rtattr((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyt __getattr__DscCstS(N(R(Rtitem((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyt __getitem__EscCsdS(N((RR tvalue((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyt __setattr__FscCsdS(N((RRR((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyt __setitem__GscCsdS(Ni((R((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyt__len__HscCs tgS(N(titer(R((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyt__iter__KscCstS(N(tFalse(RR((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyt __contains__LscCsdS(NR((R((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyt__repr__Ms( t__name__t __module__t__doc__RR R RRRRRRR(((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyR6s          t PageElementcBs@eZdZeedZdiddZeZdidddZdiddZ e Z didddZ diddZ didddZ e ZdiddZeZdiddd Zdid ZeZdidd Zd Zd ZdZdZdZdZdZdZRS(seContains the navigational information for some part of the page (either a tag or a piece of text)cCsh||_||_t|_t|_t|_|jrd|jjrd|jjd|_||j_ndS(sNSets up the initial relations between this element and other elements.iN(tparenttpreviousRtnexttpreviousSiblingt nextSiblingtcontents(RRR((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pytsetupTs     cCs|j|j|||S(sjReturns the first item that matches the given criteria and appears after this Tag in the document.(t_firstt fetchNext(Rtnametattrsttext((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pytfindNext`scCs|j|||||jS(sdReturns all items that match the given criteria and appear before after Tag in the document.(t_fetcht nextGenerator(RR&R'R(tlimit((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyR%fscCs|j|j|||S(s{Returns the closest sibling to this Tag that matches the given criteria and appears after this Tag in the document.(R$tfetchNextSiblings(RR&R'R(((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pytfindNextSiblingkscCs|j|||||jS(sqReturns the siblings of this Tag that match the given criteria and appear after this Tag in the document.(R*tnextSiblingGenerator(RR&R'R(R,((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyR-qscCs|j|j|||S(skReturns the first item that matches the given criteria and appears before this Tag in the document.(R$t fetchPrevious(RR&R'R(((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyt findPreviousvscCs|j|||||jS(scReturns all items that match the given criteria and appear before this Tag in the document.(R*tpreviousGenerator(RR&R'R(R,((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyR0{scCs|j|j|||S(s|Returns the closest sibling to this Tag that matches the given criteria and appears before this Tag in the document.(R$tfetchPreviousSiblings(RR&R'R(((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pytfindPreviousSiblingscCs|j|||||jS(srReturns the siblings of this Tag that match the given criteria and appear before this Tag in the document.(R*tpreviousSiblingGenerator(RR&R'R(R,((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyR3scCs2t}|j||d}|r.|d}n|S(sOReturns the closest parent of this Tag that matches the given criteria.ii(Rt fetchParents(RR&R'trtl((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyt findParents  cCs|j||d||jS(sFReturns the parents of this Tag that match the given criteria.N(R*tNonetparentGenerator(RR&R'R,((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyR6scCs2t}||||d}|r.|d}n|S(Nii(R(RtmethodR&R'R(R7R8((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyR$s  cCsTt|dsi|d6}ng}|}xtrOy|j}Wntk r[PnXd} t|tr|s| s|j||rt} xE|jD]7\} } |j | } |j| | st } PqqW| r|} qqqn$|r|j||r|} qn| r1|j | |rLt ||krLPqLq1q1W|S(s8Iterates over a generator looking for things that match.titemstclassN( thasattrtTrueRt StopIterationR:t isinstancetTagt_matchesR=tgetRtappendtlen(RR&R'R(R,t generatortresultstgtitfoundtmatchR t matchAgainsttcheck((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyR*s:      ccs%|}x|r |j}|Vq WdS(N(R(RRK((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyR+s  ccs%|}x|r |j}|Vq WdS(N(R!(RRK((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyR/s  ccs%|}x|r |j}|Vq WdS(N(R(RRK((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyR2s  ccs%|}x|r |j}|Vq WdS(N(R (RRK((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyR5s  ccs%|}x|r |j}|Vq WdS(N(R(RRK((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyR;s  cCst|rVt|t rVx3|D]+}t|tr#|j||r#tSq#WtSt|rl||St|tr|j}nt|t st |}nt |dr|j |St|r||kSt |dr|j |St ||kS(NRMR=(tisListRBRCt NavigableTextRDR@RtcallableR&t basestringtstrR?tsearchthas_key(Rtchunkt howToMatchttag((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyRDs$ !       N(RRRRR#R:R)t firstNextR%R.tfirstNextSiblingR-R1R0t firstPreviousR4tfirstPreviousSiblingR3R9t firstParentR6R$R*R+R/R2R5R;RD(((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyRPs4     #     RQcBseZdZRS(cCs-|dkr|Std|jj|fdS(s7For backwards compatibility, text.string gives you texttstrings!'%s' object has no attribute '%s'N(tAttributeErrort __class__R(RR ((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyR s (RRR (((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyRQstNavigableStringcBseZRS((RR(((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyRb stNavigableUnicodeStringcBseZRS((RR(((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyRcsRCcBsLeZdZdeedZddZdZdZdZ dZ dZ dZ d Z d Zd Zd Zd ZdZdZdddZddZdddZedZeddZdieddZeZdiedddZeZdZdZdZ dZ!dZ"RS(s=Represents a found HTML tag with its attributes and contents.cCsM||_|dkrg}n||_g|_|j||t|_dS(sBasic constructor.N(R&R:R'R"R#Rthidden(RR&R'RR((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyt__init__s     cCs|jj||S(sReturns the value of the 'key' attribute for the tag, or the value given for 'default' if it doesn't have that attribute.(t _getAttrMapRE(Rtkeytdefault((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyRE!scCs|j|S(sqtag[key] returns the value of the 'key' attribute for the tag, and throws an exception if it's not there.(Rf(RRg((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyR'scCs t|jS(s0Iterating over a tag iterates over its contents.(RR"(R((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyR,scCs t|jS(s:The length of a tag is the length of its list of contents.(RGR"(R((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyR0scCs ||jkS(N(R"(Rtx((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyR4scCstS(s-A tag is non-None even if it has no contents.(R@(R((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyt __nonzero__7scCs|j||j|tdt|jD]$}|j||j|krtSqWtS(sReturns true iff this tag has the same name, the same attributes, and the same contents (recursively) as the given tag. NOTE: right now this will return false if two tags have the same attributes in a different order. Should this be fixed?R&R'R"i(R?R&R'RGRRlR"R@(RtotherRK((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyt__eq__`s lcCs ||k S(sZReturns true iff this tag is not identical to the other tag, as defined in __eq__.((RRu((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyt__ne__mscCs t|S(sRenders this tag as a string.(RT(R((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyRrscCs |jdS(Ni(t__str__(R((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyt __unicode__vscCsg}|jrCx1|jD]#\}}|jd||fqWnd}d}|jrdd}n d|j}d }|d kr|}|js|d7}qn|j|d|} |rdd|} n|jr| } ng} d} |rddj|} n|r| j| n| jd |j| |f| j| |rj|d krj| j| n| j|dj| } t| t j k} |r| rt | } n!| r|t krt | } n| S( sReturns a string or Unicode representation of this tag and its contents. NOTE: since Python's HTML parser consumes whitespace, this method is not certain to reproduce the whitespace present in the original string.s%s="%s"ts /sit needUnicodes %st s<%s%s%s>N(R'RFt isSelfClosingR&R:RdtrenderContentstjointtypettypest UnicodeTypetunicodeRRT(RR{tshowStructureIndentR'RgtvaltclosetcloseTagtindentIncrementR"tspacetstattributeStringt isUnicode((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyRxysJ           cCs|j|dtS(NR(RxR@(RR{((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pytprettifyscCsg}x|D]}d}t|ts=t|tjkrLt|}nLt|trw|j|j ||n!|rt|}n t |}|r |dkr|ddkr|d }qn|j|q q Wdj |S(sIRenders the contents of this tag as a (possibly Unicode) string.is RzN( R:RBRcRRRRRCRFRxRTR(RRR{RtcR(((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyR~s  $  cCs|jd|d|S(sConvenience method to retrieve the first piece of text matching the given criteria. 'text' can be a string, a regular expression object, a callable that takes a string and returns whether or not the string 'matches', etc.t recursiveR((Rs(RR(R((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyt firstTextscCs|jd|d|d|S(sConvenience method to retrieve all pieces of text matching the given criteria. 'text' can be a string, a regular expression object, a callable that takes a string and returns whether or not the string 'matches', etc.RR(R,(Rp(RR(RR,((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyt fetchTextscCs8t}|j||||d}|r4|d}n|S(sLReturn only the first child of this Tag matching the given criteria.ii(RRp(RR&R'RR(R7R8((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyRss  cCs4|j}|s|j}n|j|||||S(sExtracts a list of Tag objects that match the given criteria. You can specify the name of the Tag and any attributes you want the Tag to have. The value of a key-value pair in the 'attrs' map can be a string, a list of strings, a regular expression object, or a callable that takes a string and returns whether or not the string matches for some custom definition of 'matches'. The same is true of the tag name.(trecursiveChildGeneratortchildGeneratorR*(RR&R'RR(R,RH((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyRps  cCs|jtjkS(sReturns true iff this is a self-closing tag as defined in the HTML standard. TODO: This is specific to BeautifulSoup and its subclasses, but it's used by __str__(R&t BeautifulSouptSELF_CLOSING_TAGS(R((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyR}scCs|jj|dS(s2Appends the given tag to the contents of this tag.N(R"RF(RRY((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyRFscCsIt|dsBi|_x'|jD]\}}||j|            0       cCs+t|dp*t|tjtjfkS(stConvenience method that works with all 2.x versions of Python to determine whether or not something is listlike.R(R?RRtListTypet TupleType(R8((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyRPscGsi}x{|D]s}t|drLx[|jD]\}}|||d?fej d@dAfej dBdCfgZ dDZ dYe e dEZdFZdGZdHZdIZdJZdKZdLZe dMZdNZdOdPZdQZdRZdSZdTZdUZdVZdWZdXZ RS(ZsdThis class contains the basic parser and fetch code. It defines a parser that knows nothing about tag behavior except for the following: You can't close a tag without closing all the tags it encloses. That is, "" actually means "". [Another possible explanation is "", but since this class defines no SELF_CLOSING_TAGS, it will never use that explanation.] This class is useful for parsing XML or made-up markup languages, or when BeautifulSoup makes an assumption counter to what you were expecting.s€sR|ss‚ssƒss„ss…ss†ss‡ss⁁st%ssŠss<ssŒst?stZssss‘ss’ss“ss”ss•ss–ss—ss˜ss™ssšss>ssœsstzssŸss (<[^<>]*)/>cCs|jddS(Nis />(tgroup(Ri((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pytoss]*)>cCsd|jddS(Ns(R(Ri((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyRqss([-])cCstjj|jdS(Ni(RtMS_CHARSRER(Ri((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyRsss [document]cCstj||j|r2t| r2|j}n||_tj|g|_d|_|j t |dr|j }n|r|j |n|r|j ndS(sInitialize this as the 'root tag' and feed in any text to the parser. NOTE about avoidParserProblems: sgmllib will process most bad HTML, and BeautifulSoup has tricks for dealing with some HTML that kills sgmllib, but Beautiful Soup can nonetheless choke or lose data if your data uses self-closing tags or declarations incorrectly. By default, Beautiful Soup sanitizes its input to avoid the vast majority of these problems. The problems are relatively rare, even in bad HTML, so feel free to pass in False to avoidParserProblems if they don't apply to you, and you'll get better performance. The only reason I have this turned on by default is so I don't get so many tech support questions. The two most common instances of invalid HTML that will choke sgmllib are fixed by the default parser massage techniques:
(No space between name of closing tag and tag close) (Extraneous whitespace in declaration) You can pass in a custom list of (RE object, replace method) tuples to get Beautiful Soup to scrub your input the way you want.itreadN(RCRet ROOT_TAG_NAMERPtPARSER_MASSAGEtavoidParserProblemsRt quoteStackRdtresetR?Rtfeedtdone(RR(RtinitialTextIsEverything((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyRexs       cCs~|jddks?|jddks?|jddkrOtj||S|jddkrttj||StdS(sThis method routes method call requests to either the SGMLParser superclass or the Tag superclass, depending on the method name.tstart_itend_tdo_RqN(RtRR RCR`(Rt methodName((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyR s *cCsL|jr8x,|jD]\}}|j||}qWntj||dS(N(RtsubRR(RR(tfixtm((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyRs cCs4|jx#|jj|jkr/|jq WdS(s^Called when you're done parsing, so that the unclosed tags can be correctly processed.N(tendDatat currentTagR&RtpopTag(R((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyRs cCs9tj|g|_d|_g|_|j|dS(N(RRt currentDataR:RttagStacktpushTag(R((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyRs     cCs||jj}t|jjdkrYt|jjdtrY|jjd|j_n|jru|jd|_n|jS(Niii(RRRGRR"RBRQR_(RRY((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyRs cCs@|jr|jj|n|jj||jd|_dS(Ni(RRFR(RRY((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyRs cCsdj|j}|r|jsBd|kr9d}qBd}nt}t|tjkrft}n||}|j|j |j |j r||j _ n||_ |j j j |ng|_dS(NRzs R|(RRtstripRbRRRRcR#RRRR"RF(RRRto((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyRs         cCs||jkrdSd}d}xTtt|jdddD]3}||j|jkr?t|j|}Pq?q?W|s|d}nx#td|D]}|j}qW|S(sPops the tag stack up to and including the most recent instance of the given tag. If inclusivePop is false, pops the tag stack up to but *not* including the most recent instqance of the given tag.Niii(RR:RlRGRR&R(RR&t inclusivePoptnumPopst mostRecentTagRK((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyt _popToTags& c Cs|jj|}|dk}|jj|}d}t}xtt|jdddD]}|j|}| s|j |kr| r|}Pn|dkr|j |ks|dkr|r|jj|j r|j }t }Pn|j }q\W|r|j ||ndS(s We need to pop up to the previous tag of this type, unless one of this tag's nesting reset triggers comes between this tag and the previous tag of this type, OR unless this tag is a generic nesting trigger and another generic nesting trigger comes between this tag and the previous tag of this type. Examples:

FooBar

should pop to 'p', not 'b'.

FooBar

should pop to 'table', not 'p'.

Foo

Bar

should pop to 'tr', not 'p'.

FooBar

should pop to 'p', not 'b'.

    • *
    • * should pop to 'ul', not the first 'li'.
    • ** should pop to 'table', not the first 'tr' tag should implicitly close the previous tag within the same
    ** should pop to 'tr', not the first 'td' iiiN( t NESTABLE_TAGSRER:tRESET_NESTING_TAGSRVR@RlRGRR&RRR( RR&tnestingResetTriggerst isNestabletisResetNestingtpopTot inclusiveRKtp((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyt _smartPops( &    icCs |jr?djtd|}|jd||fdS|j||jkro| ro|j|nt|||j|j }|j r||j _ n||_ |j ||s||jkr|j n||j kr|jj|d|_ndS(NRzcSs|\}}d||fS(Ns %s="%s"((t.0Rity((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyR4ss<%s%s>i(RRtmapt handle_dataRRRRCRRRRRt QUOTE_TAGSRFtliteral(RR&R't selfClosingRY((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pytunknown_starttag/s"      cCs|jr1|jd|kr1|jd|dS|j|j||jr|jd|kr|jjt|jdk|_ndS(Nisi(RRRRRRGR(RR&((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pytunknown_endtagFs   cCs|jj|dS(N(RRF(Rtdata((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyRRscCs|jd|dS(s0Propagate processing instructions right through.sN(R(RR(((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyt handle_piUscCs|jd|dS(s!Propagate comments right through.s N(R(RR(((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pythandle_commentYscCs|jd|dS(s"Propagate char refs right through.s&#%s;N(R(Rtref((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pythandle_charref]scCs|jd|dS(s$Propagate entity refs right through.s&%s;N(R(RR((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pythandle_entityrefascCs|jd|dS(s.Propagate DOCTYPEs and the like right through.sN(R(RR((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyt handle_declescCsd}|j||d!dkr{|jjd|}|dkrSt|j}n|j|j|d|!|d}nTytj||}Wn;tk r|j|}|j||t|}nX|S(s^Treat a bogus SGML declaration as raw data. Treat a CDATA declaration as regular data.i s iiN(R:trawdataRtRGRRtparse_declarationR(RRKtjRttoHandle((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyRis     N(!RRRRRRRRtretcompileRRR:R@ReR RRRRRRRRRRRRRRRRR(((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyR4s        *        0       Rc Bs-eZdZed*ddddddddd g Zid*d 6Zd d d dddddgZdddddgZigd6gd6ddgd6gd6dgd6dgd6Z igd6ddd d!gd"6d"gd#6d"gd$6Z d%d&d'd(gZ ed*ed)e e e Z egeee e Z RS(+sThis parser knows the following facts about HTML: * Some tags have no closing tag and should be interpreted as being closed as soon as they are encountered. * The text inside some tags (ie. 'script') may contain tags which are not really part of the document and which should be parsed as text, not tags. If you want to parse the text as tags, you can always fetch it and parse it explicitly. * Tag nesting rules: Most tags can't be nested at all. For instance, the occurance of a

    tag should implicitly close the previous

    tag.

    Para1

    Para2 should be transformed into:

    Para1

    Para2 Some tags can be nested arbitrarily. For instance, the occurance of a

    tag should _not_ implicitly close the previous
    tag. Alice said:
    Bob said:
    Blah should NOT be transformed into: Alice said:
    Bob said:
    Blah Some tags can be nested, but the nesting is reset by the interposition of other tags. For instance, a
    , but not close a tag in another table.
    BlahBlah should be transformed into:
    BlahBlah but, Blah
    Blah should NOT be transformed into Blah
    Blah Differing assumptions about tag nesting rules are a major source of problems with the BeautifulSoup class. If BeautifulSoup is not treating as nestable a tag your page author treats as nestable, try ICantBelieveItsBeautifulSoup before writing your own subclass.tbrthrtinputtimgtmetatspacertlinktframetbasetscripttspantfonttqtobjecttbdoRtsuptcentert blockquotetdivtfieldsettinstdeltoltultlitdltddtdtttablettbodyttfootttheadttrttdtthtaddresstformRtpretnoscriptN(RRRRR:RRtNESTABLE_INLINE_TAGStNESTABLE_BLOCK_TAGStNESTABLE_LIST_TAGStNESTABLE_TABLE_TAGStNON_NESTABLE_BLOCK_TAGSRR(((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyR|s0.            tICantBelieveItsBeautifulSoupcBsheZdZddddddddd d d d d ddddgZdgZegejeeZRS(sThe BeautifulSoup class is oriented towards skipping over common HTML errors like unclosed tags. However, sometimes it makes errors of its own. For instance, consider this fragment: FooBar This is perfectly valid (if bizarre) HTML. However, the BeautifulSoup class will implicitly close the first b tag when it encounters the second 'b'. It will think the author wrote "FooBar", and didn't close the first 'b' tag, because there's no real-world reason to bold something that's already bold. When it encounters '' it will close two more 'b' tags, for a grand total of three tags closed instead of two. This can throw off the rest of your document structure. The same is true of a number of other tags, listed below. It's much more common for someone to forget to close (eg.) a 'b' tag than to actually use nested 'b' tags, and the BeautifulSoup class handles the common case. This class handles the not-co-common case: where you can't believe someone wrote what they did, but it's valid HTML and BeautifulSoup screwed up by assuming it wouldn't be. If this doesn't do what you need, try subclassing this class or BeautifulSoup, and providing your own list of NESTABLE_TAGS.temtbigRKtsmallttttabbrtacronymtstrongtcitetcodetdfntkbdtsamptvartbR(RRRt*I_CANT_BELIEVE_THEYRE_NESTABLE_INLINE_TAGSt)I_CANT_BELIEVE_THEYRE_NESTABLE_BLOCK_TAGSRRR(((s=/scratch/rashmi/Condor_Script/src/mechanize/_beautifulsoup.pyRs   t BeautifulSOAPcBseZdZdZRS(sThis class will push a tag with only a single string child into the tag's parent as an attribute. The attribute's name is the tag name, and the value is the string child. An example should give the flavor of the change: baz => baz You can then access fooTag['bar'] instead of fooTag.barTag.string. This is, of course, useful for scraping structures that tend to use subelements instead of attributes, such as SOAP messages. Note that it modifies its input, so don't print the modified version out. I'm not sure how many people really want to use this class; let me know if you do. Mainly I like the name.cCst|jdkr|jd}|jd}|jt|trt|jdkrt|jdtr|jj|j  r|jd||j (sB      I[''