ó ¾÷Xc@`sØdZddlmZddlmZddlZddlZddlZddlm Z ddlm Z ddl Z ej dkr‘ej Z n ej Z ded d „Zded d „Zd efd „ƒYZdS(sQUtilities for text input preprocessing. May benefit from a fast Cython rewrite. i(tabsolute_import(tdivisionN(trange(tzipis!!"#$%&()*+,-./:;<=>?@[\]^_`{|}~ t cC`sc|r|jƒ}n|jt||t|ƒƒƒ}|j|ƒ}g|D]}|rM|^qMS(s>Converts a text to a sequence of word indices. # Arguments text: Input text (string). filters: Sequence of characters to filter out. lower: Whether to convert the input to lowercase. split: Sentence split marker (string). # Returns A list of integer word indices. (tlowert translatet maketranstlentsplit(ttexttfiltersRR tseqti((s7/tmp/pip-build-isqEY4/keras/keras/preprocessing/text.pyttext_to_word_sequences "cC`sMt|d|d|d|ƒ}g|D]$}tt|ƒƒ|dd^q%S(NR RR i(Rtabsthash(R tnR RR R tw((s7/tmp/pip-build-isqEY4/keras/keras/preprocessing/text.pytone_hot+s   t TokenizercB`sbeZdZd deded„Zd„Zd„Zd„Z d„Z dd „Z dd „Z RS( sÇText tokenization utility class. This class allows to vectorize a text corpus, by turning each text into either a sequence of integers (each integer being the index of a token in a dictionary) or into a vector where the coefficient for each token could be binary, based on word count, based on tf-idf... # Arguments num_words: the maximum number of words to keep, based on word frequency. Only the most common `num_words` words will be kept. filters: a string where each element is a character that will be filtered from the texts. The default is all punctuation, plus tabs and line breaks, minus the `'` character. lower: boolean. Whether to convert the texts to lowercase. split: character or string to use for token splitting. char_level: if True, every character will be treated as a word. By default, all punctuation is removed, turning the texts into space-separated sequences of words (words maybe include the `'` character). These sequences are then split into lists of tokens. They will then be indexed or vectorized. `0` is a reserved index that won't be assigned to any word. s!!"#$%&()*+,-./:;<=>?@[\]^_`{|}~ RcK`s–d|kr+tjdƒ|jdƒ}n|rJtdt|ƒƒ‚ni|_i|_||_||_||_ ||_ d|_ ||_ dS(Ntnb_wordssDThe `nb_words` argument in `Tokenizer` has been renamed `num_words`.s Unrecognized keyword arguments: i( twarningstwarntpopt TypeErrortstrt word_countst word_docsR R Rt num_wordstdocument_countt char_level(tselfRR RR Rtkwargs((s7/tmp/pip-build-isqEY4/keras/keras/preprocessing/text.pyt__init__Qs         c C`s«d|_xÖ|D]Î}|jd7_|jr4|nt||j|j|jƒ}x@|D]8}||jkr„|j|cd7„streverseN(RRRR RR RtsetRtlisttitemstsorttTruetdictRRRt word_indext index_docs( R ttextsR R Rtwcountstwct sorted_voctc((s7/tmp/pip-build-isqEY4/keras/keras/preprocessing/text.pyt fit_on_textshs,    7 "cC`s|t|ƒ|_i|_x]|D]U}t|ƒ}x@|D]8}||jkr]d|j|isUnknown vectorization mode:( RR-Rt ValueErrorRtnptzerost enumerateR(R)tlogR.R<( R R5R?RR$R R tcountstjR3ttftidf((s7/tmp/pip-build-isqEY4/keras/keras/preprocessing/text.pyR@Ûs@           N( t__name__t __module__t__doc__R=R+tFalseR"R4R6R;R7RAR@(((s7/tmp/pip-build-isqEY4/keras/keras/preprocessing/text.pyR6s  %    (i(RPt __future__RRtstringtsystnumpyRFt six.movesRRRt version_infoRRR+RRtobjectR(((s7/tmp/pip-build-isqEY4/keras/keras/preprocessing/text.pyts"