ó ¾÷Xc@@sxddlmZddlZddlZddlmZdddddd„Zdd „Z d d e e dd „Z dS( i(tabsolute_importN(trangetint32tpregc C@s.t|dƒstdƒ‚ng}xI|D]A}t|dƒsYtdt|ƒƒ‚n|jt|ƒƒq+Wt|ƒ}|d krštj|ƒ}ntƒ} x:|D]2} t| ƒdkrªtj | ƒj d} PqªqªWtj ||f| ƒ|j |ƒ}x!t |ƒD]\} } t| ƒs1qn|dkrK| | } n)|dkrd| | } ntd|ƒ‚tj | d |ƒ} | j d| kr¿td | j d| | fƒ‚n|dkrê| || d t| ƒ…fcC@syd}tjtt|ƒƒƒ}d|d<|tj|ƒ|ddd|}||}tjd|tj|ƒƒS(s‹Generates a word rank-based probabilistic sampling table. This generates an array where the ith element is the probability that a word of rank i would be sampled, according to the sampling distribution used in word2vec. The word2vec formula is: p(word) = min(1, sqrt(word.frequency/sampling_factor) / (word.frequency/sampling_factor)) We assume that the word frequencies follow Zipf's law (s=1) to derive a numerical approximation of frequency(rank): frequency(rank) ~ 1/(rank * (log(rank) + gamma) + 1/2 - 1/(12*rank)) where gamma is the Euler-Mascheroni constant. # Arguments size: int, number of possible words to sample. sampling_factor: the sampling factor in the word2vec formula. # Returns A 1D Numpy array of length `size` where the ith entry is the probability that a word of rank i should be sampled. gX9´Èvâ?iigà?gð?g(@(R tarraytlistRtlogtminimumtsqrt(tsizetsampling_factortgammatranktinv_fqtf((s;/tmp/pip-build-isqEY4/keras/keras/preprocessing/sequence.pytmake_sampling_tableSs  ' igð?cC@s+g}g}xüt|ƒD]î\} } | s1qn|dk r\|| tjƒkr\qq\ntd| |ƒ} tt|ƒ| |dƒ} xxt| | ƒD]g} | | krœ|| }|sÄqœn|j| |gƒ|ró|jddgƒq|jdƒqœqœWqW|dkrÒtt|ƒ|ƒ}g|D]}|d^q4}tj |ƒ|gt|ƒD]0} || t|ƒtj d|dƒg^qg7}|r¾|ddgg|7}qÒ|dg|7}n|r!tj ddƒ}tj |ƒtj |ƒtj |ƒtj |ƒn||fS(sQGenerates skipgram word pairs. Takes a sequence (list of indexes of words), returns couples of [word_index, other_word index] and labels (1s or 0s), where label = 1 if 'other_word' belongs to the context of 'word', and label=0 if 'other_word' is randomly sampled # Arguments sequence: a word sequence (sentence), encoded as a list of word indices (integers). If using a `sampling_table`, word indices are expected to match the rank of the words in a reference dataset (e.g. 10 would encode the 10-th most frequently occuring token). Note that index 0 is expected to be a non-word and will be skipped. vocabulary_size: int. maximum possible word index + 1 window_size: int. actually half-window. The window of a word wi will be [i-window_size, i+window_size+1] negative_samples: float >= 0. 0 for no negative (=random) samples. 1 for same number as positive samples. etc. shuffle: whether to shuffle the word couples before returning them. categorical: bool. if False, labels will be integers (eg. [0, 1, 1 .. ]), if True labels will be categorical eg. [[1,0],[0,1],[0,1] .. ] sampling_table: 1D array of size `vocabulary_size` where the entry i encodes the probabibily to sample a word of rank i. # Returns couples, labels: where `couples` are int pairs and `labels` are either 0 or 1. # Note By convention, index 0 in the vocabulary is a non-word and will be skipped. iigÐcAN( RR trandomRtminR RR tinttshuffletrandinttseed(tsequencetvocabulary_sizet window_sizetnegative_samplesR1t categoricaltsampling_tabletcouplestlabelstitwit window_startt window_endtjtwjtnum_negative_samplestctwordsR3((s;/tmp/pip-build-isqEY4/keras/keras/preprocessing/sequence.pyt skipgramsssF%      A   ( t __future__RtnumpyR R.t six.movesRR R!R-tTruetFalseRE(((s;/tmp/pip-build-isqEY4/keras/keras/preprocessing/sequence.pyts  I !