Xc @`scdZddlmZmZmZddlZddlZddlZddlmZm Z m Z m Z ddl m Z ddlmZddlmZddlmZmZmZdd lmZdd lmZd e jfd YZee jd dZejed dZejee j ddidd6d dZ!ej"ee j#dde jfdYZ$e$e jd dZ%eje%d dZ&eje$e j ddidd6d dZ'ej"e&e j#de j(egdZ)ej*j+dj,de)dZ-e j(egdZ.ej*j+dj,de.de jfdYZ/e/e jd dZ0eje0d d Z1ej"e1e j#d d!Z2e j3ej4ed"ffej5e1ej5d"fffd#e6d$ed%e2Z7d&Z8e j3ej4ej9e:d'd(d)e8ed"fffej5e1d"ffd#e6d$ed%e2Z;e j3ej<ej=d"ffe1d"fd$ed#e6Z>e j3ej<ej5ed"fffej5e1d"ffd$ed#e6Z?ej@e7d d*ej@e;d d+ej@e>d d,ej@e?d d-e6d.ZAd/ZBd0ZCd1ZDd2ZEej@e j(ejFgd3ZGd4ZHd5ZId6ZJd7ZKdddddddd8ZMej@e j(ejNgd9ZOej@e j(ejPgd:ZQe j(ej9gd;ZReSZTeTr_ejUeRndS(<s Ops and optimizations: sigmoid, softplus. These functions implement special cases of exp and log to improve numerical stability. i(tabsolute_importtprint_functiontdivisionN(tconfigtgoftprintingtscalar(timap(tpprint(tbasic(telemwisetopttNotScalarConstantError(tvalues_eq_approx_remove_inf(tcopy_stack_tracet ScalarSigmoidcB`sYeZdZedZdZdZdZdZdZ edZ RS(s5 This is just speed opt. Not for stability. cC`sv|dkrdS|dkr dStt|dd}|d kr`ddtj| d d Sddtj| S( Ng>gg>@g?tdtypettint8tuint8tsigtf(RR(tstrtgetattrtnumpytexp(txtx_dtype((s7/tmp/pip-build-X4mzal/theano/theano/tensor/nnet/sigm.pytst_impl"s   cC`s tj|S(N(RR(tselfR((s7/tmp/pip-build-X4mzal/theano/theano/tensor/nnet/sigm.pytimpl/scC`sX|\}|\}t|}||d|}|jjjddksQt|gS(Ng?tfloati(tscalar_sigmoidttypeRtfindtAssertionError(RtinptgradsRtgztytrval((s7/tmp/pip-build-X4mzal/theano/theano/tensor/nnet/sigm.pytgrad2s    !cC`s|\}|\}|jdjtjksD|jdjtjkrOdtS|jdjtjkrsdtStddS(NisQ%(z)s = %(x)s < -88.0f ? 0.0 : %(x)s > 15.0f ? 1.0f : 1.0f /(1.0f + exp(-%(x)s));sK%(z)s = %(x)s < -709.0 ? 0.0 : %(x)s > 19.0 ? 1.0 : 1.0 /(1.0+exp(-%(x)s));s!only floatingpoint is implemented(tinputsR!Rtfloat32tfloat16tlocalstfloat64tNotImplementedError(RtnodetnameR$touttsubRtz((s7/tmp/pip-build-X4mzal/theano/theano/tensor/nnet/sigm.pytc_code<s    cC`s+tt|j}|r#d|S|SdS(Ni(i(tsuperRtc_code_cache_version(Rtv((s7/tmp/pip-build-X4mzal/theano/theano/tensor/nnet/sigm.pyR7Wsc C`s|\}|\}tjjj p?|jdj|jdjkrWtjjj n|jdj}|dko|j t k rd}d} n<|dko|j t k rd}d} ntjjj dt Stjjj ( NiR+Rt amd_vrsa_expfR.tdoublet amd_vrda_exps npy_intp n = PyArray_SIZE(%(z)s); %(dtype)s * x = (%(dtype)s*) PyArray_DATA(%(x)s); %(dtype)s * z = (%(dtype)s*) PyArray_DATA(%(z)s); // We block to keep the data in l1 // normal l1 size = 32k: 32k/2(input + output)/8(nb bytes of double)=2k // We stay bellow the 2k limit to let space for // This is faster than the not blocking version for(int i=0;i=0) { %(z)s = (x<1.7 ? (1.5*x/(1+x)) : (x<3 ? (0.935409070603099 + 0.0458812946797165*(x-1.7)): 0.99505475368673)); } else { %(dtype)s xx = -x; %(z)s = -(xx<1.7 ? (1.5*xx/(1+xx)) : (xx<3 ? (0.935409070603099 + 0.0458812946797165*(xx-1.7)): 0.99505475368673)); } //%(z)s = 0.5*(ultrafasttanh(0.5*x)+1.); %(z)s = 0.5*(%(z)s+1.); (R?R!t dtype_specsR-( RR0R1R$R2R3RR4R((s7/tmp/pip-build-X4mzal/theano/theano/tensor/nnet/sigm.pyR5s  (RlRmRnRoRRR5(((s7/tmp/pip-build-X4mzal/theano/theano/tensor/nnet/sigm.pyRrs tultra_fast_scalar_sigmoidRJtultra_fast_sigmoid_inplacecC`snt|jtjrj|jjtkrjt|jd}t|j d|d}||j _ |gSdS(sl When enabled, change all sigmoid to ultra_fast_sigmoid. For example do mode.including('local_ultra_fast_sigmoid') or use the Theano flag optimizer_including=local_ultra_fast_sigmoid. This speeds up the sigmoid op by using an approximation. This is done after the stabilization and specialize phases to avoid interacting with them. icS`stjj||ddS(Ntatolg{Gz?(RGt TensorTypetvalues_eq_approx(tatb((s7/tmp/pip-build-X4mzal/theano/theano/tensor/nnet/sigm.pyt values_eq_approx_remove_low_prec sN( t isinstancetopRGtElemwiset scalar_opR RJR*RR?ttagRy(R0R2R|((s7/tmp/pip-build-X4mzal/theano/theano/tensor/nnet/sigm.pytlocal_ultra_fast_sigmoids  tuncanonicalizeRcC`svtjtjd|jdj}tjdd|}tjdd|}|||}tj|dd}|S(s An approximation of sigmoid. More approximate and faster than ultra_fast_sigmoid. Approx in 3 parts: 0, scaled linear, 1. Removing the slope and shift does not make it faster. Rig?g?i(Rtupgrade_to_floattScalarRRGtconstanttclip(Rt out_dtypetslopetshift((s7/tmp/pip-build-X4mzal/theano/theano/tensor/nnet/sigm.pyRIs %cC`snt|jtjrj|jjtkrjt|jd}t|j d|d}||j _ |gSdS(NicS`stjj||ddS(NRwg?(RGRxRy(RzR{((s7/tmp/pip-build-X4mzal/theano/theano/tensor/nnet/sigm.pyR|4s( R}R~RGRRR RIR*RR?RRy(R0R2R|((s7/tmp/pip-build-X4mzal/theano/theano/tensor/nnet/sigm.pytlocal_hard_sigmoid-s  RtScalarSoftpluscB`sAeZdZedZdZdZdZdZRS(s) This helps numerical stability. cC`sv|dkrdS|dkr |Stt|dd}|d kr`tjtj|dd Stjtj|S( Ng>gg>@RRRRRR(sint8suint8(RRRtlog1pR(RR((s7/tmp/pip-build-X4mzal/theano/theano/tensor/nnet/sigm.pyt static_implCs   cC`s tj|S(N(RR(RR((s7/tmp/pip-build-X4mzal/theano/theano/tensor/nnet/sigm.pyRPscC`s#|\}|\}|t|gS(N(R (RR$R%RR&((s7/tmp/pip-build-X4mzal/theano/theano/tensor/nnet/sigm.pyR)Ss  cC`s|\}|\}|jdjtjksD|jdjtjkrOdtS|jdjtjkrsdtStddS(NisJ%(z)s = %(x)s < -103.0f ? 0.0 : %(x)s > 14.0f ? %(x)s : log1p(exp(%(x)s));sH%(z)s = %(x)s < -745.0 ? 0.0 : %(x)s > 16.0 ? %(x)s : log1p(exp(%(x)s));s!only floatingpoint is implemented(R*R!RR+R,R-R.R/(RR0R1R$R2R3RR4((s7/tmp/pip-build-X4mzal/theano/theano/tensor/nnet/sigm.pyR5Xs    cC`s+tt|j}|r#d|S|SdS(Ni(i(R6RR7(RR8((s7/tmp/pip-build-X4mzal/theano/theano/tensor/nnet/sigm.pyR7ls( RlRmRnRoRRR)R5R7(((s7/tmp/pip-build-X4mzal/theano/theano/tensor/nnet/sigm.pyR?s    tscalar_softplustsoftpluscC`sj|jrf|jjtjkrfg|jjD]}t|s+|^q+}t|dkrf|dSndS(Nii(townerR~RGtmulR*t_is_1tlen(trtitnot_is_1((s7/tmp/pip-build-X4mzal/theano/theano/tensor/nnet/sigm.pyt _skip_mul_1ys+Rtallow_multiple_clientsRytskip_identities_fncC`s?y#tj|}tj|dSWntjk r:tSXdS(sX Returns ------- bool True iff expr is a constant close to 1. iN(R tget_scalar_constant_valueRtallcloseRGR tFalse(texprR8((s7/tmp/pip-build-X4mzal/theano/theano/tensor/nnet/sigm.pyRs tpatternR't constrainttlogsigm_to_softplustlog1msigm_to_softplustlog1pexp_to_softplusslog1p_neg_sigmoid,cC`s|jr|jjtjkrtj|jjd|\}}}t|dkr|d}|jr|jjtjkr|r|d}x|dD]}||}qWt j |drt |jjdfSnt j jrtj dqqqndS(s Returns ------- object If 't' is of the form (1+exp(x)), return (False, x). Else return None. tonly_process_constantsiis<Although your current code is fine, please note that Theano versions prior to 0.5 (more specifically, prior to commit 7987b51 on 2011-12-18) may have yielded an incorrect result. To remove this warning, either set the `warn.identify_1pexp_bug` config option to False, or `warn.ignore_bug_before` to at least '0.4.1'.N(RR~RGtaddR tscalarconsts_restR*RRRRRRtwarntidentify_1pexp_bugtwarningsRC(ttRtscalarst scalar_inputst nonconstst maybe_exptscal_sumts((s7/tmp/pip-build-X4mzal/theano/theano/tensor/nnet/sigm.pytis_1pexps    cC`sct}t|}|dk r-t}|}n|jr_|jjtjkr_||jjdfSdS(s Match a variable with either of the `exp(x)` or `-exp(x)` patterns. Parameters ---------- var The Variable to analyze. Returns ------- tuple A pair (b, x) with `b` a boolean set to True if `var` is of the form `-exp(x)` and False if `var` is of the form `exp(x)`. If `var` cannot be cast into either form, then return `None`. iN( Rtis_negRCRYRR~RGRR*(tvartnegtneg_info((s7/tmp/pip-build-X4mzal/theano/theano/tensor/nnet/sigm.pytis_exps   cC`s0|jr(|jjtjkr(|jjSdSdS(s  Match a variable with `x * y * z * ...`. Parameters ---------- var The Variable to analyze. Returns ------- object A list [x, y, z, ...] if `var` is of the form `x * y * z * ...`, or None if `var` cannot be cast into this form. N(RR~RGRR*RC(R((s7/tmp/pip-build-X4mzal/theano/theano/tensor/nnet/sigm.pytis_muls c C`s|jr-|jjtjkr-|jj}n |g}g}t}g}xY|D]Q}||}|dkr}|j|qO|\}}|j|||N}qOW|||fS(N(RR~RGRR*RRCtappend( RRRztf_termsRtrestRtf_ttneg_t((s7/tmp/pip-build-X4mzal/theano/theano/tensor/nnet/sigm.pytpartition_num_or_denom s      cC`s|j}|sdS|jtjkr0|jdS|jtjkr t|jdkr xt|jD]\}}y%t j |}t j |d}Wnt k rt}nX|rgt|jdkr|jd|Stj|jd|!|j|dSqgqgWndS(s Match a variable with the `-x` pattern. Parameters ---------- var The Variable to analyze. Returns ------- object `x` if `var` is of the form `-x`, or None otherwise. iiiiN(RRCR~RGRR*RRt enumerateR RRRR R(Rtapplytidxt mul_inputRt is_minus_1((s7/tmp/pip-build-X4mzal/theano/theano/tensor/nnet/sigm.pyR"s$  '  c C`s|jtjkr|j\}}t|t\}}}t|t\}}}g} xl|D]d} | |kr| jt| ||j | =n| jt| t |j d| dq^W| sdS| g|D]} tj | ^q|} t | dkr| d} ntj| } ||Ar<| } nt || t |dkrb| gSt |dkr| |d} n| tj|} t |j d| | gSdS(sF exp(x)/(1+exp(x)) -> sigm(x) c/(1+exp(x)) -> c*sigm(-x) iiNi(R~RGttrue_divR*RRRRRKtindexRR?RRR( R0tnumtdenomt num_exp_xtnum_resttnum_negt denom_1pexpt denom_restt denom_negtsigmoidsRtnew_numR2((s7/tmp/pip-build-X4mzal/theano/theano/tensor/nnet/sigm.pytlocal_exp_over_1_plus_expLs6   *    cC`swt|}|dkrZt|}|dkr:t|gSt|\}}| |gSntttt|gSdS(sQ Parse a tree of multiplications starting at the given root. Parameters ---------- root The variable at the root of the tree. Returns ------- object A tree where each non-leaf node corresponds to a multiplication in the computation of `root`, represented by the list of its inputs. Each input is a pair [n, x] with `n` a boolean value indicating whether sub-tree `x` should be negated. Examples -------- x * y -> [False, [[False, x], [False, y]]] -(x * y) -> [True, [[False, x], [False, y]]] -x * y -> [False, [[True, x], [False, y]]] -x -> [True, x] (x * y) * -z -> [False, [[False, [[False, x], [False, y]]], [True, z]]] N(RRCRRtparse_mul_treetlisttmap(troottmul_infoRRtsub_tree((s7/tmp/pip-build-X4mzal/theano/theano/tensor/nnet/sigm.pyRs     cC`swxpt|D]b\}}|d|kr |ddc|N<|||dd<|j||j|tSq WtS(s Attempt to replace a leaf of a multiplication tree. We search for a leaf in `leaves` whose argument is `arg`, and if we find one, we remove it from `leaves` and add to `new_leaves` a leaf with argument `arg` and variable `op(arg)`. Parameters ---------- arg The argument of the leaf we are looking for. leaves List of leaves to look into. Each leaf should be a pair (x, l) with `x` the argument of the Op found in the leaf, and `l` the actual leaf as found in a multiplication tree output by `parse_mul_tree` (i.e. a pair [boolean, variable]). new_leaves If a replacement occurred, then the leaf is removed from `leaves` and added to the list `new_leaves` (after being modified by `op`). op A function that, when applied to `arg`, returns the Variable we want to replace the original leaf variable with. neg : bool If True, then the boolean value associated to the leaf should be swapped. If False, then this value should remain unchanged. Returns ------- bool True if a replacement occurred, or False otherwise. ii(RtpopRRYR(targtleavest new_leavesR~RRR((s7/tmp/pip-build-X4mzal/theano/theano/tensor/nnet/sigm.pyt replace_leafs!  cC`s|\}}t|trg}xEtt|D]4}|ddkrX||dN}q1|j|q1W|s~|dg}qt|dkr|ddc|N<|d}q||g}n|}|S(s Simplify a multiplication tree. Parameters ---------- tree A multiplication tree (as output by `parse_mul_tree`). Returns ------- object A multiplication tree computing the same output as `tree` but without useless multiplications by 1 nor -1 (identified by leaves of the form [False, None] or [True, None] respectively). Useless multiplications (with less than two inputs) are also removed from the tree. iiN(R}RRt simplify_mulRCRR(ttreeRR*ts_inputsts_iR(((s7/tmp/pip-build-X4mzal/theano/theano/tensor/nnet/sigm.pyRs  cC`sq|\}}|dkr'tdn6t|trWtjttt|}n|}|rm| }n|S(s Compute the Variable that is the output of a multiplication tree. This is the inverse of the operation performed by `parse_mul_tree`, i.e. compute_mul(parse_mul_tree(tree)) == tree. Parameters ---------- tree A multiplication tree (as output by `parse_mul_tree`). Returns ------- object A Variable that computes the multiplication represented by the tree. seFunction `compute_mul` found a missing leaf, did you forget to call `simplify_mul` on the tree first?N(RCR#R}RRGRRt compute_mul(RRR*R(((s7/tmp/pip-build-X4mzal/theano/theano/tensor/nnet/sigm.pyRs   ! cC`s|dkrg}n|dkr*g}n|dkr?g}n|dkrTg}n|dkri|}ntrtdtd|td|td|td|td|td|n|\}} t| trOt} xWt| D]I\} } | td| d |d | d |d |d |d|d|O} qW| S| } t}t| }|dk r|\}}||N}t|}|dkrt |||t |s|j ||ft }qqt |||d|s|j ||ft }qn| j r| j jt kr| j jd}t|}|dkrt |||d|s|j ||ft }qqt |||t |s|j ||ft }qnt }|s|dk sttdg|d|s full_tree = %ss tree = %ss exp_x = %ss exp_minus_x = %ss sigm_x = %ss sigm_minus_x= %sRtparentt child_idxtexp_xt exp_minus_xtsigm_xt sigm_minus_xt full_treecS`s t| S(N(RK(R((s7/tmp/pip-build-X4mzal/theano/theano/tensor/nnet/sigm.pytsicS`s t| S(N(RK(R((s7/tmp/pip-build-X4mzal/theano/theano/tensor/nnet/sigm.pyRsiN(RCRR`R}RRtperform_sigm_times_expRRRRKRRYRR~R*R#(RRRRRRRRRR*R(tsub_idxRRtkeep_ittexp_infotexp_negtexp_argtneg_argtsigm_arg((s7/tmp/pip-build-X4mzal/theano/theano/tensor/nnet/sigm.pyR%sv+                              cC`sr|jtjkrdSt|jd}t|}|s?dSt|}t|}t |jd||gS(s exp(x) * sigm(-x) -> sigm(x) exp(-x) * sigm(x) -> sigm(-x) todo: add stack traces to the intermediate variables iN( R~RGRRCRR?RRRR(R0tmul_treet did_somethingR2((s7/tmp/pip-build-X4mzal/theano/theano/tensor/nnet/sigm.pytlocal_sigm_times_exps    cC`s|jtjkr|jd}|jr|jjtjkrtj|jjdt\}}}t |dkr|djr|djjtj kr|rt j t j |drtjttj|djjd|}t|d||jdg||SqqqndS(s# 1/(1+exp(x)) -> sigm(-x) iRiN(R~RGtinvR*RRR RRYRRRRtsumt _fill_chainRKRRR?(R0tinv_argRRRR2((s7/tmp/pip-build-X4mzal/theano/theano/tensor/nnet/sigm.pytlocal_inv_1_plus_exps  $&! cC`s|jtjkr|j\}}t|jdkr:dS|jr|jjtkrytj |}Wnt k r|dSXt j t j |drt|jjd }t||jdg||gSqndS(s 1-sigm(x) -> sigm(-x) iNi(R~RGR3R*RtclientsRRKR Rt ExceptionRRRRR?(R0tsub_ltsub_rtval_lR2((s7/tmp/pip-build-X4mzal/theano/theano/tensor/nnet/sigm.pytlocal_1msigmoids (VRnt __future__RRRRRR<RRRRt theano.compatRttheano.printingRt theano.tensorR RGR R R ttheano.tensor.typeR ttheano.gof.optRt UnaryScalarOpRRR RRKt transfer_typeRqtassigntFunctionPrinterRrRuRJRvtlocal_optimizerRtcompiletoptdbtregisterRIRRRRRt PatternSubtlogRRYRRR3tdictRRRRtlog1p_neg_sigmoidtregister_stabilizeRRRRRRRRRRRRCRRRRRRRtregister_local_1msigmoidtregister_canonicalize(((s7/tmp/pip-build-X4mzal/theano/theano/tensor/nnet/sigm.pyts   "  6      3          (    *!5 , + + !   v!!