ó àÆ÷Xc @`sdZddlmZmZmZddlmZddljj Z ddl Z ddl Z ddl Z ddl Z ddlmZddlZddlmZddlmZmZddlmZdd lmZmZdd lmZmZdd lmZdd lm Z m!Z!m"Z"e Z#d Z$dZ%dZ&dZ'dZ(e j)dƒZ*e+a,dZ-da.d„Z/dd„Z0dd„Z1dejj2j3fd„ƒYZ4e4ƒZ5d„Z6e+dd„Z7e+de8e+ddd„Z9e+e+e:d„Z;d „Z<d!„Z=d"e>fd#„ƒYZ?d$e@fd%„ƒYZAe+d&„ZBd'„ZCd(„ZDd)eEfd*„ƒYZFd+„ZGd,e+e+e+e+e+e+e:e8d-„ ZHd.eIfd/„ƒYZJeJeH_Ke+dd0„ZLe+dd1„ZMd2„ZNd3e fd4„ƒYZOeOƒZPd5„ZQd6e fd7„ƒYZReRƒZSd8„ZTd9e fd:„ƒYZUeUƒZVd;„ZWd<e fd=„ƒYZXd>„ZYd?e fd@„ƒYZZdA„Z[dS(Bs!Driver for gradient calculations.i(tabsolute_importtprint_functiontdivision(t OrderedDictN(t itervalues(tgof(tutilstVariable(tizip(txrangetreduce(tNullTypet null_type(tget_debug_values(tViewOptFAST_RUNt DebugModes?James Bergstra, Razvan Pascanu, Arnaud Bergeron, Ian Goodfellows (c) 2011, Universite de Montreals3-clause BSD Licenses(theano-dev srestructuredtext enstheano.gradients op.grad(...) returned a non-listcC`sË|o | stdƒ‚|s%|rRt|ttfƒ rR|rH|gS|fSnu|p[| ršt|ttfƒršt|ƒdks’tdƒ‚|dS|s¦|rÃ|r¶t|ƒSt|ƒSn|SdS(s» Formats the outputs according to the flags `use_list` and `use_tuple`. If `use_list` is True, `outputs` is returned as a list (if `outputs` is not a list or a tuple then it is converted in a one element list). If `use_tuple` is True, `outputs` is returned as a tuple (if `outputs` is not a list or a tuple then it is converted into a one element tuple). Otherwise (if both flags are false), `outputs` is returned. s(Both flags cannot be simultaneously Trueis,Wrong arguments. Expected a one element listiN(tAssertionErrort isinstancetlistttupletlen(tuse_listt use_tupletoutputs((s//tmp/pip-build-X4mzal/theano/theano/gradient.pyt format_as-s  " "    tcC`std||||fƒƒS(sä Return an un-computable symbolic variable of type `x.type`. If any call to tensor.grad results in an expression containing this un-computable variable, an exception (NotImplementedError) will be raised indicating that the gradient on the `x_pos`'th input of `op` has not been implemented. Likewise if any call to theano.function involves this variable. Optionally adds a comment to the exception explaining why this gradient is not implemented. scThis variable is Null because the grad method for input %s (%s) of the %s op is not implemented. %s(R (toptx_postxtcomment((s//tmp/pip-build-X4mzal/theano/theano/gradient.pytgrad_not_implementedJscC`std||||fƒƒS(sâ Return an un-computable symbolic variable of type `x.type`. If any call to tensor.grad results in an expression containing this un-computable variable, an exception (GradUndefinedError) will be raised indicating that the gradient on the `x_pos`'th input of `op` is mathematically undefined. Likewise if any call to theano.function involves this variable. Optionally adds a comment to the exception explaining why this gradient is not defined. slThis variable is Null because the grad method for input %s (%s) of the %s op is mathematically undefined. %s(R (RRRR((s//tmp/pip-build-X4mzal/theano/theano/gradient.pytgrad_undefined^stDisconnectedTypecB`sDeZdZedd„Zd„Zd„Zed„Z d„Z RS(s A type indicating that a variable is a result of taking the gradient of c with respect to x when c is not a function of x. A symbolic placeholder for 0, but to convey the extra information that this gradient is 0 because it is disconnected. cC`stdƒ‚dS(NsyIf you're assigning to a DisconnectedType you're doing something wrong. It should only be used as a symbolic placeholder.(R(tselftdatatstricttallow_downcast((s//tmp/pip-build-X4mzal/theano/theano/gradient.pytfilter}scC`stdƒ‚dS(NsyIf you're assigning to a DisconnectedType you're doing something wrong. It should only be used as a symbolic placeholder.(R(R"tother((s//tmp/pip-build-X4mzal/theano/theano/gradient.pytfiter_variable…scC`stS(N(tFalse(tatb((s//tmp/pip-build-X4mzal/theano/theano/gradient.pytmay_share_memoryscC`stdƒ‚dS(NsyIf you're assigning to a DisconnectedType you're doing something wrong. It should only be used as a symbolic placeholder.(R(R*R+tforce_same_dtype((s//tmp/pip-build-X4mzal/theano/theano/gradient.pytvalue_eqscC`sdS(NR!((R"((s//tmp/pip-build-X4mzal/theano/theano/gradient.pyt__str__˜sN( t__name__t __module__t__doc__R)tNoneR&R(R,tTrueR.R/(((s//tmp/pip-build-X4mzal/theano/theano/gradient.pyR!ss    c `sŽddlm‰t|tƒ}t|tƒ}tˆttfƒsOˆg‰ntˆttfƒspˆg‰nt|ttfƒs‘|g}ntˆƒtˆƒks¯t‚xîttˆˆƒƒD]×}|d}|d\}}t|t j ƒsˆ|ƒ}nt|t j ƒs'ˆ|ƒ}nya|j j |j j kr‡t dt|ƒddt|j j ƒdt|j j ƒƒ‚nWqÅtk r›qÅXqÅWtƒ‰‡‡‡‡‡fd†‰x|D]} ˆ| jƒqËWg} x“|D]‹} | ˆkr| jˆˆj| ƒƒqïˆ| j| jjj| ƒd krSt d ƒ‚qï| jˆ| j| jjj| ƒƒqïWt||| ƒS( sh Computes the R operation on `f` wrt to `wrt` evaluated at points given in `eval_points`. Mathematically this stands for the jacobian of `f` wrt to `wrt` right muliplied by the eval points. :type f: Variable or list of Variables `f` stands for the output of the computational graph to which you want to apply the R operator :type wrt: Variable or list of `Variables`s variables for which you compute the R operator of the expression described by `f` :type eval_points: Variable or list of Variables evalutation points for each of the variables in `wrt` :rtype: :class:`~theano.gof.Variable` or list/tuple of Variables depending on type of f :return: symbolic expression such that R_op[i] = sum_j ( d f[i] / d wrt[j]) eval_point[j] where the indices in that expression are magic multidimensional indices that specify both the position within a list and all coordinates of the tensor element in the last. If `wrt` is a list/tuple, then return a list/tuple with the results. i(tas_tensor_variableisElement s# of wrt/eval_point have mismatched sdimensionality: s versus c`s6|dkrdS|j}|j}g}xî|D]æ}|ˆkr^|jˆˆj|ƒƒq/|jdkr¨y|j|jƒƒWqtk r¤|jdƒqXq/|jˆkrá|jˆ|j|jjj|ƒƒq/ˆ|jƒ|jˆ|j|jjj|ƒƒq/Wg}x÷t ||ƒD]æ\}}|dk rt |t j ƒshˆ|ƒ}nt |t j ƒs‰ˆ|ƒ}ny|j j|ƒ}Wn>tk rßtjj||j jƒ}|j j|ƒ}nX|j |j ksøt‚|j|ƒq/|j|ƒq/W|j|j|ƒˆ|/d) provided that both cost and wrt have names known_grads : OrderedDict, optional A ordered dictionary mapping variables to their gradients. This is useful in the case where you know the gradient on some variables but do not know the original cost. return_disconnected : {'zero', 'None', 'Disconnected'} - 'zero' : If wrt[i] is disconnected, return value i will be wrt[i].zeros_like() - 'None' : If wrt[i] is disconnected, return value i will be None - 'Disconnected' : returns variables of type DisconnectedType null_gradients : {'raise', 'return'} Defines the behaviour if some of the variables in `wrt` have a null gradient. The possibles values are: - 'raise' : raise a NullTypeGradError exception - 'return' : return the null gradients Returns ------- variable or list/tuple of variables (matches `wrt`) symbolic expression of gradient of `cost` with respect to each of the `wrt` terms. If an element of `wrt` is not differentiable with respect to the output, then a zero variable is returned. i(RAs(cost and known_grads can't both be None.s3Can't differentiate a NaN cost.cost is NaN because scost must be a scalar.sowrt must not be a set. sets have no defined iteration order, so we can't return gradients in a matching order.sExpected Variable, got s of type s$known_grads must be an OrderedDict. iRCR=sooutput grads must be theano variables.Ambiguous whether %s should be made into tensor or sparse theano variabletfloatslGradients must always be NullType, DisconnectedType, or continuous, but grad was given a known_grad of type c`sud|}ˆdkrnXˆdkr;tj|ddƒn6ˆdkretj|ƒ}t|ƒ‚n tdƒ‚dS(Ns¶grad method was asked to compute the gradient with respect to a variable that is not part of the computational graph of the cost, or is used only by a non-differentiable operator: %stignoretwarnt stackleveliR^s_Invalid value for keyword 'disconnected_inputs', valid values are 'ignore', 'warn' and 'raise'.(twarningsRkRtget_variable_trace_stringtDisconnectedInputErrorRQ(tvartmessage(Rb(s//tmp/pip-build-X4mzal/theano/theano/gradient.pythandle_disconnecteds   R^stensor.grad encountered a NaN. treturnRhR3t DisconnectedN(%ttimeRAR3R@RRR=R RQtwhy_nullRPR?tsetRRRRRR7textendtkeyst_populate_var_to_app_to_idxRRt_float_ones_likethasattrRCtcontinuous_dtypestastypeR!tdisconnected_typetnamet float_dtypest_populate_grad_dictR tNullTypeGradErrort_float_zeros_liket grad_time(R_RMRaRbt add_namesR`treturn_disconnectedtnull_gradientstt0RURVtelemRtvar_to_app_to_idxt grad_dicttmtg_costRptg_varRrt cost_nametgR\RXtt1((Rbs//tmp/pip-build-X4mzal/theano/theano/gradient.pyRcks°;      .     -             "      c `sÔ|d k s|d k st‚t|tƒs3t‚t|tƒsHt‚|d k rlt|tƒslt‚ntt||ƒƒ}d }d }|d k rÍttjdd d|d|d|ddƒƒ}n|d k rttjd|d|d|ddƒƒ}nd }|d kr!|}nF|}|d k rgx1tt |ƒƒD]} || c|| 7Ïsc3`s|]}ˆ|VqdS(N((R“R”(R•(s//tmp/pip-build-X4mzal/theano/theano/gradient.pys ÐsN( R3RRRtdictRwR@RctrangeRRR( RMtendtstartR_tdetailstparamst start_gradst cost_gradsRdRXt wrt_gradst end_grads((R•s//tmp/pip-build-X4mzal/theano/theano/gradient.pyt subgraph_gradGsBa         cC`sÚt|jdƒr|jj|ƒ}t|tƒsatdd|jd|t|ƒfƒ‚nt|ƒt|jƒkr¯t d|jt|jƒfdt|ƒƒ‚nxt |ƒD]M\}}t|tƒs¼td|jd|d |t|ƒfƒ‚q¼q¼Wn2g|jD]"}g|j D] }t ^q*^q}t|tƒsWt ‚t|ƒt|jƒksxt ‚x[tt|jƒƒD]D}t||tƒs­t ‚t||ƒt|j ƒksŽt ‚qŽW|S( sÓ given an apply node, obtain its connection pattern this is just a wrapper around Op.connection_pattern that does type checking and supplies the default value if the method is not implemented tconnection_patterns$Op.connection_pattern should return s#list of list of bool, but for Op=%ssgot %s with type %s.s$%s.connection_pattern should have %ds rows but has %d.s#%s.connection_pattern should returns a list of lists, but element %dsis %s of type %s.(R|RR¡RRR?R=RR6RQRORR4RR (RER¡tiitoutput_patterntipttoutput((s//tmp/pip-build-X4mzal/theano/theano/gradient.pyt_node_to_patternØs4/!)c`sVˆdkrg‰n€ytˆƒWn-tk rRtdttˆƒƒƒ‚nXx?ˆD]7}t|tjƒsZtdtt|ƒƒƒ‚qZqZWtƒ‰t gƒ‰‡‡‡‡fd†‰x|D]}ˆ|ƒqÉWt gƒ‰‡‡‡fd†‰x|D]}ˆ|ƒqWt ˆj ƒƒ}x$|D]}|ˆkr2ˆ|=q2q2WˆS(sñ Helper function for grad function. outputs: a list of variables we want to take gradients of wrt: a list of variables we want to take the gradient with respect to. consider_constant: a list of variables not to backpropagate through. returns: var_to_app_to_idx: A dictionary mapping a variable to a second dictionary. The second dictionary maps apply nodes acting on this variable to the variable's index in the apply node's input list. This dictionary will only contain variables that meet two criteria: 1) The elements of at least one output are a function of the elements of the variable 2) The elements of the variable are a function of the elements of at least one member of wrt. This set is exactly the set of variables that connect the variables in wrt to the cost being differentiated. (A variable in consider_constant is not a function of anything) s6consider_constant must be an iterable collection, got s9Elements of consider_constant must be variables, but got c`s |ˆkrdSˆj|ƒ|ˆkr-dS|jdk r |j}t|ƒ}|jj|ƒ}x£t|jƒD]\}}|||s“qsn|ˆkr¯tƒˆ|WdS(N(R§R¦ROR(RptnodesRER¡R«R¢R¥(R‹tvisittvisited(s//tmp/pip-build-X4mzal/theano/theano/gradient.pyR¯xs      N( R3titerR?RRR=RRRRRwRRy(RRMRaRŠR¥t orig_varsRp((R¬R­RaR‹R¯R°s//tmp/pip-build-X4mzal/theano/theano/gradient.pyRzÿs2'     +     RƒcB`seZdZRS(s1 Raised when grad encounters a NullType. (R0R1R2(((s//tmp/pip-build-X4mzal/theano/theano/gradient.pyRƒ’sRocB`seZdZRS(s‰ Raised when grad is asked to compute the gradient with respect to a disconnected input and disconnected_inputs='raise'. (R0R1R2(((s//tmp/pip-build-X4mzal/theano/theano/gradient.pyRo˜sc`sVtƒ‰‡‡fd†‰‡‡‡‡fd†‰g|D]}ˆ|ƒ^q:}|S(s= Helper function for grad function. var_to_app_to_idx: a dictionary mapping a variable to a second dictionary. the second dictionary maps apply nodes acting on this variable to the variable's index in the apply node's input list grad_dict: A dictionary mapping variables to their gradients. Should be populated by grad function, which should: -Set the gradient with respect to the cost to 1 -Load all gradients from known_grads, possibly overriding the cost -Set the gradient for disconnected inputs to a variable with type DisconnectedType() wrt: the minimal set of variables that must be included in grad_dict cost_name: The name of the cost being differentiated, optional. used to name the grad with respect to x as (d/dx) returns: a list of gradients corresponding to wrt c6`sÖ|ˆkrÎ|j}g|jD]}ˆ|ƒ^q}g|D]}t|jtƒ ^q>}t|ƒ}g|D]:}tgt||ƒD]\}} |ož| ^q‰k^qp} g|jD]0} t| jdƒoç| jj t j j k^qº} g|D]} t| jt ƒ^q÷} g|D]G}tgt||| ƒD]"\}}}|oW|oW| ^q8k^q}t| kr”g|D]}tƒ^q|}n_t|krãg}xJ| D]/}|rÌ|jtƒƒq­|jtƒƒq­Wny4gt|jjƒD]}|j|d^qù‰Wntk r0g‰nX‡fd†}g|D]}||ƒ^qG}g}x”t|j|ƒD]€\}}t|jddƒ}t|jddƒ}|t j j krë|rë||krë|j|j|ƒƒqx|j|ƒqxWx}t|j|ƒD]i\}}t|jddƒ}t|jddƒ}|dk r|t j j kr||ksxt‚qqWx5|D]-}t|jddƒt j j ksƒt‚qƒWxÁtt|j|ƒƒD]§\}} | \}!}"t|!dƒsúqÍnt|"jtƒrqÍnx_t| ŒD]Q\}#}$|#j}%|$j}&|%|&krtdt|%ƒdt|&ƒƒ‚qqWqÍW|jj ||j|ƒ}|dkr»t!dt|jƒƒ‚nt"|ƒt"|ƒkrótdd t|jƒƒ‚nt#|ƒ}xwt$t"|ƒƒD]c}'xZt$t"| ƒƒD]F}(| |(r+||'|(r+t||'jtƒ r+||(||' You may want to implement a connection_pattern method for it.N(syessnosmaybe(*R6RRR=R!R¦R4R<R|RCR@RAtdiscrete_dtypesR RR)R7R RRt destroy_mapRStgetattrR3R~RRORR R¶RQRRtL_opR?RRR—Rt_is_zerotget_scalar_constant_valueRmRk(6RER6Rpt output_gradsR‘toutputs_connectedR¡tinput_to_outputstinput_to_outputtoutput_to_costtinputs_connectedR¥t output_is_intt ograd_is_nant in_to_outst in_to_outt out_to_costtout_nantonly_connected_to_nanR¤t input_gradst connectedRRµtnew_output_gradstotogto_dttog_dttngtng_dtR«tpackedt orig_outputtnew_output_gradt orig_output_vtnew_output_grad_vto_shapetg_shapetinp_idxtout_idxtout_inttonly_connected_to_intRXttermtorig_iptt orig_ipt_vtterm_vti_shapett_shapetis_zerotmsgtigtactually_connected(taccess_grad_cachet term_dict(R´s//tmp/pip-build-X4mzal/theano/theano/gradient.pytaccess_term_cacheÀs  "# D="Q    1               Q     #      " ( .    c`sâ|ˆkrÚ|ˆkrÊg}g}ˆ|}x|D]û}xò||D]æ}ˆ|ƒ|}t|tjƒs–tdt|jƒt|ƒfƒ‚nt|jtƒr»|j|ƒqFnt|jt ƒrÓqFnt |dƒr|j |j krt dt|jƒ|j |j fƒ‚n|j|ƒqFWq5Wt |ƒdkrW|dˆ|ss (d%s/d%s)(RRRR?RRRR=R R7R!R|RPRQRR RR3R€(Rpt null_termsttermst node_to_idxRER«Rá(RíRRŒR‹(s//tmp/pip-build-X4mzal/theano/theano/gradient.pyRëîs@      !%  (R(R‹RŒRMRRŠR\((RëRíRRŒRìR‹s//tmp/pip-build-X4mzal/theano/theano/gradient.pyR‚ s  ÿ/7cC`s>|jƒ}|jjjdƒdkr+|S|jtjjƒS(sM Like zeros_like, but forces the object to have a a floating point dtype Riiÿÿÿÿ(R:R=RCtfindR~R@tconfigtfloatX(RR\((s//tmp/pip-build-X4mzal/theano/theano/gradient.pyR„*s cC`s:|jj}|tjkr*tjj}n|jd|ƒS(sJ Like ones_like, but forces the object to have a floating point dtype RC(R=RCRARR@RóRôt ones_like(RRC((s//tmp/pip-build-X4mzal/theano/theano/gradient.pyR{6s t numeric_gradcB`s‰eZdZidd6dd6dd6dejdƒ6dejdƒ6dejdƒ6Zd d d„Zed„ƒZ d „Z d „Z RS( s_ Compute the numeric derivative of a scalar-valued function at a particular point. gH¯¼šò×z>tfloat64ga2U0*©3?tfloat32gš™™™™™¹?tfloat16c`s¡d„‰t}t|ttfƒs6|g}t}ng|D]}tj|ƒ^q=}g|D]}|j^q_}g|D]}t|j ƒ^q{} t j ‡fd†|Dƒƒ} t j ‡fd†| Dƒƒd} tj | fd| ƒ} |d k r&|jdƒr&tj | fd|ƒ} ntj | fd| ƒ} |d krlt j‡fd†| Dƒƒ}nd}gˆ_x‰t|ƒD]{\}}ˆ|jƒ}| |||!j|jƒ||<ˆjj| |||!j|jƒƒ|||d <||7}qˆW|g|D]}|jƒ^qŒ}| jƒ}xIt| ƒD];}|| (| |c|7<||Œ}|||| |Œsc3`s"|]}ˆj||fVqdS(N(ttype_eps(R“tdt(R"(s//tmp/pip-build-X4mzal/theano/theano/gradient.pys siRCtcomplexc3`s|]}ˆj|VqdS(N(Rü(R“Rý(R"(s//tmp/pip-build-X4mzal/theano/theano/gradient.pys šsi.N(R)RRRR4tnumpytarrayR¶RRRCtbuiltinstsumtmintndarrayR3t startswithtmaxtgfROtreshapeR7R³R (R"RTtpttepstout_typet packed_pttptapttshapestdtypest total_sizet working_dtypeRtgxtcur_posRXtp_sizetf_xtx_copytf_eps((RúR"s//tmp/pip-build-X4mzal/theano/theano/gradient.pyt__init__esD   "" " !'%  cC`s^t||ƒ}|tjt|ƒt|ƒdƒ}tj|ƒ}tj|ƒ}||fS(s»Return absolute and relative error between a and b. The relative error is a small number when a and b are close, relative to how big they are. Formulas used: abs_err = abs(a - b) rel_err = abs_err / max(abs(a) + abs(b), 1e-8) The denominator is clipped at 1e-8 to avoid dividing by 0 when a and b are both close to 0. The tuple (abs_err, rel_err) is returned g:Œ0âŽyE>(tabsRÿtmaximumtasarray(R*R+tabs_errtrel_err((s//tmp/pip-build-X4mzal/theano/theano/gradient.pyt abs_rel_err½s &cC`sÂt|ƒt|jƒkr3tdt|ƒƒ‚ng}x‚tt||jƒƒD]h\}\}}|j|jkr¡td|t|j|jfƒfƒ‚n|jtj ||ƒƒqRW|S(s(Return the abs and rel error of gradient estimate `g_pt` `g_pt` must be a list of ndarrays of the same length as self.gf, otherwise a ValueError is raised. Corresponding ndarrays in `g_pt` and `self.gf` must have the same shape or ValueError is raised. s%argument has wrong number of elementss&argument element %i has wrong shape %s( RRRQROR<R¶RRR7RöR(R"tg_ptterrsRXR*R+((s//tmp/pip-build-X4mzal/theano/theano/gradient.pytabs_rel_errors×s +%cC`sGg}g}g}g}|j|ƒ}xì|D]ä\} } tjtj| ƒƒsjtdt| ƒƒ‚ntjtj| ƒƒsštdt| ƒƒ‚ntj| || |ƒ} | jƒ} |j| ƒ|j| j ƒ| ƒ|j| j ƒ| ƒ|j| j ƒ| ƒq.Wtj|ƒ} || }| ||| || fS(s~Find the biggest error between g_pt and self.gf. What is measured is the violation of relative and absolute errors, wrt the provided tolerances (abs_tol, rel_tol). A value > 1 means both tolerances are exceeded. Return the argmax of min(abs_err / abs_tol, rel_err / rel_tol) over g_pt, as well as abs_err and rel_err at this point. sabs_err not finitesrel_err not finite( R"RÿtalltisfiniteRQtreprtminimumtargmaxR7tflatten(R"R tabs_toltrel_toltposR!tabs_errstrel_errst abs_rel_errsRRt scaled_errtmax_itmax_argtmax_pos((s//tmp/pip-build-X4mzal/theano/theano/gradient.pytmax_errìs&    N( R0R1R2RÿRCRüR3Rt staticmethodRR"R3(((s//tmp/pip-build-X4mzal/theano/theano/gradient.pyRöAs X cC`s0t|tƒr(|j}tjd|ƒS|SdS(Nt optimizer(RRR5Rtclone(tmodetopt((s//tmp/pip-build-X4mzal/theano/theano/gradient.pytmode_not_debugs ic # `s)ddlm‰m} ddl} ddlm} m}t|ttfƒsSt ‚g|D]}t j |ƒ^qZ}xEt |ƒD]7\}}|j d!kr‚td||j fƒ‚q‚q‚Wtdd dd dd ƒ‰|dkrtj‡fd †|Dƒƒ}n|dkr4tj‡fd †|Dƒƒ}nˆdkrOtdƒ‚n|‡fd†}gt |ƒD]:\}}|| |ƒj | |ƒjƒdd|ƒ^qn}||Œ‰tˆtƒrØtdƒ‚n||ˆddƒ}|g|D]}|jƒ^q÷Œ‰tˆtƒs0tˆtƒr?tdƒ‚n‡‡‡‡fd†}| |ƒƒ}d|_| jj|ˆƒ}| ršt|ƒ}n|}|||ddd|ƒ}t||ddƒ}|||ddƒ}x=t|ƒD]/}yït|g|D]}|jƒ^q||ƒ}|g|D]}|jƒ^q3Œ}t|tƒsct ‚|j|||ƒ\}}} }!| |kr½|!|kr½tj||| |!||ƒ‚n||dkræ|j |ƒdt!ƒnWqòt"k r }"|"j#d|d|d |f7_#‚qòXqòWdS("s3Test a gradient by Finite Difference Method. Raise error on failure. Example: >>> verify_grad(theano.tensor.tanh, ... (numpy.asarray([[2,3,4], [-1, 3.3, 9.9]]),), ... rng=numpy.random) Raises an Exception if the difference between the analytic gradient and numerical gradient (computed through the Finite Difference Method) of a random projection of the fun's output to a scalar exceeds the given tolerance. :param fun: a Python function that takes Theano variables as inputs, and returns a Theano variable. For instance, an Op instance with a single output. :param pt: the list of numpy.ndarrays to use as input values. These arrays must be either float16, float32, or float64 arrays. :param n_tests: number of times to run the test :param rng: random number generator used to sample u, we test gradient of sum(u * fun) at pt :param eps: stepsize used in the Finite Difference Method (Default None is type-dependent) Raising the value of eps can raise or lower the absolute and relative errors of the verification depending on the Op. Raising eps does not lower the verification quality for linear operations. It is better to raise eps than raising abs_tol or rel_tol. :param out_type: dtype of output, if complex (i.e. 'complex32' or 'complex64') :param abs_tol: absolute tolerance used as threshold for gradient comparison :param rel_tol: relative tolerance used as threshold for gradient comparison :param cast_to_output_type: if the output is float32 and cast_to_output_type is True, cast the random projection to float32. Otherwise it is float64. float16 is not handled here. :param no_debug_ref: Don't use DebugMode for the numerical gradient function. :note: This function does not support multiple outputs. In tests/test_scan.py there is an experimental verify_grad that covers that case as well by using random projections. i(tcompiletsharedN(R5t TensorTypeRùRøR÷sRverify_grad can work only with floating point inputs, but input %i has dtype "%s".gš™™™™™©?g{®Gáz„?g-Cëâ6?c3`s"|]}ˆt|jƒVqdS(N(RRRC(R“R (t _type_tol(s//tmp/pip-build-X4mzal/theano/theano/gradient.pys Zsc3`s"|]}ˆt|jƒVqdS(N(RRRC(R“R (R=(s//tmp/pip-build-X4mzal/theano/theano/gradient.pys \ssŸrng should be a valid instance of numpy.random.RandomState. You may want to use theano.tests.unittest_tools.verify_grad instead of theano.gradient.verify_grad.c `s4ˆj||dtdtd|ddd|ƒ}|S(Ntaccept_inplacetallow_input_downcastR7ton_unused_inputRjR€(tfunctionR4(R6R¥R€R7RT(R:(s//tmp/pip-build-X4mzal/theano/theano/gradient.pyRAgs R€sinput %is9cant (yet) autotest gradient of fun with multiple outputssgradient.py fwds‘It seems like you are trying to use verify_grad on an op or a function which outputs a list: there should be a single (array-like) output insteadc`sBˆjˆjŒd}ˆr>ˆjdkr>tj|ˆjƒS|S(Ngà?Rø(trandR¶RCRÿR(tplain(tcast_to_output_typeto_fn_outto_outputtrng(s//tmp/pip-build-X4mzal/theano/theano/gradient.pytrandom_projectionˆsRHsgradient.py costR7RbRjsgradient.py symbolic graditborrows. The error happened with the following inputs:s The value of eps is:s The out_type is:(sfloat16sfloat32sfloat64($R@R:R;RNR5R<RRRRRÿRRORCR?R–R3RRt broadcastabletNotImplementedErrorR³R€RARR9RcR RöR3t verify_gradtE_gradt set_valueR4R;targs(#tfunR tn_testsRGR R R)R*R7RDt no_debug_refR;R@R5R<R RXRAt tensor_ptto_fnRHtt_rR_t mode_for_costtcost_fnt symbolic_gradtgrad_fnttest_numtnum_gradt analytic_gradR1t max_err_post max_abs_errt max_rel_errte((R=RDR:RERFRGs//tmp/pip-build-X4mzal/theano/theano/gradient.pyRLs~1 "  " " J %       " %  t GradientErrorcB`s eZdZd„Zd„ZRS(sBThis error is raised when a gradient is calculated, but incorrect.cC`sGtj|ƒ||_||_||_||_||_||_dS(N(R;Rtargterr_posRRR)R*(R"RbRcRRR)R*((s//tmp/pip-build-X4mzal/theano/theano/gradient.pyRÂs      cC`sKdjd„|jDƒƒ}d|j|j|j|j|j|j|fS(Ns, cs`s|]}t|ƒVqdS(N(RR(R“R*((s//tmp/pip-build-X4mzal/theano/theano/gradient.pys ÍssåGradientError: numeric gradient and analytic gradient exceed tolerance: At position %i of argument %i, abs. error = %f, abs. tolerance = %f rel. error = %f, rel. tolerance = %f Exception args: %s(tjoinRORcRbRR)RR*(R"targs_msg((s//tmp/pip-build-X4mzal/theano/theano/gradient.pyR/Ës   (R0R1R2RR/(((s//tmp/pip-build-X4mzal/theano/theano/gradient.pyRaÀs c `s/ddlm}t|tƒs+tdƒ‚|jdksFtdƒ‚t|tƒ}t|tƒ}t|ttfƒrˆt|ƒ}n |g}|jdkrÅt||t ||dˆdˆƒƒS‡‡fd†}t j |d ||j dƒd |g|ƒ\}} | std ƒ‚t|||ƒS( s3 :type expression: Vector (1-dimensional) Variable :type wrt: Variable or list of Variables :param consider_constant: a list of expressions not to backpropagate through :type disconnected_inputs: string :param disconnected_inputs: Defines the behaviour if some of the variables in ``wrt`` are not part of the computational graph computing ``cost`` (or if all links are non-differentiable). The possible values are: - 'ignore': considers that the gradient on these parameters is zero. - 'warn': consider the gradient zero, and print a warning. - 'raise': raise an exception. :return: either a instance of Variable or list/tuple of Variables (depending upon `wrt`) repesenting the jacobian of `expression` with respect to (elements of) `wrt`. If an element of `wrt` is not differentiable with respect to the output, then a zero variable is returned. The return value is of same type as `wrt`: a list/tuple or TensorVariable in all cases. i(taranges2tensor.jacobian expects a Variable as `expression`ishtensor.jacobian expects a 1 dimensional variable as `expression`. If not use flatten to make it a vectorRaRbc`s_|d}|d}g}x>|dD]2}t|||dˆdˆƒ}|j|ƒq%W|S(NiiiRaRb(RcR7(ROR«texprtrvalsRGR\(RaRb(s//tmp/pip-build-X4mzal/theano/theano/gradient.pytinner_function s    t sequencest non_sequencess‹Scan has returned a list of updates. This should not happen! Report this to theano-users (also include the script that generated the error)( RNRfRRRRPRRRRcR@tscanR¶( t expressionRMRaRbRfRURVRitjacobstupdates((RaRbs//tmp/pip-build-X4mzal/theano/theano/gradient.pytjacobianÜs.        c `sfddlm}t|tƒs+tdƒ‚|jdksFtdƒ‚t|tƒ}t|tƒ}t|ttfƒrˆt|ƒ}n |g}g}x¼|D]´}t|tƒs¿tdƒ‚|jdksÚtdƒ‚t||dˆd |ƒ} t j ‡fd †d || j dƒd | |gƒ\} } | sEtd ƒ‚|j | ƒqžWt |||ƒS(s| :type cost: Scalar (0-dimensional) Variable. :type wrt: Vector (1-dimensional tensor) 'Variable' or list of vectors (1-dimensional tensors) Variables :param consider_constant: a list of expressions not to backpropagate through :type disconnected_inputs: string :param disconnected_inputs: Defines the behaviour if some of the variables in ``wrt`` are not part of the computational graph computing ``cost`` (or if all links are non-differentiable). The possible values are: - 'ignore': considers that the gradient on these parameters is zero. - 'warn': consider the gradient zero, and print a warning. - 'raise': raise an exception. :return: either a instance of Variable or list/tuple of Variables (depending upon `wrt`) repressenting the Hessian of the `cost` with respect to (elements of) `wrt`. If an element of `wrt` is not differentiable with respect to the output, then a zero variable is returned. The return value is of same type as `wrt`: a list/tuple or TensorVariable in all cases. i(Rfs+tensor.hessian expects a Variable as `cost`s9tensor.hessian expects a 0 dimensional variable as `cost`s4tensor.hessian expects a (list of) Variable as `wrt`isBtensor.hessian expects a (list of) 1 dimensional variable as `wrt`RaRbc`st|||dˆddƒS(NRaRbRj(Rc(RXRIR(Ra(s//tmp/pip-build-X4mzal/theano/theano/gradient.pyRîZsRjRks‹Scan has returned a list of updates. This should not happen! Report this to theano-users (also include the script that generated the error)(RNRfRRRRPRRRcR@RlR¶R7R( R_RMRaRbRfRURVthessianstinputRgthessRo((Ras//tmp/pip-build-X4mzal/theano/theano/gradient.pythessian%s2         cC`s¨t|dƒs"tj|dkƒSt|jtƒr8dSt|jtƒrNdSt}ytj |ƒ}t }Wntj j j k r‰nX|r”dS|dkr¤dSdS(sÄ Returns 'yes', 'no', or 'maybe' indicating whether x is always 0. 'maybe' means that x is an expression that is complicated enough that we can't tell that it simplifies to 0. R=gR¸R·R¹(R|tnpR#RR=R R!R4R@R¿R)RAtbasictNotScalarConstantError(Rtno_constant_valuetconstant_value((s//tmp/pip-build-X4mzal/theano/theano/gradient.pyR¾is"  tConsiderConstantcB`seZd„ZRS(cC`s g|D]}|j|ƒ^qS(N(R:(R"ROtg_outstg_out((s//tmp/pip-build-X4mzal/theano/theano/gradient.pyRcˆs(R0R1Rc(((s//tmp/pip-build-X4mzal/theano/theano/gradient.pyRz‡scC`stjdddƒt|ƒS(sc DEPRECATED: use zero_grad() or disconnected_grad() instead. Consider an expression constant when computing gradients. The expression itself is unaffected, but when its gradient is computed, or the gradient of another expression that this expression is a subexpression of, it will not be backpropagated through. In other words, the gradient of the expression is truncated to 0. :param x: A Theano expression whose gradient should be truncated. :return: The expression is returned unmodified, but its gradient is now truncated to 0. .. versionadded:: 0.7 sRconsider_constant() is deprecated, use zero_grad() or disconnected_grad() instead.Rli(RmRktconsider_constant_(R((s//tmp/pip-build-X4mzal/theano/theano/gradient.pyRastZeroGradcB`seZd„ZRS(cC`s g|D]}|j|ƒ^qS(N(R:(R"ROR{R|((s//tmp/pip-build-X4mzal/theano/theano/gradient.pyRc«s(R0R1Rc(((s//tmp/pip-build-X4mzal/theano/theano/gradient.pyR~ªscC`s t|ƒS(s Consider an expression constant when computing gradients. The expression itself is unaffected, but when its gradient is computed, or the gradient of another expression that this expression is a subexpression of, it will be backpropagated through with a value of zero. In other words, the gradient of the expression is truncated to 0. :param x: A Theano expression whose gradient should be truncated. :return: The expression is returned unmodified, but its gradient is now truncated to 0. (t zero_grad_(R((s//tmp/pip-build-X4mzal/theano/theano/gradient.pyt zero_grad²stDisconnectedGradcB`s#eZd„Zd„Zd„ZRS(cC`sg|D]}tƒ^qS(N(R(R"ROR{R|((s//tmp/pip-build-X4mzal/theano/theano/gradient.pyRcÅscC`sdgS(N(R3(R"R6RK((s//tmp/pip-build-X4mzal/theano/theano/gradient.pyRDÈscC`s tggS(N(R)(R"RE((s//tmp/pip-build-X4mzal/theano/theano/gradient.pyR¡Ës(R0R1RcRDR¡(((s//tmp/pip-build-X4mzal/theano/theano/gradient.pyRÄs  cC`s t|ƒS(sÿ Consider an expression constant when computing gradients, while effectively not backpropagating through it. The expression itself is unaffected, but when its gradient is computed, or the gradient of another expression that this expression is a subexpression of, it will not be backpropagated through. This is effectively equivalent to truncating the gradient expression to 0, but is executed faster than zero_grad(), which stilll has to go through the underlying computational graph related to the expression. :param x: A Theano expression whose gradient should not be backpropagated through. :return: The expression is returned unmodified, but its gradient is now effectively truncated to 0. (tdisconnected_grad_(R((s//tmp/pip-build-X4mzal/theano/theano/gradient.pytdisconnected_gradÒstGradClipcB`s eZdZd„Zd„ZRS(cC`s.||_||_|j|jks*t‚dS(N(tclip_lower_boundtclip_upper_boundR(R"R…R†((s//tmp/pip-build-X4mzal/theano/theano/gradient.pyRìs  cC`s/g|D]$}tjj||j|jƒ^qS(N(R@RAtclipR…R†(R"ROR{R|((s//tmp/pip-build-X4mzal/theano/theano/gradient.pyRcós((R0R1t __props__RRc(((s//tmp/pip-build-X4mzal/theano/theano/gradient.pyR„ès cC`st||ƒ|ƒS(s­ This op do a view in the forward, but clip the gradient. This is an elemwise operation. :param x: the variable we want its gradient inputs clipped :param lower_bound: The lower bound of the gradient value :param upper_bound: The upper bound of the gradient value. :examples: x = theano.tensor.scalar() z = theano.tensor.grad(grad_clip(x, -1, 1)**2, x) z2 = theano.tensor.grad(x**2, x) f = theano.function([x], outputs = [z, z2]) print(f(2.0)) # output (1.0, 4.0) :note: We register an opt in tensor/opt.py that remove the GradClip. So it have 0 cost in the forward and only do work in the grad. (R„(Rt lower_boundt upper_bound((s//tmp/pip-build-X4mzal/theano/theano/gradient.pyt grad_clipùst GradScalecB`seZd„Zd„ZRS(cC`s ||_dS(N(t multiplier(R"R((s//tmp/pip-build-X4mzal/theano/theano/gradient.pyRscC`sg|D]}|j|^qS(N(R(R"ROR{R|((s//tmp/pip-build-X4mzal/theano/theano/gradient.pyRcs(R0R1RRc(((s//tmp/pip-build-X4mzal/theano/theano/gradient.pyRŒs cC`st|ƒ|ƒS(s$ This op scale or inverse the gradient in the backpropagation. :param x: the variable we want its gradient inputs scale :param multiplier: scale of the gradient :examples: x = theano.tensor.fscalar() fx = theano.tensor.sin(x) fp = theano.tensor.grad(fx, wrt=x) fprime = theano.function([x], fp) print(fprime(2))#-0.416 f_inverse=grad_scale(fx,-1.) fpp = theano.tensor.grad(f_inverse, wrt=x) fpprime = theano.function([x], fpp) print(fpprime(2))#0.416 (RŒ(RR((s//tmp/pip-build-X4mzal/theano/theano/gradient.pyt grad_scales(\R2t __future__RRRt collectionsRtsix.moves.builtinstmovesRtloggingRuRmRÿtsixRR@Rt theano.gofRRt theano.compatRt six.movesR R ttheano.gof.null_typeR R t theano.gof.opR ttheano.compileRRRRut __authors__t __copyright__t __license__t __contact__t __docformat__t getLoggert_loggerR3RAt _msg_retTypeR…RRR R=tTypeR!RR]RgR4RcR)R R¦RzR?RƒRQRoR‚R„R{tobjectRöR9RLR;RaRMRpRtR¾RzR}RaR~RR€RR‚RƒR„R‹RŒRŽ(((s//tmp/pip-build-X4mzal/theano/theano/gradient.pyts’         )  ‘ 5 Ù‘ ' “ ÿŠ Î    ¦  H C