Multiply the Hessian of `ys` wrt `xs` by `v`. This is an efficient construction that uses a backprop-like approach to compute the product between the Hessian and another vector. The Hessian is usually too large to be explicitly computed or even represented, but this method allows us to at l
(ys, xs, v)
| 277 | |
| 278 | # TODO(vrv): Make this available when we want to make it public. |
| 279 | def _hessian_vector_product(ys, xs, v): |
| 280 | """Multiply the Hessian of `ys` wrt `xs` by `v`. |
| 281 | |
| 282 | This is an efficient construction that uses a backprop-like approach |
| 283 | to compute the product between the Hessian and another vector. The |
| 284 | Hessian is usually too large to be explicitly computed or even |
| 285 | represented, but this method allows us to at least multiply by it |
| 286 | for the same big-O cost as backprop. |
| 287 | |
| 288 | Implicit Hessian-vector products are the main practical, scalable way |
| 289 | of using second derivatives with neural networks. They allow us to |
| 290 | do things like construct Krylov subspaces and approximate conjugate |
| 291 | gradient descent. |
| 292 | |
| 293 | Example: if `y` = 1/2 `x`^T A `x`, then `hessian_vector_product(y, |
| 294 | x, v)` will return an expression that evaluates to the same values |
| 295 | as (A + A.T) `v`. |
| 296 | |
| 297 | Args: |
| 298 | ys: A scalar value, or a tensor or list of tensors to be summed to |
| 299 | yield a scalar. |
| 300 | xs: A list of tensors that we should construct the Hessian over. |
| 301 | v: A list of tensors, with the same shapes as xs, that we want to |
| 302 | multiply by the Hessian. |
| 303 | |
| 304 | Returns: |
| 305 | A list of tensors (or if the list would be length 1, a single tensor) |
| 306 | containing the product between the Hessian and `v`. |
| 307 | |
| 308 | Raises: |
| 309 | ValueError: `xs` and `v` have different length. |
| 310 | |
| 311 | """ |
| 312 | |
| 313 | # Validate the input |
| 314 | length = len(xs) |
| 315 | if len(v) != length: |
| 316 | raise ValueError("xs and v must have the same length.") |
| 317 | |
| 318 | # First backprop |
| 319 | grads = gradients(ys, xs) |
| 320 | |
| 321 | assert len(grads) == length |
| 322 | elemwise_products = [ |
| 323 | math_ops.multiply(grad_elem, array_ops.stop_gradient(v_elem)) |
| 324 | for grad_elem, v_elem in zip(grads, v) |
| 325 | if grad_elem is not None |
| 326 | ] |
| 327 | |
| 328 | # Second backprop |
| 329 | return gradients(elemwise_products, xs) |
| 330 | |
| 331 | |
| 332 | @tf_export(v1=["hessians"]) |