Generate camera rays (origin and direction) in the world coordinate given uv coordinate on the image. The uv coordinate on image is origin at top left, u to right, v to bottom. Args: cam_poses: (m, 4, 4) homegeneous matrix that transforms the camera coord to wor
(
cam_poses: torch.Tensor, # (m, 4, 4) target camera pose, cam to world
intrinsics: torch.Tensor, # (m, 3, 3) intrinsic matrix of the camrea
uv: torch.Tensor, # (m, *p, 2)
device=torch.device('cpu'),
)
| 1612 | |
| 1613 | |
| 1614 | def generate_camera_rays_from_uv( |
| 1615 | cam_poses: torch.Tensor, # (m, 4, 4) target camera pose, cam to world |
| 1616 | intrinsics: torch.Tensor, # (m, 3, 3) intrinsic matrix of the camrea |
| 1617 | uv: torch.Tensor, # (m, *p, 2) |
| 1618 | device=torch.device('cpu'), |
| 1619 | ) -> T.Union[torch.Tensor, torch.Tensor]: |
| 1620 | """ |
| 1621 | Generate camera rays (origin and direction) in the world coordinate given |
| 1622 | uv coordinate on the image. The uv coordinate on image is origin at top left, |
| 1623 | u to right, v to bottom. |
| 1624 | |
| 1625 | Args: |
| 1626 | cam_poses: |
| 1627 | (m, 4, 4) homegeneous matrix that transforms the camera coord to world coord |
| 1628 | Note that to use the rays to render an image, you can have the y axis inverted in the cam_poses. |
| 1629 | intrinsics: |
| 1630 | (m, 3, 3) intrinsic matrix for each camera pose |
| 1631 | width_px: |
| 1632 | number of pixels on the sensor (before subsample) |
| 1633 | height_px: |
| 1634 | number of pixels on the sensor (before subsample) |
| 1635 | uv: |
| 1636 | uv coordinate. uv[..., 0]: u coordinate [0, w], uv[..., 1]: v coordinate [0, h], |
| 1637 | device: |
| 1638 | |
| 1639 | Returns: |
| 1640 | ray_origins_w: (m, *p, 3) |
| 1641 | ray_directions_w: (m, *p, 3) normalized to unit norm |
| 1642 | |
| 1643 | Note: |
| 1644 | The function does not flip the y axis (which should be handled by the image coordinate) |
| 1645 | To use the rays to render an image, you can have the y axis inverted in the cam_poses. |
| 1646 | """ |
| 1647 | |
| 1648 | m = cam_poses.size(0) |
| 1649 | _m, *p_shape, _2 = uv.shape |
| 1650 | assert m == _m |
| 1651 | assert _2 == 2 |
| 1652 | |
| 1653 | uv1 = torch.cat( |
| 1654 | ( |
| 1655 | uv, |
| 1656 | torch.ones(_m, *p_shape, 1, dtype=uv.dtype, device=uv.device) |
| 1657 | ), dim=-1).to(dtype=cam_poses.dtype, device=device) # (m, *p, 3) |
| 1658 | |
| 1659 | # compute the inverse of the intrinsic matrices |
| 1660 | inv_intrinsics = torch.linalg.inv(intrinsics.to(device=device)) # (m, 3, 3) |
| 1661 | inv_intrinsics = inv_intrinsics.reshape(m, *([1] * len(p_shape)), 3, 3) # (m, 1, 1, 3, 3) |
| 1662 | ray_directions_c = inv_intrinsics @ uv1.unsqueeze(-1) # (m, *p, 3, 1) |
| 1663 | |
| 1664 | # cam coord -> world coord |
| 1665 | cam_poses = cam_poses.reshape(m, *([1] * len(p_shape)), 4, 4).to(device=device) # (m, *p, 4, 4) |
| 1666 | ray_directions_w = cam_poses[..., :3, :3] @ ray_directions_c # (m, *p, 3, 1), not normalized |
| 1667 | ray_origins_w = cam_poses[..., :3, 3].clone().expand(m, *p_shape, 3) # (m, *p, 3) |
| 1668 | |
| 1669 | # normalize direction |
| 1670 | ray_directions_w = ray_directions_w.squeeze(-1) # (m, *p, 3) |
| 1671 | ray_directions_w = ray_directions_w / torch.linalg.vector_norm( |