ImageMarker

Find the four corners of Aruco markers in an image, using OpenCV. The upper-left corner is the “origin” of the marker (idx=0). The ImageMarker class can hold information about just the origin point of the markers, or it can be converted to a four point model which uses all the corners. By default, the model loads an image using the one point model. In the one point model, the point ID is the aruco marker ID. In the four point model, the point ID is the Aruco marker ID * 4 plus the corner index (ranging from 0 to 4).

class opencsp.common.lib.photogrammetry.ImageMarker.ImageMarker(image: ndarray, point_ids: ndarray[int], pts_im_xy: ndarray, img_id: int, camera: Camera)

Bases: object

Class to hold images of Aruco markers. Contains methods to process locations of Aruco markers.

__init__(image: ndarray, point_ids: ndarray[int], pts_im_xy: ndarray, img_id: int, camera: Camera)

Instantiates ImageMarker class.

Parameters:

  • image (ndarray) – 2D image array.

  • point_ids (ndarray) – 1d array of point IDs.

  • pts_im_xy (ndarray) – Nx2 array of point locations in image.

  • img_id (int) – ID of image.

  • camera (opencsp.common.lib.camera.Camera.Camera) – Camera object of camera that captured image.

attempt_calculate_pose() → int

Calculates pose of camera if enough points in image are located

Returns:

0=already known, 1=calculated successfully, -1=not calculated successfully

Return type:

int

calc_reprojection_errors() → ndarray

Calculates reprojection error for all located points

Returns:

Shape (N,) array of reprojection errors. N=number of image points

Return type:

ndarray

convert_to_four_corner(**kwargs) → None

Converts from using only origin point to all four marker corners

classmethod load_aruco_origin(file: str, img_id: int, camera: Camera, **kwargs) → ImageMarker

Loads an image file, finds Aruco markers, saves the origin point.

Parameters:
Return type:

ImageMarker

plot_image_with_points() → None

Plots captured image with image point and reprojected point locations

save_as_hdf(file: str) → None

Writes all information to given HDF file

set_point_id_located(id_: int, pt: ndarray) → None

Sets given point ID as located_markers_mask

set_point_id_unlocated(id_: int) → None

Sets given point ID as unlocated

set_pose(rvec: ndarray, tvec: ndarray) → None

Sets the pose of the camera known

Parameters:

  • rvec (ndarray) – Shape (3,) rotation vector

  • tvec (ndarray) – Shape (3,) translation vector

unset_pose() → None

Removes the pose of the camera

property located_marker_points_image: ndarray

Returns only located_markers_mask image points of markers

property located_marker_points_object: ndarray

Returns only located_markers_mask object points of markers

property located_point_ids: ndarray

Returns only located_markers_mask marker IDs

property num_located_markers: int

Returns number of located markes

property num_markers: int

Returns number of total markers

property unlocated_point_ids: ndarray

Returns only unlocated marker IDs

Bundle Adjustment

Bundle adjustment algorithm based on example at: https://scipy-cookbook.readthedocs.io/items/bundle_adjustment.html

opencsp.common.lib.photogrammetry.bundle_adjustment.bundle_adjust(rvecs: ndarray, tvecs: ndarray, pts_obj: ndarray, camera_indices: ndarray, point_indices: ndarray, pts_img: ndarray, intrinsic_mat: ndarray, dist_coefs: ndarray, opt_type: Literal['camera', 'points', 'both'], verbose: int) → tuple[ndarray, ndarray, ndarray]

Perform bundel adjustment algorithm on object points and camera poses. Nim = number of images Npts = number of points Nobs = number of observations

Parameters:

  • rvecs (np.ndarray) – (Nim,3) rvecs.

  • tvecs (np.ndarray) – (Nim,3) tvecs.

  • pts_obj (np.ndarray) – (Npts,3) 3D object point array.

  • camera_indices (np.ndarray) – (Nobs,) array.

  • point_indices (np.ndarray) – (Nobs,) array.

  • pts_img (np.ndarray) – (Nobs,2) image point array.

  • intrinsic_mat (np.ndarray) – (3,3) intrinsic camera matrix.

  • dist_coefs (np.ndarray) – (n,) distortion coefficients array.

  • opt_type (str) – What to optimize: {‘camera’, ‘points’, ‘both’}

  • verbose (int) – Level of verbosity of least squares solver [0, 1, 2]

Returns:

  • rvecs_opt (np.ndarray) – (Nim,3) optimized rvecs.

  • tvecs_opt (np.ndarray) – (Nim,3) optimized tvecs.

  • pts_obj_opt (np.ndarray) – (Nimx,3) optimized object points.

opencsp.common.lib.photogrammetry.bundle_adjustment.bundle_adjustment_sparsity(n_cameras: int, n_points: int, camera_indices: ndarray, point_indices: ndarray, opt_type: str)

Returns Jacobian sparsity structure.

opencsp.common.lib.photogrammetry.bundle_adjustment.fun(params: ndarray, n_cameras: int, n_points: int, camera_indices: ndarray, point_indices: ndarray, points_2d: ndarray, intrinsic_mat: ndarray, dist_coefs: ndarray) → ndarray

Compute reprojection errors in x and y.

opencsp.common.lib.photogrammetry.bundle_adjustment.project(points: ndarray, camera_params: ndarray, intrinsic_mat: ndarray, dist_coefs: ndarray)

Convert 3-D points to 2-D by projecting onto camera sensor. A ray with normal incidence has a (0, 0) coordinate.

opencsp.common.lib.photogrammetry.bundle_adjustment.rotate(points: ndarray, rot_vecs: ndarray)

Rotate points by given rotation vectors.

Photogrammetry

Library of photogrammetry-related functions and algorithms

opencsp.common.lib.photogrammetry.photogrammetry.align_points(pts_obj: Vxyz, vals: Vxyz, scale: bool = False) → tuple[TransformXYZ, float, ndarray[float]]

Returns 2D homogeneous transform to apply to input data according to alignment criteria. Values are scaled (if applicable) FIRST, then spatially transformed.

Parameters:

  • pts_obj (Vxyz) – Object points, meters.

  • vals (Vxyz) –

    [(x1, y1, z1), (x2, y2, z2), ...] The expected coordinate values of each coordinate index that correspond to points in pts_obj. If a coordinate is to be ignored, set it to np.nan.

    For example: [(np.nan, 0, np.nan), (np.nan, 0, np.nan), (0, 0, np.nan)]

  • scale (bool) – To apply a scaling factor to points (default False)

Returns:

Point cloud Transform object Point cloud scale factor Point alignment error, meters

Return type:

TransformXYZ, float, ndarray[float]

opencsp.common.lib.photogrammetry.photogrammetry.dist_from_rays(v_pt: Vxyz, u_ray_dir: Vxyz | Uxyz, v_ray_ori: Vxyz) → ndarray

Calculates perpendicular distances from a point to N rays.

Parameters:

  • v_pt (Vxyz) – Length 1, Point to calculate distance to.

  • u_ray_dir (Vxyz | Uxyz) – Length N, Ray pointing directions. Must be a UNIT VECTOR.

  • v_ray_ori (Vxyz) – Length N, Ray origin points.

Returns:

Length N array of distances, meters.

Return type:

np.ndarray

opencsp.common.lib.photogrammetry.photogrammetry.find_aruco_marker(image: ndarray, adaptiveThreshConstant: float = 10, minMarkerPerimeterRate: float = 0.01) → tuple[ndarray[int], list[ndarray]]

Finds aruco marker corners in given image to the nearest pixel.

Parameters:

  • image (ndarray) – 2D grayscale image.

  • adaptiveThreshConstant (float, optional) – aruco parameter. The default is 10.

  • minMarkerPerimeterRate (float, optional) – aruco parameter. The default is 0.01.

Returns:

  • ids (ndarray[int, …]) – 1D array of IDs of aruco markers seen in each image.

  • pts (list[ndarray, …]) – List of 4x2 image point arrays of all four corners of aruco markers seen by each camera.

opencsp.common.lib.photogrammetry.photogrammetry.load_image_grayscale(file: str) → ndarray

Loads image. Converts to grayscale if needed

opencsp.common.lib.photogrammetry.photogrammetry.nearest_ray_intersection(p_origins: Vxyz, u_dirs: Vxyz | Uxyz) → tuple[Vxyz, ndarray]

Finds the least squares point of intersection between N skew rays. And calculates residuals. Source: https://en.wikipedia.org/wiki/Line-line_intersection

Parameters:

  • p_origins (Vxyz) – Vector of XYZ origin of each ray.

  • u_dirs (Vxyz | Uxyz) – Vector of unit pointing direction vectors. Must be UNIT VECTORS.

Returns:

Least squares XYZ intersection point, perpendicular distances from point to rays

Return type:

Vxyz, ndarray

opencsp.common.lib.photogrammetry.photogrammetry.plot_pts_3d(ax: Axes, pts_obj: ndarray, rots: list[Rotation], tvecs: Vxyz, needle_length: float = 1) → None

Plots 3D points and camera poses (points with needles defined by rvec/tvec).

Parameters:

  • ax (plt.Axes) – 3D axes to plot on.

  • pts_obj (ndarray) – Nx3 object points.

  • rots (list[Rotation]) – N rotation objects.

  • tvecs (Vxyz) – N tvecs.

  • needle_length (float, optional) – Length of camera pointing needles, meters. The default is 1.

opencsp.common.lib.photogrammetry.photogrammetry.reprojection_errors(rvecs: ndarray, tvecs: ndarray, pts_obj: ndarray, camera: Camera, camera_indices: ndarray, point_indices: ndarray, points_2d: ndarray) → ndarray

Calculates the xy reprojection error for each point. See bundle_adjustment.bundle_adjust() for more information.

Returns:

Num observations x 2 ndarray.

Return type:

ndarray

opencsp.common.lib.photogrammetry.photogrammetry.scale_points(pts_obj: Vxyz, point_ids: ndarray, point_pairs: ndarray, dists: ndarray) → ndarray[float]

Scales object points and tvecs. A list of point pairs is given, and the corresponding expected distance between them. The object points are scaled to the average of the calculated scales.

Parameters:

  • pts_obj (Vxyz) – Object xyz points, meters.

  • point_ids (ndarray) – 1d array, point IDs corresponding to pts_obj.

  • point_pairs (ndarray) – 2d array of form: [[a1, a2], [b1, b2], …], point pairs.

  • dists (ndarray) – 1d array of expected distances between point pairs, meters.

Returns:

scales – Calculated scale factor for each point pair.

Return type:

ndarray

opencsp.common.lib.photogrammetry.photogrammetry.triangulate(cameras: list[Camera], rots: list[Rotation], tvecs: Vxyz | list[Vxyz], pts_img: Vxy | list[Vxy]) → tuple[Vxyz, ndarray]

Triangulates position of unknown marker.

Parameters:

  • cameras (list[opencsp.common.lib.camera.Camera.Camera]) – N Camera objects used to capture images

  • rots (list[Rotation]) – N world to camera Rotations

  • tvecs (Vxyz | list[Vxyz]) – N camera to world translation vectors (camera coordinates)

  • pts_img (Vxy | list[Vxy]) – N Vxy image points

Returns:

Intersection point, perpendicular distances from point to rays

Return type:

tuple[Vxyz, ndarray]

opencsp.common.lib.photogrammetry.photogrammetry.valid_camera_pose(camera: Camera, rvec: ndarray, tvec: ndarray, pts_image: ndarray, pts_object: ndarray, reproj_thresh: float = 100.0) → bool

Returns image IDs that have points behind the camera or have high reprojection error.

Parameters:

  • camera (opencsp.common.lib.camera.Camera.Camera) – Camera object.

  • rvecs (ndarray) – Shape (3,) rvec array.

  • tvecs (ndarray) – Shape (3,) tvec array.

  • pts_image (ndarray) – Nx2 array of 2d image points

  • pts_object (ndarray) – Nx3 array of 3d points to project.

  • reproj_thresh (float, optional) – Threshold reprojection error. The default is 100.

Returns:

True if camera pose is valid

Return type:

bool