Dictionaries

opencsp.common.lib.tool.dict_tools.list_of_values_in_sorted_key_order(input_dict)

Return a list of values from the dictionary in the order of sorted keys.

Parameters:: input_dict (dict) – The dictionary from which to extract values.
Returns:: A list of values corresponding to the sorted keys of the input dictionary.
Return type:: list

opencsp.common.lib.tool.dict_tools.number_of_keys(input_dict)

Return the number of keys in the given dictionary.

Parameters:: input_dict (dict) – The dictionary for which to count the keys.
Returns:: The number of keys in the input dictionary.
Return type:: int

opencsp.common.lib.tool.dict_tools.one_level_dict_csv_data_line(key_list, one_level_dict)

Generate a CSV data line from a one-level dictionary.

Parameters:

key_list (list) – A list of keys to extract values from the dictionary.
one_level_dict (dict) – The one-level dictionary to convert to a CSV data line.

Returns:

A string representing the CSV data line.

Return type:

str

opencsp.common.lib.tool.dict_tools.one_level_dict_csv_heading_line(list_of_one_level_dicts, first_key)

Generate the heading line for a CSV file from a list of one-level dictionaries.

Parameters:

list_of_one_level_dicts (list of dict) – A list of one-level dictionaries to extract headings from.
first_key (str, optional) – A key to prioritize in the heading line.

Returns:

A tuple containing the list of keys and the heading line as a string.

Return type:

tuple

opencsp.common.lib.tool.dict_tools.one_level_dict_csv_heading_line_aux(first_one_level_dict, first_key)

Auxiliary function to generate the heading line for a CSV file from a one-level dictionary.

Parameters:

first_one_level_dict (dict) – The first one-level dictionary to extract headings from.
first_key (str, optional) – A key to prioritize in the heading line.

Returns:

A tuple containing the list of keys and the heading line as a string.

Return type:

tuple

opencsp.common.lib.tool.dict_tools.one_level_dict_pair_csv_data_line(key_list_1, key_list_2, one_level_dict_pair)

Generate a CSV data line from a pair of one-level dictionaries.

Parameters:

key_list_1 (list) – A list of keys to extract values from the first dictionary.
key_list_2 (list) – A list of keys to extract values from the second dictionary.
one_level_dict_pair (list) – A pair of one-level dictionaries to convert to a CSV data line.

Returns:

A string representing the CSV data line for the pair of dictionaries.

Return type:

str

opencsp.common.lib.tool.dict_tools.one_level_dict_pair_csv_heading_line(list_of_one_level_dict_pairs, first_key_1, first_key_2)

Generate the heading line for a CSV file from a list of one-level dictionary pairs.

This routine extracts the headings from each of the two dicts, adding suffixes “_1” or “_2” to the first and second dictionary keys, respectively. These suffixes are added regardless of whether the dictionaries have the same or different keys.

Parameters:

list_of_one_level_dict_pairs (list of list) – A list of pairs of one-level dictionaries to extract headings from.
first_key_1 (str, optional) – A key to prioritize in the heading line for the first dictionary.
first_key_2 (str, optional) – A key to prioritize in the heading line for the second dictionary.

Returns:

A tuple containing the list of keys for the first dictionary, the list of keys for the second dictionary, and the combined heading line as a string.

Return type:

tuple

opencsp.common.lib.tool.dict_tools.print_dict(input_dict, max_keys=10, max_value_length=70, indent=None)

Print a simple dictionary, limiting the output length both laterally and vertically.

Parameters:

input_dict (dict) – The dictionary to print.
max_keys (int, optional) – The maximum number of keys to print. An ellipsis will be shown if there are more keys.
max_value_length (int, optional) – The maximum length of values to print. An ellipsis will be shown if values exceed this length.
indent (int, optional) – The number of spaces to print at the beginning of each line for indentation.

Return type:

None

opencsp.common.lib.tool.dict_tools.print_dict_of_dict_of_dicts(input_dict, max_keys_1=5, max_keys_2=5, max_keys_3=8, max_value_3_length=70, indent_1=0, indent_2=4, indent_3=4)

Print a two-level nested dictionary, limiting the output length both laterally and vertically.

Parameters:

input_dict (dict) – The dictionary to print, where values are dictionaries that themselves contain dictionaries.
max_keys_1 (int, optional) – The maximum number of top-level keys to print. An ellipsis will be shown if there are more keys.
max_keys_2 (int, optional) – The maximum number of second-level keys to print. An ellipsis will be shown if there are more keys.
max_keys_3 (int, optional) – The maximum number of third-level keys to print. An ellipsis will be shown if there are more keys.
max_value_3_length (int, optional) – The maximum length of third-level values to print. An ellipsis will be shown if values exceed this length.
indent_1 (int, optional) – The number of spaces to print at the beginning of each top-level line for indentation.
indent_2 (int, optional) – The number of additional spaces to print for each second-level line.
indent_3 (int, optional) – The number of additional spaces to print for each third-level line.

Return type:

None

opencsp.common.lib.tool.dict_tools.print_dict_of_dicts(input_dict, max_keys_1=5, max_keys_2=8, max_value_2_length=70, indent_1=0, indent_2=4)

Print a one-level nested dictionary, limiting the output length both laterally and vertically.

Parameters:

input_dict (dict) – The dictionary to print, where values are themselves dictionaries.
max_keys_1 (int, optional) – The maximum number of top-level keys to print. An ellipsis will be shown if there are more keys.
max_keys_2 (int, optional) – The maximum number of second-level keys to print. An ellipsis will be shown if there are more keys.
max_value_2_length (int, optional) – The maximum length of second-level values to print. An ellipsis will be shown if values exceed this length.
indent_1 (int, optional) – The number of spaces to print at the beginning of each top-level line for indentation.
indent_2 (int, optional) – The number of additional spaces to print for each second-level line.

Return type:

None

opencsp.common.lib.tool.dict_tools.save_list_of_one_level_dict_pairs(list_of_one_level_dict_pairs, output_dir, output_body, explain, error_if_dir_not_exist, first_key_1=None, first_key_2=None)

Save a list of one-level dictionary pairs to a CSV file, one dict per line.

Parameters:

list_of_one_level_dict_pairs (list of list) – A list of pairs of one-level dictionaries to save to the CSV file.
output_dir (str) – The directory where the CSV file will be saved.
output_body (str) – The body of the output filename (without extension).
explain (str) – An explanatory string to include in notification output.
error_if_dir_not_exist (bool) – If True, raise an error if the output directory does not exist; if False, create the directory if necessary.
first_key_1 (str, optional) – A key to prioritize in the heading line for the first dictionary.
first_key_2 (str, optional) – A key to prioritize in the heading line for the second dictionary.

Returns:

The full path to the saved CSV file.

Return type:

str

opencsp.common.lib.tool.dict_tools.save_list_of_one_level_dicts(list_of_one_level_dicts, output_dir, output_body, explain, error_if_dir_not_exist, first_key=None)

Save a list of one-level dictionaries to a CSV file, one dictionary per line.

Parameters:

list_of_one_level_dicts (list of dict) – A list of dictionaries to save to the CSV file.
output_dir (str) – The directory where the CSV file will be saved.
output_body (str) – The body of the output filename (without extension).
explain (str) – An explanatory string to include in notification output.
error_if_dir_not_exist (bool) – If True, raise an error if the output directory does not exist; if False, create the directory if necessary.
first_key (str, optional) – A key to prioritize in the CSV heading line.

Returns:

The full path to the saved CSV file.

Return type:

str

opencsp.common.lib.tool.dict_tools.sorted_keys(input_dict)

Return the keys of the dictionary in a sorted list.

Parameters:: input_dict (dict) – The dictionary whose keys are to be sorted.
Returns:: A sorted list of the keys in the input dictionary.
Return type:: list

Notes

This function constructs a list of keys and sorts it, which may be slower than using keys(), but provides the keys in a predictable order.

Lists

Utilities for list processing.

opencsp.common.lib.tool.list_tools.binary_search(sorted_list: list[_T], search_val: _V, comparator: Callable[[_T, _V], int] | None = None, key: Callable[[_T], _V] | None = None, err_if_not_equal=False)

Does a binary search to get the index and item in the list corresponding to the given search_val.

This only works when sorted_list is sorted by the desired search key. If it is not, then there is no guarantee that this will return anything useful and list.indexof should probably be used instead.

If _T is a number, then this functionality is extended to return the closest item when there isn’t an exact match.

Parameters:

sorted_list (-) – A list that is sorted by its natural order (by the desired attribute if a list of objects).
search_val (-) – The desired value. Must either be of type _T, or comparator must be defined.
comparator (-) – A function to order list values compared to the search_val (-1 if T<V, 1 if T>V, 0 if equal). Useful for lists of objects. Defaults to None.
key (-) – A function to return the numeric value used for sorting. Useful for lists of objects when you don’t need a fancy comparator. Defaults to None.
err_if_not_equal (-) – If the value to be returned is not equal to the search value, then error out. Defaults to false.

Returns:

index of the list for the closest matching - _T: the closest matching item

Return type:

int

opencsp.common.lib.tool.list_tools.contains_duplicates(list)

Check if a list contains any duplicate elements.

Parameters:: list (list) – The input list to check for duplicates.
Returns:: True if the list contains duplicates, False otherwise.
Return type:: bool

Examples

>>> contains_duplicates([1, 2, 3, 4])
False
>>> contains_duplicates([1, 2, 2, 3])
True

opencsp.common.lib.tool.list_tools.get_range(data_keys: list[float], data_values: list[_T], key_subset_range: tuple[float, float], exclude_outside_range=False) → tuple[list[float], list[_T]]

Select a subset of the data_keys[] and data_values[], limited to the given key_subset_range. Chooses the keys closest to the given range start and end points, inclusively.

Ideally, the input data_keys and data_values lists should be the same length. The returned data_keys and data_values lists should be the same length.

Parameters:

data_keys (list[float]): The list of keys for the corresponding data_values points in “data_values”. This list must be sorted.

data (list): The data_values to get a subset of.

key_subset_range (tuple[float,float]): The start and end range to include in the returned subset.

exclude_outside_range (bool, optional): If False, then values outside the range could be included based on the closest key match.
If True, then only values in the range will be returned. Default False.

Returns:

data_keys_subset (list[float]): A subset of the input data_keys, approximately limited to the given key_subset_range

data_values_subset (list[float]): A subset of the input data_values, approximately limited to the given key_subset_range

opencsp.common.lib.tool.list_tools.natural_sort(values: list[str])

Sorts the given list naturally, so that numbers are sorted from lowest to highest.

Adapted from https://stackoverflow.com/questions/11150239/natural-sorting

opencsp.common.lib.tool.list_tools.print_list(input_list, max_items=10, max_item_length=70, indent=None): Prints a list, limiting print-out length both laterally and vertically.

opencsp.common.lib.tool.list_tools.remove_duplicates(list)

Remove duplicate elements from a list while preserving the original order.

Parameters:: list (list) – The input list from which duplicates will be removed.
Returns:: A new list containing the unique elements from the input list, in the order they first appeared.
Return type:: list

Examples

>>> remove_duplicates([1, 2, 2, 3, 4, 4, 5])
[1, 2, 3, 4, 5]
>>> remove_duplicates(['a', 'b', 'a', 'c'])
['a', 'b', 'c']

opencsp.common.lib.tool.list_tools.rindex(values: list, needle): Like values.index(needle), but search for the last occurance.

opencsp.common.lib.tool.list_tools.zamboni_shuffle(segment_list)

Shuffle elements so input consecutive elements are separated. Inspired by the path of a Zamboni in an ice rink.

Examples

zamboni_shuffle([1,2,3,4,5,6]) –> [1, 4, 2, 5, 3, 6]

zamboni_shuffle([1,2,3,4,5,6,7]) –> [1, 5, 2, 6, 3, 7, 4]

Strings

Files for string manipulation.

opencsp.common.lib.tool.string_tools.add_to_last_sentence(base: str, add: str) → str

Adds “add” string to “base” string, inserting before final period if present.

Example 1:: base: “My initial caption.” add: “, plus more” –> “My initial caption, plus more.”
Example 2:: base: “Characters ABC” add: “DEF” –> “Characters ABCDEF”

opencsp.common.lib.tool.string_tools.camel_case_split(to_split: str) → list[str]

Splits the given string into pieces at leading uppercase letters.

For example:

camel_case_split(“TheABCsOfPython”) # [‘The’, ‘ABCs’, ‘Of’, ‘Python’]

Parameters:: to_split (str) – The CamelCase string to be split
Returns:: The to_split string, split into camel case sections.
Return type:: list[str]

opencsp.common.lib.tool.string_tools.convert_true_false_string_to_boolean(true_or_false_str: str) → bool

Accepts string with value either “True” or “False” and returns corresponding Boolean value. Throws an error if not one of these two values.

Parameters:: true_or_false_str (str) – The string to convert to Boolean. Must be “True” or “False”.
Returns:: The corresponding value as a Boolean type.
Return type:: bool

opencsp.common.lib.tool.string_tools.float_or_none(num_or_none_str: str) → bool

Accepts string with value either “None” or a valid floating-point number (e.g., “-1.234”) and returns either the corresponding number, or None. Throws an error if not one of these two cases.

Parameters:: num_or_none_str (str) – The string to convert to floating point. Must be either a valid number, or “None”.
Returns:: The corresponding value as a float type, or None.
Return type:: float | None

opencsp.common.lib.tool.string_tools.verify_contiguous(token_str: str) → str

Checks string to ensure that is a contiguous string with no white space, newlines, etc. Throws an error if any white space is found.

Parameters:: token_str (str) – The string to check.
Returns:: The input string, assuming it passes the check. (Throws an error if not.)
Return type:: str

Dates

Convenience functions for manipulating time and dates.

opencsp.common.lib.tool.time_date_tools.add_seconds_to_ymdhmsz(ymdhmsz: list[int, int, int, int, int, float, int], time_sec: float, ignore_legacy=None) → list[int, int, int, int, int, float, int]

Add a specified number of seconds to a date and time represented as a list.

The input list is expected to be in the format: [year, month, day, hour, minute, second, zone].

Parameters:

ymdhmsz (list[int, int, int, int, int, float, int]) – A list representing the date and time, where: - year (int): The year. - month (int): The month (1-12). - day (int): The day of the month (1-31). - hour (int): The hour (0-23). - minute (int): The minute (0-59). - second (float): The second (0-59.999…). - zone (int): The time zone offset.
time_sec (float) – The number of seconds to add to the date and time.
ignore_legacy (bool, optional) – If set to False, a warning message will be printed indicating that this function is a legacy function.

Returns:

A new list representing the updated date and time after adding the specified seconds.

Return type:

list[int, int, int, int, int, float, int]

Raises:

AssertionError – If rolling over a day boundary is not implemented.

Notes

This function currently does not handle rolling over a day boundary or month boundary.

Examples

>>> add_seconds_to_ymdhmsz([2023, 3, 15, 12, 30, 45.0, 0], 30)
[2023, 3, 15, 12, 31, 15.0, 0]

opencsp.common.lib.tool.time_date_tools.current_date_string(now: date | None = None) → str: %Y-%m-%d

opencsp.common.lib.tool.time_date_tools.current_date_string_forfile() → str

Get the current date as a formatted string suitable for filenames.

The format of the string is “%Y%m%d”.

Returns:: The current date as a string suitable for use in filenames.
Return type:: str

opencsp.common.lib.tool.time_date_tools.current_date_time_string() → str: %Y-%m-%d %H:%M:%S

opencsp.common.lib.tool.time_date_tools.current_date_time_string_forfile() → str: %Y%m%d_%H%M%S

opencsp.common.lib.tool.time_date_tools.current_time() → float

Get the current time in seconds since the epoch.

Returns:: The current time in seconds since the epoch (Unix time).
Return type:: float

opencsp.common.lib.tool.time_date_tools.current_time_string(now: <module 'time' (built-in)> = None) → str: %H:%M:%S

opencsp.common.lib.tool.time_date_tools.current_time_string_forfile() → str

Get the current time as a formatted string suitable for filenames.

The format of the string is “%H%M%S”.

Returns:: The current time as a string suitable for use in filenames.
Return type:: str

opencsp.common.lib.tool.time_date_tools.date_string_forfile(d: date) → str: %Y%m%d

opencsp.common.lib.tool.time_date_tools.date_time_string_forfile(dt: datetime) → str: %Y%m%d_%H%M%S

opencsp.common.lib.tool.time_date_tools.elapsed_time_since_start_sec(start_time: float) → float

Calculate the elapsed time in seconds since a given start time.

Parameters:: start_time (float) – The start time in seconds since the epoch.
Returns:: The elapsed time in seconds since the start time.
Return type:: float

opencsp.common.lib.tool.time_date_tools.from_datetime(dt: datetime) → list[int, int, int, int, int, float, int]: Converts from a datetime instance to a list of [year,month,day,hour,minute,second,utcoffset]

opencsp.common.lib.tool.time_date_tools.print_current_date() → None

Print the current date to the log.

Return type:: None

opencsp.common.lib.tool.time_date_tools.print_current_date_time() → None

Print the current date and time to the log.

Return type:: None

opencsp.common.lib.tool.time_date_tools.print_current_time() → None

Print the current time to the log.

Return type:: None

opencsp.common.lib.tool.time_date_tools.subtract_seconds_from_ymdhmsz(ymdhmsz: list[int, int, int, int, int, float, int], time_sec: float, ignore_legacy=None) → list[int, int, int, int, int, float, int]

Subtract a specified number of seconds from a date and time represented as a list.

The input list is expected to be in the format: [year, month, day, hour, minute, second, zone].

Parameters:

ymdhmsz (list[int, int, int, int, int, float, int]) – A list representing the date and time, where: - year (int): The year. - month (int): The month (1-12). - day (int): The day of the month (1-31). - hour (int): The hour (0-23). - minute (int): The minute (0-59). - second (float): The second (0-59.999…). - zone (int): The time zone offset.
time_sec (float) – The number of seconds to subtract from the date and time.
ignore_legacy (bool, optional) – If set to False, a warning message will be printed indicating that this function is a legacy function.

Returns:

A new list representing the updated date and time after subtracting the specified seconds.

Return type:

list[int, int, int, int, int, float, int]

Raises:

AssertionError – If rolling over a month boundary is not implemented.

Notes

This function currently does not handle rolling over a month boundary.

Examples

>>> subtract_seconds_from_ymdhmsz([2023, 3, 15, 12, 30, 45.0, 0], 30)
[2023, 3, 15, 12, 30, 15.0, 0]

opencsp.common.lib.tool.time_date_tools.time_string_forfile(t: <module 'time' (built-in)>) → str: %H%M%S

opencsp.common.lib.tool.time_date_tools.to_datetime(ymdhmsz: list[int, int, int, int, int, float, int] | datetime): Converts from a list of [year,month,day,hour,minute,second,utcoffset] to a datetime instance

opencsp.common.lib.tool.time_date_tools.tz(name_or_offset: str | float | timedelta): Create a timezone based off of the name (“US/Mountain”), the short name (“MDT” or “MST”), the UTC offset (-6), or the UTC timedelta offset (datetime.timedelta(hours=-6)).

Exceptions

opencsp.common.lib.tool.exception_tools.ignored(*exceptions)

Easy wrapper to ignore specific kinds of exceptions.

From https://stackoverflow.com/questions/15572288/general-decorator-to-wrap-try-except-in-python

Ignore specific exception types. A common use case is to free attributes within the deconstructor without wanting to worry if the attribute has already been free’d. In this case you can ignore any non-fatal exceptions with the catch-all “Exception”. For example:

def __del__(self):
    with ignored(Exception):
        self.window.close()

Or a more specific example:

with ignored(ArithmeticError):
    dbz = 1 / 0

Files

Utilities for file_handling.

opencsp.common.lib.tool.file_tools.add_row_to_output_dict(input_row: tuple[any, any], output_dict: dict)

Add the [key,value] to the given dictionary.

If either key or value is a string that can be represented as an integer, then it is converted to an integer.

If the value is a string that can be represented as an float, then it is converted to an float.

opencsp.common.lib.tool.file_tools.binary_count_items_in_directory(input_dir: str, name_pattern: str, start=1) → int

Counts the number of items n in a directory by looking for one file at a time.

Starts with file 1, then 3, then 7, …, then n, then n-n/2+n/4, …, n.

Parameters:

input_dir (str) – The directory to search in.
name_pattern (str) – The pattern for the filenames to match. For example “C0044_s2m25_d870.MP4.id_%06.JPG”.
start (int) – Where to start searching at. Most useful for files indexed at 0.

Returns:

The number of files found.

Return type:

int

opencsp.common.lib.tool.file_tools.body_ext_given_file_dir_body_ext(inputdir_body_ext: str)

Return the name+ext for the given path+name+ext.

Args:

input_dir (str): The directory to remove files from.

globexp (str): The glob expression used to match files to be deleted, eg “.jpg” for image files. “” for all files.

Returns:

removed (list[str]): The files that were matched and removed.

opencsp.common.lib.tool.file_tools.directories_in_directory(directory, sort=True): Returns a list of all the directory names contained within the input directory.

opencsp.common.lib.tool.file_tools.directories_with_no_leading_underscore(directory, sort=True): Returns a list of all the directory names contained within the input directory, which do not begin with a leading underscore.

opencsp.common.lib.tool.file_tools.directory_exists(input_dir: str, error_if_exists_as_file=True, follow_symlinks=False): Determines whether the given directory exists. If the specified input directory exists but is a file instead of a directory, then halts with an error, if error_if_exists_as_file==True.

opencsp.common.lib.tool.file_tools.directory_is_empty(input_dir): Determines whether the given directory is empty. The standard Unix files “.” and “..” are ignored. Halts with an error if the directory does not exist.

opencsp.common.lib.tool.file_tools.file_exists(input_dir_body_ext: str, error_if_exists_as_dir=True, follow_symlinks=False): Determines whether the given file exists. If the specified input path exists but is a directory instead of a file, then halts with an error, if error_if_exists_as_dir==True.

opencsp.common.lib.tool.file_tools.file_size(input_dir_body_ext, error_if_exists_as_dir=True): Returns the size of the given file in bytes.

opencsp.common.lib.tool.file_tools.file_size_pair_name(file_size_pair) → str: Get the file name from a [file,size] pair, such as from file_size() or files_in_directory_with_associated_sizes()

opencsp.common.lib.tool.file_tools.file_size_pair_size(file_size_pair) → int: Get the file size from a [file,size] pair, such as from file_size() or files_in_directory_with_associated_sizes()

opencsp.common.lib.tool.file_tools.files_in_directory(input_dir, sort=True, files_only=False, recursive=False)

Returns a list [ file1, file2, …] of files in the given directory.

The returned values include the “name.ext” of the file. For files_only=False, all entries in the directory, including the names of directories, are returned. Does not include the unix “.” (current) and “..” (parent) directories.

Args:

sort: bool: If True, then the list is sorted in order of ascending file name. Default True.
files_only: bool: If True, then return only the file entries. Default False.
recursive: bool: If true, then walk through all files in the given directory and all subdirectories. Does not follow symbolic links to directories. Default False.

Returns:

files_name_ext (list[str]): The list of file name_exts (example [“a.csv”, “b.csv”, …])

opencsp.common.lib.tool.file_tools.files_in_directory_by_extension(input_dir: str, extensions: list[str], sort=True, case_sensitive=False, recursive=False)

Generates a list of { ext: [file1, file2, …], … }. Only returns the files with one of the given extensions.

Arguments:

input_dir: str: directory to search for files
extensions: list[str]: list of extensions for the files (with or without leading periods “.”)
sort: bool: if True, then sort the files by name before returning. Defaults to True
case_sensitive: bool: If True, then the file extensions matching are case sensitive. Defaults to False
recursive: bool: If true, then walk through all files in the given directory and all subdirectories. Does not follow symbolic links to directories. Default False.

Returns:

files: dict[str,list[str]]: The found file name_exts, by their associated extension. For example { “csv”: [“a.csv”,”b.csv”], “txt”: [“foo.txt”,”bar.txt”] }

opencsp.common.lib.tool.file_tools.files_in_directory_with_associated_sizes(input_dir, sort=True, follow_symlinks=True)

Returns a list [ [file1 size1], [file2, size2], …] of files name_ext and associated sizes. If sort==True, then the list is sorted in order of ascending file name.

See also: file_size_pair_name(), file_size_pair_size()

opencsp.common.lib.tool.file_tools.from_csv(description: str | None, input_path: str, input_file_name_ext: str)

Reads a csv file and returns the rows, including the header row.

Concise example:

parser = scsv.SimpleCsv("example file", file_path, file_name_ext)
for row_dict in parser:
    print(row_dict)

Verbose example:

lines = ft.from_csv("example file", file_path, file_name_ext)
header_row = lines[0]
cols = csv.CsvColumns.SimpleColumns(header_row)

data_rows = lines[1:]
for row in data_rows:
    row_dict = cols.parse_data_row(row)
    print(row_dict)

Parameters:

description (str): printed to the log input_path (str): directory containing the file to read input_file_name_ext (str): name and extension of the file to read

Returns:

rows: list[list[str]]
All the rows from the csv file, split on the comma ‘,’ delimeter.

opencsp.common.lib.tool.file_tools.get_temporary_file(suffix: str | None = None, dir: str | None = None, text: bool = True, include_file_handle: bool = True) → tuple[int, str]

Creates a temporary file to write to. Example usage:

fd, fname = ft.get_temporary_file()
try:
    with open(fd, 'w') as fout:
        fout.write("foo")
    ...
finally:
    ft.delete_file(fname)

Args:

suffix (str): Suffix of the file name. Be sure to include the leading “.” for file extension suffixes.

dir (str): Where to save the temporary file with the list of frame names.
Defaults to the opencsp_temporary_dir if writable, or else the home directory if writable, or else /tmp.

text (bool): True to open the file in text mode. False to open it in byte mode.

Returns:

int: The file descriptor for the new file.

str: The path_name_ext of the new file.

opencsp.common.lib.tool.file_tools.join(*path_components: str)

Joins and normalize the given path components. For example:

join(“a”, “b/c.txt”)

…is the same as:

norm_path(os.path.join(“a”, “b/c.txt”))

opencsp.common.lib.tool.file_tools.merge_files(in_files: list[str], out_file: str, overwrite=False, remove_in_files: bool = False)

Merges the given files into a single file. The order of the input files is preserved in the output file.

Parameters:

in_files (list[str]) – A list of file path_name_exts to be merged
out_file (str) – A file path_name_ext to merge into
overwrite (bool) – If False, then check that out_file doesn’t already exist, and error if it does. Defaults to False.
remove_in_files (bool) – Remove the input files as they are added to the output file. Defaults to False.

opencsp.common.lib.tool.file_tools.norm_path(path: str, allow_extended_length_path=True): Normalizes the given path (use system-style slashes) and prepend the long-path signifier, as necessary.

opencsp.common.lib.tool.file_tools.path_components(input_dir_body_ext: str)

Breaks input string “~/foo/bar/blech.grrr.txt” into:: dir: “~/foo/bar” body: “blech.grrr” ext: “.txt”

Returns:

tuple: dir (sans last slash), body (name of the file), ext (includes the leading “.”)

See also: body_ext_given_file_dir_body_ext()

opencsp.common.lib.tool.file_tools.path_to_cmd_line(path: str): Normalizes and surrounds a path with quotes, as necessary.

opencsp.common.lib.tool.file_tools.read_csv_file(description, input_path, input_file_name, log_warning=True): Deprecated. Use from_csv() instead, to better match naming in pandas.

opencsp.common.lib.tool.file_tools.read_dict(input_dict_dir_body_ext): Reads a dictionary. Assumes input file exists, is a csv file, and each row has two entries. The first entry is the key, and the second entry is a value. If the key can be parsed as an integer, it is converted to an integer object. If the value can be parsed as a float or integer, it is converted to the corresponding type; otherwise, the value is added to the dict as a literal string.

opencsp.common.lib.tool.file_tools.read_json(description: str | None, input_dir: str, input_file_body_ext: str) → any

Like json.loads(file_contents) but with more safety checks, and ignoring any lines starting with “//” as comments.

Parameters:

description (str | None) – A human-readable description of what this file is for, to be logged to the command line. If None, then doesn’t log.
input_dir (str) – The source directory where the file exists.
input_file_body_ext (str) – The source name+ext of the file. For example “foo.json”.

Returns:

The json-parsed contents of the file.

Return type:

any

opencsp.common.lib.tool.file_tools.read_text_file(input_dir_body_ext): Reads a text file. Assumes input file exists, and is a text file. File extension may be arbitrary. Returns a list of strings, one per line. No parsing.

opencsp.common.lib.tool.file_tools.rename_directory(input_dir: str, output_dir: str)

Move a directory from input to output.

Verifies that input is a directory, and that the output doesn’t exist. We check for existence of the output directory after the rename, and raise a FileNotFoundError if it can’t be found.

Parameters:

input_dir (str) – The directory to be renamed.
output_dir (str, optional) – The new destination name for the input_dir.

opencsp.common.lib.tool.file_tools.rename_file(input_dir_body_ext: str, output_dir_body_ext: str, is_file_check_only=False, nattempts=20, delay=2, cross_filesys_check=True)

Move a file from input to output.

Verifies that input is a file, and that the output doesn’t exist. We check for existence of the output file after the rename, and raise a FileNotFoundError if it can’t be found.

Parameters:

input_dir_body_ext (str) – The “dir/body.ext” of the source file to be renamed.
output_dir_body_ext (str, optional) – The destinaion “dir/body.ext” of the file.
is_file_check_only (bool, optional) – If True, then only check that input_dir_body_ext is a file. Otherwise, check everything. Default is False.
nattempts (int, optional) – The number of times to try to save the file. Should be at least 1. Default is 20.
delay (float, optional) – How many seconds to wait between a failed rename attempt and another attempt. Default is 2s.
cross_filesys_check (bool, optional) – If True, then check for a cross-filesystem error after the first rename fails. If that detected as the cause then rename the file using copy_and_delete_file() instead. Default is True.
also (See)

opencsp.common.lib.tool.file_tools.resolve_symlink(input_dir_body_ext: str): If the input is a symbolic link (os.path.islink), then get the real path.

opencsp.common.lib.tool.file_tools.to_csv(description: str | None, output_dir: str, output_file_body: str, heading_line: str | bool | None, data_lines: list[str | CsvInterface], error_if_dir_not_exist: bool = True, overwrite=False)

Writes a “.csv” file with a heading line and subsequent data lines.

This implementation expects that each line is a simple string, with “,” separators already added.

Parameters:

description (str|None) – Explanatory string to include in notification output. None to skip.
heading_line (str|None|True) – First line to write to file. None to skip. True to use data_lines[0].csv_header() for CsvInterface data types.
error_if_dir_not_exist (bool) – If True, error if not exist. If False, create dir if necessary.

opencsp.common.lib.tool.file_tools.write_csv_file(description, output_dir, output_file_body, heading_line, data_lines, error_if_dir_not_exist=True, log_warning=True)

Deprecated. Use to_csv() instead, to better match naming in pandas.

Writes a “.csv” file with a heading line and subsequent data lines. This implementation expects that each line is a simple string, with “,” separators already added.

opencsp.common.lib.tool.file_tools.write_dict_file(description, output_dir, output_body, output_dict, decimal_places=9, error_if_dir_not_exist=True): Writes a dictionary to a “.csv” file, with a comma separating dictionary keys from values. Calls str(value) on all dictionary values except floats. For floats, writes a decimal with the specified number of decimal places.

opencsp.common.lib.tool.file_tools.write_json(description: str | None, output_dir: str, output_file_body: str, output_object: any, error_if_dir_not_exist=True)

Like json.dump(output_object, output_file_body) but with a few more safety checks and automatic “.json” extension appending.

Parameters:

description (str | None) – A human-readable description of what this file is for, to be logged to the command line. If None, then no log is created.
output_dir (str) – The destination directory for the file.
output_file_body (str) – The destination name for the file. Should not include an extension. For example: “foo” is ok, but “foo.json” is not.
output_object (any) – The object to be saved to the given file.
error_if_dir_not_exist (bool, optional) – If True, then first check if the given output_dir exists. By default True.

opencsp.common.lib.tool.file_tools.write_text_file(description: str, output_dir: str, output_file_body: str, output_string_list: list[any], error_if_dir_not_exist=True) → str

Writes a strings to a “.txt” file, with each string on a new line.

Parameters:

description (str) – Explanatory string to include in notification output. None to skip.
output_dir (str) – Which directory to write the file to. See below if not exist.
output_file_body (str) – Name of the file without an extension. A standard “.txt” extension will be automatically appended.
output_string_list (list) – List of values to write to the file, one per line. Newlines “n” will be automatically appended to each line.
error_if_dir_not_exist (bool, optional) – If True and output_dir doesn’t exists, raise an error. If False, create output_dir as necessary. By default True.

Returns:

output_dir_body_ext – The “path/name.ext” of the newly created file.

Return type:

str

HDF5

class opencsp.common.lib.tool.hdf5_tools.HDF5_IO_Abstract

Bases: HDF5_SaveAbstract

Abstract class for loading from HDF5 format

abstract classmethod load_from_hdf(file: str, prefix: str = '')

Loads data from given file. Assumes data is stored as: PREFIX + Folder/Field_1

Parameters:

file (str) – HDF file to load from
prefix (str, optional) – Prefix to append to folder path within HDF file (folders must be separated by “/”). Default is empty string ‘’.

class opencsp.common.lib.tool.hdf5_tools.HDF5_SaveAbstract

Bases: ABC

Abstract class for saving to HDF5 format

abstract save_to_hdf(file: str, prefix: str = '') → None

Saves data to given file. Data is stored as: PREFIX + Folder/Field_1

Parameters:

file (str) – HDF file to save to
prefix (str, optional) – Prefix to append to folder path within HDF file (folders must be separated by “/”). Default is empty string ‘’.

opencsp.common.lib.tool.hdf5_tools.get_groups_and_datasets(hdf5_path_name_ext: str | File)

Get the structure of an HDF5 file, including all group and dataset names, and the dataset shapes.

Parameters:

hdf5_path_name_ext (str | h5py.File) – The HDF5 file to parse the structure of.

Returns:

group_names (list[str]) – The absolute names of all the groups in the file. For example: “foo/bar”
file_names_and_shapes (list[ tuple[str,tuple[int]] ]) – The absolute names of all the datasets in the file, and their shapes. For example: “foo/bar/baz”, (1920,1080)

opencsp.common.lib.tool.hdf5_tools.is_dataset_and_shape(object: Group | Dataset) → tuple[bool, tuple]

Returns whether the given object is an hdf5 dataset and, if it is, then also what it’s shape is.

Parameters:

object (h5py.Group | h5py.Dataset) – The object to check the type of.

Returns:

is_dataset (bool) – True if object is a dataset, False otherwise
shape (tuple[int]) – The shape of the dataset. Empty tuple() object if not a dataset.

opencsp.common.lib.tool.hdf5_tools.load_hdf5_datasets(datasets: list, file: str): Loads datasets from HDF5 file

opencsp.common.lib.tool.hdf5_tools.save_hdf5_datasets(data: list, datasets: list, file: str): Saves data to HDF5 file

opencsp.common.lib.tool.hdf5_tools.unzip(hdf5_path_name_ext: str, destination_dir: str, dataset_format='npy')

Unpacks the given HDF5 file into the given destination directory.

Unpacks the given HDF5 file into the given destination directory. A new directory is created in the destination with the same name as the hdf5 file. String values are extracted as .txt files, and images are extracted as .png files. Everything else is saved with numpy as .npy files.

Parameters:

hdf5_path_name_ext (str) – The HDF5 file to unpack.
destination_dir (str) – The directory in which to create a directory for the HDF5 file.
dataset_format (str, optional) – Format in which to save non-image, non-string datasets. Can be one of “npy” or “csv”. If the dataset has more than 2 dimension then it will be forceably saved to npy. Default npy.

Returns:

output_dir – The path to the newly created directory into which the HDF5 files were extracted into.

Return type:

str

Images

Utilities for image processing.

opencsp.common.lib.tool.image_tools.dims_and_nchannels(image: ndarray)

Get the (y,x) dimensions of the image, and the number of color channels.

Raises:

ValueError: If the given array doesn’t have the correct number of dimensions (2 or 3).

opencsp.common.lib.tool.image_tools.get_exif_value(data_dir: str, image_path_name_exts: str | list[str], exif_val: str = 'EXIF:ISO', parser: ~typing.Callable[[str], ~opencsp.common.lib.tool.image_tools.T] = <class 'int'>) → T | list[T]: Returns the exif_val Exif information on the given images, if they have such information. If not, then None is returned for those images.

opencsp.common.lib.tool.image_tools.getsizeof_approx(img: <module 'PIL.Image' from '/home/docs/checkouts/readthedocs.org/user_builds/opencsp/envs/latest/lib/python3.10/site-packages/PIL/Image.py'>) → int

Get the number of bytes of memory used by the given image.

Note that this value is approximate. It should be close based on the basic assumptions of uncompressed data in memory, and one byte per pixel per color channel.

Parameters:: img (Image) – The image to get the size of.
Returns:: Number of bytes of memory that the image object + image data together occupy.
Return type:: int

opencsp.common.lib.tool.image_tools.image_files_in_directory(dir: str, allowable_extensions: list[str] | None = None, recursive=False) → list[str]

Get a list of all image files in the given directory, as determined by the file extension.

Parameters:

dir (str) – The directory to get files from.
allowable_extensions (list[str], optional) – The allowed extensions, such as [“png”]. By default pil_image_formats_rw.
recursive (bool) – If true, then walk through all files in the given directory and all subdirectories. Does not follow symbolic links to directories. Default False.

Returns:

image_file_names_exts – A sorted list of the name.ext for each image file in the given directory.

Return type:

list[str]

opencsp.common.lib.tool.image_tools.images_are_identical(image_1: ndarray, image_2: ndarray, tolerance_pixel: int)

Checks if two images are identical.

Parameters:

image_1 (-) – The first image to compare, such as from cv.imread().
image_2 (-) – The second image to compare, such as from cv.imread().
tolerance_pixel (-) – How many pixels are allowed to be different and the images are still considered identical.

Returns:

True if identical, False otherwise

Return type:

bool

opencsp.common.lib.tool.image_tools.min_max_colors(image: ndarray) → tuple[ndarray, ndarray]

Get the minimum and maximum values for each of the color channels in the given image.

Parameters:

image (np.ndarray) – The image to get the minimum and maximum color values for. Must have 2-3 dimensions.

Returns:

min_colors (ndarray) – The minimum values for each color channel. If there is only 1 color channel, then this array will have shape=(1).
max_colors (ndarray) – The maximum values for each color channel. If there is only 1 color channel, then this array will have shape=(1).

opencsp.common.lib.tool.image_tools.numpy_to_image(arr: ndarray, rescale_or_clip='rescale', rescale_max=-1) → Image

Convert the numpy representation of an image to a Pillow Image.

Coverts the given arr to an Image. The array is converted to an integer type, as necessary. The color information is then rescaled/clipd to fit within an 8-bit color depth.

In theory, images can be saved with higher bit-depth information using opencv imwrite(‘12bitimage.png’, arr), but I (BGB) haven’t tried very hard and haven’t had any luck getting this to work.

Parameters:

arr (np.ndarray) – The array to be converted.
rescale_or_clip (str, optional) – Whether to rescale the value in the array to fit within 0-255, or to clip the values so that anything over 255 is set to 255. By default ‘rescale’.
rescale_max (int, optional) – The maximum value expected in the input arr, which will be set to 255. When less than 0, the maximum of the input array is used. Only applicable when rescale_or_clip=’rescale’. By default -1.

Returns:

image – The image representation of the input array.

Return type:

PIL.Image

opencsp.common.lib.tool.image_tools.range_for_threshold(image: ndarray, threshold: int) → tuple[int, int, int, int]

Get the start (inclusive) and end (exclusive) range for which the given image is >= the given threshold.

Parameters:

image (np.ndarray) – The 2d numpy array to be searched, row major (y axis 0, x axis 1).
threshold (int) – The cutoff value that the returned region should have pixels greater than.

Returns:

start_y, end_y, start_x, end_x – The start (inclusive) and end (exclusive) matching range. Returns the full image size if there are no matching pixels.

Return type:

tuple[int, int, int, int]

opencsp.common.lib.tool.image_tools.set_exif_value(data_dir: str, image_path_name_ext: str, exif_val: str, exif_name: str = 'EXIF:ISO')

Set the specified EXIF tag value for an image file.

This function uses the ExifTool to set the value of a specified EXIF tag (defaulting to “EXIF:ISO”) for a given image file located in the specified directory. It attempts to overwrite the original image file with the new EXIF data. If the initial attempt fails due to an execution error, the function is designed to handle the error gracefully (though the error handling code is currently commented out and requires implementation).

Parameters:

data_dirstr: The directory where the image file is located.
image_path_name_extstr: The name of the image file, including its extension.
exif_valstr: The value to set for the specified EXIF tag.
exif_namestr, optional: The name of the EXIF tag to set (default is “EXIF:ISO”).

Raises:

exiftool.exceptions.ExifToolExecuteError: If the ExifTool encounters an error while attempting to set the EXIF tag.

Notes:

The function currently contains commented-out code for handling poorly formatted images by rewriting them using the Pillow library. This functionality is intended to be implemented in the future.

opencsp.common.lib.tool.image_tools.to_image(img: ImageLike, output_type: str = 'numpy') → ndarray | Image

Converts the input image from whatever type it is to the given output type.

Parameters:

img (Union[str, np.ndarray, Image.Image, RenderControlFigureRecord, PowerpointImage]) – The input image to be converted.
output_type ("numpy" | "pillow", optional) – The desired output type. Can be one of “Numpy” or “Pillow”. Default is “numpy”.

Returns:

The converted image in the specified output type.

Return type:

Union[np.ndarray, Image.Image]

opencsp.common.lib.tool.image_tools.pil_image_formats_readable = ['blp', 'bmp', 'dds', 'dib', 'eps', 'gif', 'icns', 'ico', 'im', 'jpg', 'jpeg', 'msp', 'pcx', 'png', 'apng', 'pbm', 'pgm', 'ppm', 'pnm', 'sgi', 'spider', 'tga', 'tiff', 'webp', 'xbm', 'cur', 'dcx', 'fits', 'fli', 'flc', 'fpx', 'ftex', 'gbr', 'gd', 'imt', 'iptc', 'naa', 'mcidas', 'mic', 'mpo', 'pcd', 'pixar', 'psd', 'sun', 'wal', 'wmf', 'emf', 'xpm']: A list of all image image formats that can be read by the Python Imaging Library (PIL). Note that not all of these formats can be written by PIL.

opencsp.common.lib.tool.image_tools.pil_image_formats_rw = ['blp', 'bmp', 'dds', 'dib', 'eps', 'gif', 'icns', 'ico', 'im', 'jpg', 'jpeg', 'msp', 'pcx', 'png', 'apng', 'pbm', 'pgm', 'ppm', 'pnm', 'sgi', 'spider', 'tga', 'tiff', 'webp', 'xbm']: A list of all image image formats that can be read and written by the Python Imaging Library (PIL)

opencsp.common.lib.tool.image_tools.pil_image_formats_writable = ['blp', 'bmp', 'dds', 'dib', 'eps', 'gif', 'icns', 'ico', 'im', 'jpg', 'jpeg', 'msp', 'pcx', 'png', 'apng', 'pbm', 'pgm', 'ppm', 'pnm', 'sgi', 'spider', 'tga', 'tiff', 'webp', 'xbm', 'palm', 'pdf', 'xv']: A list of all image image formats that can be written by the Python Imaging Library (PIL). Note that not all of these formats can be read by PIL.

Logging

Utilities for managing logs for multi-processing.

opencsp.common.lib.tool.log_tools.critical(*vargs, **kwargs) → int

Critical error message, both to console and log file.

A serious error, indicating that the program itself may be unable to continue running. You should most likely be using the critical_and_raise version of this function.

Example:

import log_tools as lt
lt.logger(home_dir() + 'current.log')
lt.critical_and_raise(ValueError, 'In my_function(), Negative x should be impossible, but x=' + str(x) + ' encountered.')
# or
lt.critical('In my_function(), Negative x should be impossible, but x=' + str(x) + ' encountered.')

opencsp.common.lib.tool.log_tools.critical_and_raise(exception_class: type, msg: str, base_exception: Exception | None = None) → None

Logs the given message at the “critical” level and raises the given exception, also with this message.

Parameters:

exception_class (Exception.__class__) – An exception class. See error_and_raise() for a description of built-in exceptions.
msg (str) – The message to go along with the exception.
base_exception (Exception) – The exception that caused this code to be called, if any. This will be added onto the newly created exception_class’ history.

Example

import log_tools as lt
lt.logger(home_dir() + 'current.log')
lt.critical_and_raise(ValueError, 'In my_function(), Negative x should be impossible, but x=' + str(x) + ' encountered.')

opencsp.common.lib.tool.log_tools.debug(*vargs, **kwargs) → int

Output debugging information, both to console and log file.

This is for logging detailed information, typically of interest only when diagnosing problems. Can lead to lots of system-generated information in the log file, such as matplotlib font declarations, etc.

Example:

import log_tools as lt
lt.logger(home_dir() + 'current.log')
lt.debug('In my_function(), input_file  = ' + str(input_file_dir_body_ext))
lt.debug('In my_function(), output_file = ' + str(output_file_dir_body_ext))

opencsp.common.lib.tool.log_tools.error(*vargs, **kwargs) → int

Error message, both to console and log file.

Due to a more serious problem, the software has not been able to perform some function.

Example:

import log_tools as lt
lt.logger(home_dir() + 'current.log')
lt.error_and_raise(ValueError, 'In my_function(), non-positive value x=' + str(x) + ' encountered.')
# or
lt.error('In my_function(), non-positive value x=' + str(x) + ' encountered.')

opencsp.common.lib.tool.log_tools.error_and_raise(exception_class: type, msg: str, base_exception: Exception | None = None) → None

Logs the given message at the “error” level and raises the given exception, also with this message.

Parameters:

exception_class (Exception.__class__) – An exception class. See below for built-in exception types.
msg (str) – The message to go along with the exception.
base_exception (Exception) – The exception that caused this code to be called, if any. This will be added onto the newly created exception_class’ history.

Example

import log_tools as lt
lt.logger(home_dir() + 'current.log')
lt.error_and_raise(ValueError, 'In my_function(), non-positive value x=' + str(x) + ' encountered.')

Built-in Exception Types

See the python exceptions documentation for a list of built-in exceptions:

BaseException
├── BaseExceptionGroup
├── GeneratorExit
├── KeyboardInterrupt
├── SystemExit
└── Exception
    ├── ArithmeticError
    |   ├── FloatingPointError
    |   ├── OverflowError
    |   └── ZeroDivisionError
    ├── AssertionError
    ├── AttributeError
    ├── BufferError
    ├── EOFError
    ├── ImportError
    |   └── ModuleNotFoundError
    ├── LookupError
    |   ├── IndexError
    |   └── KeyError
    ├── MemoryError
    ├── NameError
    |   └── UnboundLocalError
    ├── OSError
    |   ├── BlockingIOError
    |   ├── ChildProcessError
    |   ├── ConnectionError
    |   |   ├── BrokenPipeError
    |   |   ├── ConnectionAbortedError
    |   |   ├── ConnectionRefusedError
    |   |   └── ConnectionResetError
    |   ├── FileExistsError
    |   ├── FileNotFoundError
    |   ├── InterruptedError
    |   ├── IsADirectoryError
    |   ├── NotADirectoryError
    |   ├── PermissionError
    |   ├── ProcessLookupError
    |   └── TimeoutError
    ├── ReferenceError
    ├── RuntimeError
    |   ├── NotImplementedError
    |   └── RecursionError
    ├── StopAsyncIteration
    ├── StopIteration
    ├── SyntaxError
    |   └── IndentationError
    |        └── TabError
    ├── SystemError
    ├── TypeError
    ├── ValueError
    |   └── UnicodeError
    |        ├── UnicodeDecodeError
    |        ├── UnicodeEncodeError
    |        └── UnicodeTranslateError
    └── Warning
        ├── BytesWarning
        ├── DeprecationWarning
        ├── EncodingWarning
        ├── FutureWarning
        ├── ImportWarning
        ├── PendingDeprecationWarning
        ├── ResourceWarning
        ├── RuntimeWarning
        ├── SyntaxWarning
        ├── UnicodeWarning
        └── UserWarning

opencsp.common.lib.tool.log_tools.get_log_method_for_level(level: int) → Callable

Returns one of the log methods (debug, info, warning, error, critical) based on the given level.

Parameters:: level (int) – One of log.DEBUG, log.INFO, log.WARNING, log.ERROR, or log.CRITICAL

opencsp.common.lib.tool.log_tools.info(*vargs, **kwargs) → int

Report program progress, both to console and log file.

Use this level to confirm that things are working as expected.

Example:

import log_tools as lt
lt.logger(home_dir() + 'current.log')
lt.info('In my_function(), writing file: ' + str(output_file_dir_body_ext) + '...')

opencsp.common.lib.tool.log_tools.info_strings_from_file(input_strings_file: str) → None

Reads strings from input file and writes them to the log as INFO.

Parameters:: input_strings_file (str) – Text file containing strings to log.

opencsp.common.lib.tool.log_tools.log_and_raise_value_error(local_logger, msg) → None

Logs an error and raises a ValueError with given message

Parameters:

local_logger (Logger) – Unused, kept for backwards compatibility
msg (str) – Error message

opencsp.common.lib.tool.log_tools.log_progress(percentage: int | float, carriage_return: bool | Literal['auto'] = 'auto', prev_percentage: int | None = None)

Prints the current progress as a progress bar and number.

Parameters:

percentage (int | float) – The current progress. If an integer, then the range is clipped to 0-100. If a float, then the range is clipped to 0-1, unless >1 then it is cast to an integer.
carriage_return (bool | 'auto', optional) – If True, then a carriage return ‘r’ is printed instead of a newline, which will cause the next line printed to overwrite this line. This can be used to “draw” the progress interactively in the terminal. If ‘auto’, then this will be True when percentage != 100. By default ‘auto’.
prev_percentage (int, optional) – If not None, then this is compared to the given percentage. If they are the same then nothing is printed.

Returns:

percentage – The value printed, in the range 0-100. Can be passed into the next call as prev_percentage.

Return type:

int

opencsp.common.lib.tool.log_tools.logger(log_dir_body_ext: str | None = None, level: int = 20, delete_existing_log: bool = True) → Logger

Initialize logging for single-process programs.

Creates a fresh log file, deleting the existing log file if it exists as indicated by delete_existing_log. Once this method is called, then the debug(), info(), warn(), error(), and critical() methods will use the logger created here.

Example usage:

lt.logger(home_dir() + 'current.log')
lt.info('Starting program ' + __file__)

Parameters:

log_dir_body_ext (-) – Fully qualified path to desired log file. Default [OpenCSP/dir]/../[timedate]_log.txt

level (-) –

Significance threshold for writing messages to this log. Level values:

10 debug    Detailed information, typically of interest only when diagnosing problem1s.
            Can lead to lots of system-generated information in the log file, such as
            matplotlib font declarations, etc.
20 info     Confirmation that things are working as expected.
30 warning  An indication that something unexpected happened, or indicative of some problem in the
            near future (e.g. 'disk space low'). The software is still working as expected.
40 error    Due to a more serious problem, the software has not been able to perform some function.
50 critical A serious error, indicating that the program itself may be unable to continue running.

Messages are written to the log if their severity >= level. See https://docs.python.org/3/howto/logging.html

delete_existing_log (-) – Whether to delete a log from a previous run if encountered. Default True. If False, new log information is appended to an existing file if it exists.

opencsp.common.lib.tool.log_tools.multiprocessing_logger(log_dir_body_ext=None, level=20) → Logger

Create a logger for logging across many processes.

For multiprocessing logs, it is recommended that the existing log is deleted by the user or a batch process script. Once this method is called, then the info(), warn(), and error() methods will use the logger created here.

Example usage:

lt.multiprocessing_logger(experiment_dir() + '/2021-05-13_FastScan2/4_Post/Construction/20210525/1325_NS_U/090c_PredictHeliostats/latest_run.log')
lt.info('Starting program ' + __file__)

Parameters:

log_dir_body_ext (-) – Fully qualified path to desired log file. None for a stream handler.
level (-) – Significance threshold for writing messages to this log. See https://docs.python.org/3/howto/logging.html

opencsp.common.lib.tool.log_tools.warn(*vargs, **kwargs)

Warning message, both to console and log file.

Use this level as an indication that something unexpected happened, or indicative of some problem in the near future (e.g. ‘disk space low’). The software should still be working as expected.

Example:

import log_tools as lt
lt.logger(home_dir() + 'current.log')
lt.warn('In my_function(), Plot x label is empty.')

opencsp.common.lib.tool.log_tools.warning(*vargs, **kwargs)

Warning message, both to console and log file.

Use this level as an indication that something unexpected happened, or indicative of some problem in the near future (e.g. ‘disk space low’). The software should still be working as expected.

Example:

import log_tools as lt
lt.logger(home_dir() + 'current.log')
lt.warn('In my_function(), Plot x label is empty.')

Math

Various math routines.

opencsp.common.lib.tool.math_tools.clamp(x, x_min, x_max)

Assure that x is in [x_min, x_max].

For numpy arrays, use clip(): https://numpy.org/doc/stable/reference/generated/numpy.ndarray.clip.html

opencsp.common.lib.tool.math_tools.gaussian_convolution(num_points: int)

Get the probability-density-function (y-values) of the normal distribution covering 98+% of all probabilities.

Example:

timestep = 0.1                         # 10 Hz measurement frequency for "spiky_data"
gaussian = mt.gaussian_convolution(int(1.0 / timestep))
print(["%0.2f" % p for p in gaussian]) # ['0.01', '0.04', '0.09', '0.15', '0.20', '0.20', '0.15', '0.09', '0.04', '0.01']
print(sum(gaussian))                   # 0.9902573253069845
smooth_data = np.convolve(spiky_data, gaussian, 'same')

Parameters:

num_points (int): How many bins should be used to represent the pdf?

Returns:

pdf (list[float]): probability that the value at index i is chosen, where:
i follows a gaussian curve, i is greatest at len/2, and 0.98 < sum(pdf) <= 1.00

opencsp.common.lib.tool.math_tools.is_nan(x): Determines whether an input variable is NaN. Unlike the numpy isnan() function, this deos not thrown an error if the input parameter is not a number.

opencsp.common.lib.tool.math_tools.is_number(x): Determines whether an input variable is numeric type.

opencsp.common.lib.tool.math_tools.lambda_symmetric_paraboloid(focal_length: Number) → Callable[[float, float], float]

Create a lambda function representing a symmetric paraboloid.

The symmetric paraboloid is defined by the equation:

z = (1 / (4 * f)) * (x^2 + y^2)

where ‘f’ is the focal length of the paraboloid.

Parameters:: focal_length (numbers.Number) – The focal length of the paraboloid. Must be a positive number.
Returns:: A lambda function that takes two float arguments (x, y) and returns the corresponding z value of the symmetric paraboloid.
Return type:: Callable[[float, float], float]
Raises:: ValueError – If the focal_length is not positive.

Examples

>>> paraboloid = lambda_symmetric_paraboloid(2.0)
>>> z_value = paraboloid(1.0, 1.0)
>>> print(z_value)
0.125

opencsp.common.lib.tool.math_tools.n_combinations(n, k): Returns (n choose k). All combinations of k length.

opencsp.common.lib.tool.math_tools.n_combinations_upto(n, k): Returns (n choose 1) + (n choose 2) + … + (n choose k) All combinations of up to k length.

opencsp.common.lib.tool.math_tools.ncr(n, r)

Calculate the binomial coefficient, also known as “n choose r”.

The binomial coefficient is defined as:

C(n, r) = n! / (r! * (n - r)!)

where ‘n’ is the total number of items, and ‘r’ is the number of items to choose.

Parameters:

n (int) – The total number of items. Must be a non-negative integer.
r (int) – The number of items to choose. Must be a non-negative integer less than or equal to ( n ).

Returns:

The binomial coefficient ( C(n, r) ).

Return type:

float

Raises:

ValueError – If ( n ) or ( r ) are negative, or if ( r ) is greater than ( n ).

Examples

>>> ncr(5, 2)
10.0
>>> ncr(10, 3)
120.0
>>> ncr(0, 0)
1.0

opencsp.common.lib.tool.math_tools.none_if_nan(x): If x is NaN, then returns None. Else returns x.

opencsp.common.lib.tool.math_tools.overlapping_range(range1: tuple[float, float] | list[float], range2: tuple[float, float] | list[float], *rangeN: tuple[float, float] | list[float], default=None) → list[float]

Find the overlapping region of two or more ranges of numbers.

Example:

overlap = overlapping_range([1,5], [3,8])
# overlap == [3,5]
overlap = overlapping_range([1,5], [3,8], [0,4])
# overlap == [3,4]
overlap = overlapping_range([6,7], [7,8])
# overlap == [7,7]
overlap = overlapping_range([6,7], [8,9])
# overlap == []
overlap = overlapping_range([], [])
# overlap == []

Arguments:

range1 (tuple[float,float]|list[float]): Two values that define a range.

range2 (tuple[float,float]|list[float]): Two values that define a range.

rangeN (tuple[float,float]|list[float]): Any other number of tuples or lists of two values that define a range.

default: The default value to return when there’s an error with the input or there is no overlap. Empty list [] when None. Defaults to None.

Returns:

overlap (list[float,float]|list): The overlapping range. If there is no overlap, returns an empty list.

opencsp.common.lib.tool.math_tools.rms(error_list): Computes the root mean square of the input list of error values.

opencsp.common.lib.tool.math_tools.robust_arccos(x: float) → float: Inverse cosine function, but with a small margin for values outside of [-1,1] to allow for numerical roundoff error.

opencsp.common.lib.tool.math_tools.robust_arcsin(x: float) → float: Inverse sine function, but with a small margin for values outside of [-1,1] to allow for numerical roundoff error.

opencsp.common.lib.tool.math_tools.rolling_average(data: list[float] | ndarray[Any, dtype[float64]], window_size: int)

Compute the rolling average over all values in the given list.

For each point x at index i, assign a new value equal to sum([i-w/2:i+w/2]) / 2. That is, assign the average of the surrounding window_size values.

Edge cases:

If window_size is odd, then use an equal number of values above and below x.
If window_size is even, then use one more value below than above.
At edges, divide by the window size to avoid edge effects.

Parameters:

data (list|np.ndarray): one-dimensional list of values to compute the rolling average for TODO support multiple dimensions

window_size (int): size of the rolling average window (how many values to average around each data point)

Returns:

averaged_data (list|np.ndarray): the averaged values, of the same length and type as the input data

opencsp.common.lib.tool.math_tools.string_is_float(input_str)

Returns True if the string can be parsed as a floating-point value. If this routine returns True, then float(input_str) should succeed.

Thanks to StackOverflow for this answer:: https://stackoverflow.com/questions/736043/checking-if-a-string-can-be-converted-to-float-in-python

opencsp.common.lib.tool.math_tools.string_is_integer(input_str)

Returns True if the string can be parsed as an integer. If this routine returns True, then int(input_str) should succeed.

See string_is_float() below for inspiration.

Another method:: return input_str.isnumeric()

opencsp.common.lib.tool.math_tools.zero_if_nan(x): If x is NaN, then returns zero. Else returns x.

System

opencsp.common.lib.tool.system_tools.is_cluster()

Determines if this computer is a HPC nodes.

Returns:: True if running on a HPC cluster node
Return type:: bool

opencsp.common.lib.tool.system_tools.is_notebook() → bool

Is this code running in a notebook. From https://stackoverflow.com/questions/15411967/how-can-i-check-if-code-is-executed-in-the-ipython-notebook

Returns:: True if this code is executing in a Jypter notebook, or False otherwise.
Return type:: bool

opencsp.common.lib.tool.system_tools.is_production_run()

Determines if this code is being evaluated as “production”. Defaults to False, or True if running on solo or not __debug__.

An exmple use case for this code is in verifying that a “verbose” flag isn’t accidentally left on when we go to ship our code.

opencsp.common.lib.tool.system_tools.is_solo()

Determines if this computer is one of the Solo HPC nodes.

Returns:: True if running on Solo
Return type:: bool

opencsp.common.lib.tool.system_tools.mem_status()

Get the memory status of this computer. Also consider using the psutil package.

Returns:: Total system memory, used system memory, available system memory (in GB)
Return type:: tuple[float,float,float]

opencsp.common.lib.tool.system_tools.set_is_production_run(is_production_run: bool): Setter for __is_production_run global

Tkinter

class opencsp.common.lib.tool.tk_tools.TkToolTip(widget, text='widget info')

Bases: object

A tooltip class for Tkinter widgets.

This class provides a tooltip that appears when the mouse hovers over a specified widget. The tooltip displays a text message and automatically hides when the mouse leaves the widget.

waittime

The time in milliseconds to wait before showing the tooltip.

Type:: int

wraplength

The maximum width of the tooltip in pixels.

Type:: int

id_

The identifier for the scheduled tooltip display.

Type:: int or None

tw

The tooltip window instance.

Type:: tk.Toplevel or None

__init__(widget, text='widget info')

A tooltip class for Tkinter widgets.

This class provides a tooltip that appears when the mouse hovers over a specified widget. The tooltip displays a text message and automatically hides when the mouse leaves the widget.

Parameters:

widget (tk.Widget) – The Tkinter widget to which the tooltip is attached.
text (str, optional) – The text to display in the tooltip. Default is ‘widget info’.

enter(event=None): Schedule the tooltip to be displayed.

hidetip(): Hide the tooltip window.

leave(event=None): Unschedule the tooltip and hide it.

schedule(): Schedule the tooltip to be shown after the wait time.

showtip(event=None): Create and display the tooltip window.

unschedule(): Cancel the scheduled tooltip display.

opencsp.common.lib.tool.tk_tools.window(*vargs, TopLevel=False, **kwargs)

Initialize and return a new Tkinter window or Toplevel instance.

This function attempts to create a Tkinter window. If the creation fails, it retries up to two more times. This is useful in cases where initializing a Tkinter window may fail due to environmental issues.

Parameters:

*vargs (tuple) – Positional arguments to pass to the Tkinter window constructor.
TopLevel (bool, optional) – If True, create a Toplevel window instead of a Tk window. Default is False.
**kwargs (dict) – Keyword arguments to pass to the Tkinter window constructor.

Returns:

A new Tkinter window or Toplevel instance.

Return type:

tk.Tk or tk.Toplevel

Raises:

Exception – If the window creation fails after three attempts.

Notes

If the window creation fails, a warning is logged, and the function will wait for a second before retrying.

Examples

>>> main_window = window()
>>> top_window = window(TopLevel=True)

Typing

class opencsp.common.lib.tool.typing_tools.Series(data=None, index=None, dtype: Dtype | None = None, name=None, copy: bool | None = None, fastpath: bool | lib.NoDefault = <no_default>): Bases: Series, Generic[T]

Get the default value if the primary value is None or if the value is a callable that raises an error.

Examples:

default(None, "a") # returns "a"
default(None, None) # returns None
default(None, None, None, "a") # returns "a"

lval = [ 1, 2, 3, 4 ]
dval = { 1: 2, 3: None }
default(lambda: lval[3], "a") # returns 4
default(lambda: lval[4], "a") # returns "a"
default(lambda: dval[1], "a") # returns 2
default(lambda: dval[2], "a") # returns "a"
default(lambda: dval[3], "a") # returns "a"

default(raises_runtime_error, raises_runtime_error) # raises RuntimeError
default(raises_runtime_error, "a") # returns "a"
default(raises_runtime_error, raises_value_error) # raises RuntimeError
default(raises_runtime_error, raises_value_error, None) # returns None
default("a", raises_value_error) # returns "a"
default(None, lambda: lval[4], dval[3], lambda: dval[4], "a") # returns "a"

default("a") # raises ValueError
default(None) # returns None

This is especially helpful in cases where a simple check for None would cause python to raise an error, as in the following example:

sval: string = None
lval: list = None
dval: dict = { 1: 2, 3: None }
result = sval or lval or dval[1] or "a" # returns 2
result = sval or lval or dval[2] or "a" # raises a KeyError
result = default(sval, lval, dval[2], "a") # returns "a"

Returns:

first_non_none: any | None: The first value (either primary or one of the defaults) that isn’t None and doesn’t raise an exception.

Raises:

primary_exception: Exception: If the primary value is a callable, and all default values are callables, and all throw an exception when called, then the exception from the primary value is raised.

opencsp.common.lib.tool.typing_tools.ensure_not_generic(t: type | GenericAlias) → type: converts and types.GenericAlias to a regular type

opencsp.common.lib.tool.typing_tools.ensure_not_generic_alias(t: type | GenericAlias) → type: converts and types.GenericAlias to a regular type

opencsp.common.lib.tool.typing_tools.ensure_not_string(t: type | str, class_container: type) → type: converts a string representation of a type into the actual type

opencsp.common.lib.tool.typing_tools.strict_types(func)

Decorator to enforce strict typing on function arguments.

This decorator takes keyword arguments with associated types. If an argument passed to the function does not pass an isinstance test against the defined type, the function will raise a TypeError.

Important

This decorator is not robust. If it fails, it should be very clear. In such cases, it is recommended to remove the decorator, as it is likely a bug in the decorator.

Notes

Using this decorator may block some information about the decorated function from certain Python functions that return function information.
A type represented as a string (e.g., ‘Vxyz’) will not recognize subclasses properly and may break entirely in some instances.
Beware of using types that contain a secondary type (e.g., list[int]); the inner types will be ignored and may fail entirely.
The function does not currently check the return value (this may be added in the future).

Examples

>>> @strict_types
... def add(a: int, b: int):
...     return a + b

>>> add(1, 2)
3

>>> add(1.0, 2)
TypeError: incorrect types in 'add'.
    At positional argument 1: type of '1.0' is float, should be int

>>> add(1, b=2.0)
TypeError: incorrect types in 'add'.
    At keyword argument 'b': type of '2.0' is float, should be int

>>> add(b=2.0, a=1j)
TypeError: incorrect types in 'add'.
    At keyword argument 'b': type of '2.0' is float, should be int
    At keyword argument 'a': type of '1j' is complex, should be int

Unit Conversion

Converting units, e.g, inches to meters.

opencsp.common.lib.tool.unit_conversion.dpi_to_dpm(value_dpi: float) → float

Convert a value in dots per inch (DPI) to dots per meter (DPM).

Parameters:: value_dpi (float) – The value in dots per inch to be converted.
Returns:: The equivalent value in dots per meter.
Return type:: float

Examples

>>> dpi_to_dpm(300.0)
11811.023622047244
>>> dpi_to_dpm(72.0)
2834.645669291338

opencsp.common.lib.tool.unit_conversion.dpm_to_dpi(value_dpi: float) → float

Convert a value in dots per meter (DPM) to dots per inch (DPI).

Parameters:: value_dpm (float) – The value in dots per meter to be converted.
Returns:: The equivalent value in dots per inch.
Return type:: float

Examples

>>> dpm_to_dpi(11811.023622047244)
300.0
>>> dpm_to_dpi(2834.645669291338)
72.0

opencsp.common.lib.tool.unit_conversion.foot_to_meter(value_foot: float) → float: Returns a number of meters equal to a given number of feet.

opencsp.common.lib.tool.unit_conversion.inch_to_meter(value_inch: float) → float

Convert a value in inches to meters.

Parameters:: value_inch (float) – The value in inches to be converted.
Returns:: The equivalent value in meters.
Return type:: float

Examples

>>> inch_to_meter(1.0)
0.0254
>>> inch_to_meter(12.0)
0.3048

opencsp.common.lib.tool.unit_conversion.meter_to_foot(value_meter: float) → float: Returns a number of feet equal to a given number of meters.

opencsp.common.lib.tool.unit_conversion.meter_to_inch(value_meter: float) → float

Convert a value in meters to inches.

Parameters:: value_meter (float) – The value in meters to be converted.
Returns:: The equivalent value in inches.
Return type:: float

Examples

>>> meter_to_inch(0.0254)
1.0
>>> meter_to_inch(0.3048)
12.0