5.6.5. conditions_db

Python interface to the ConditionsDB

class conditions_db.BearerAuth(token)

Simple class to present bearer token instead of username/password

class conditions_db.ConditionsDB(base_url=None, max_connections=10, retries=3)

Class to interface conditions db REST interface

BASE_URLS = ['http://belle2db.sdcc.bnl.gov/b2s/rest/']

base url to the conditions db to be used if no custom url is given

exception RequestError

Class to be thrown by request() if there is any error

check_payloads(payloads, information='payloadId')

Check for the existence of payloads in the database.

Parameters:

• payloads (list((str,str))) – A list of payloads to check for. Each payload needs to be a tuple of the name of the payload and the md5 checksum of the payload file.

• information (str) – The information to be extracted from the payload dictionary

Returns:

A dictionary with the payload identifiers (name, checksum) as keys and the requested information as values for all payloads which are already present in the database.

create_globalTag(name, description, user)

Create a new globaltag

create_iov(globalTagId, payloadId, firstExp, firstRun, finalExp, finalRun)

Create an iov.

Parameters:

• globalTagId (int) – id of the globaltag, obtain with get_globalTagId()

• payloadId (int) – id of the payload, obtain from create_payload() or get_payloads()

• firstExp (int) – first experiment for which this iov is valid

• firstRun (int) – first run for which this iov is valid

• finalExp (int) – final experiment for which this iov is valid

• finalRun (int) – final run for which this iov is valid

Returns:

payloadIovId of the created iov, None if creation was not successful

create_payload(module, filename, checksum=None)

Create a new payload

Parameters:

• module (str) – name of the module

• filename (str) – name of the file

• checksum (str) – md5 hexdigest of the file. Will be calculated automatically if not given

delete_iov(iovId)

Delete an iov

Parameters:

iovId (int) – id of the iov to be deleted

get_all_iovs(globalTag, exp=None, run=None, message=None, run_range=None, fully_contained=False)

Return list of all payloads in the given globaltag where each element is a PayloadInformation instance

Parameters:

• gobalTag (str) – name of the globaltag

• exp (int) – if given limit the list of payloads to the ones valid for the given exp,run combination

• run (int) – if given limit the list of payloads to the ones valid for the given exp,run combination

• message (str) – additional message to show when downloading the payload information. Will be directly appended to “Obtaining lists of iovs for globaltag {globalTag}”

• run_range (tuple) – if given limit the list of payloads to the ones overlapping with the given run range, if

• fully_contained (bool) – if True and the run_range is not None it limits the list of payloads to the ones fully contained in the given run range

Warning

Both, exp and run, need to be given at the same time. Just supplying an experiment or a run number will not work

static get_base_urls(given_url)

Resolve the list of server urls. If a url is given just return it. Otherwise return servers listed in BELLE2_CONDB_SERVERLIST or the builtin defaults

Parameters:

given_url (str) – Explicit base_url. If this is not None it will be returned as is in a list of length 1

Returns:

a list of urls to try for database connectivity

get_globalTagInfo(name)

Get the id of the globaltag with the given name. Returns either the id or None if the tag was not found

get_globalTagType(name)

Get the dictionary describing the given globaltag type (currently one of DEV or RELEASE). Returns None if tag type was not found.

get_globalTags()

Get a list of all globaltags. Returns a dictionary with the globaltag names and the corresponding ids in the database

get_iovs(globalTagName, payloadName=None)

Return existing iovs for a given tag name. It returns a dictionary which maps (payloadId, first runId, final runId) to iovId

Parameters:

• globalTagName (str) – Global tag name.

• payloadName (str) – Payload name (if None, selection by name is not performed.

get_payloads(global_tag=None)

Get a list of all defined payloads (for the given global_tag or by default for all). Returns a dictionary which maps (module, checksum) to the payload id.

get_revisions(entries)

Get the revision numbers of payloads in the database.

Parameters:

entries (list) – A list of payload entries. Each entry must have the attributes module and checksum.

Returns:

True if successful.

has_globalTag(name)

Check whether the globaltag with the given name exists.

modify_iov(iovId, firstExp, firstRun, finalExp, finalRun)

Modify the validity range of a given iov

Parameters:

• iovId (int) – id of the iov to be modified

• firstExp (int) – first experiment for which this iov is valid

• firstRun (int) – first run for which this iov is valid

• finalExp (int) – final experiment for which this iov is valid

• finalRun (int) – final run for which this iov is valid

request(method, url, message=None, *args, **argk)

Request function, similar to requests.request but adding the base_url

Parameters:

• method (str) – GET, POST, etc.

• url (str) – url for the request, base_url will be prepended

• message (str) – message to show when starting the request and if it fails

All other arguments will be forwarded to requests.request.

set_authentication_token(token)

Set authentication token when talking to the database

Parameters:

token (str) – JWT to hand to the database. Will not be checked for validity here.

staging_request(filename, normalize, data, password)

Upload a testing payload storage to a staging globaltag and create or update a jira issue

Parameters:

• filename (str) – filename of the testing payload storage file that should be uploaded

• normalize (Union[bool, str]) – if True the payload root files will be normalized to have the same checksum for the same content, if normalize is a string in addition the file name in the root file metadata will be set to it

• data (dict) –

  a dictionary with the information provided by the user:

  • task: category of globaltag, either main, online, prompt, data, mc, or analysis

  • tag: the globaltag name

  • request: type of request, either Update, New, or Modification. The latter two imply task == main because if new payload classes are introduced or payload classes are modified then they will first be included in the main globaltag. Here a synchronization of code and payload changes has to be managed. If new or modified payload classes should be included in other globaltags they must already be in a release.

  • pull-request: number of the pull request containing new or modified payload classes, only for request == New or Modified

  • backward-compatibility: description of what happens if the old payload is encountered by the updated code, only for request == Modified

  • forward-compatibility: description of what happens if a new payload is encountered by the existing code, only for request == Modified

  • release: the required release version

  • reason: the reason for the request

  • description: a detailed description for the globaltag manager

  • issue: identifier of an existing jira issue (optional)

  • user: name of the user

  • time: time stamp of the request

• password – the password for access to jira or the access token and secret for oauth access

Returns:

True if the upload and jira issue creation/upload was successful

upload(filename, global_tag, normalize=False, ignore_existing=False, nprocess=1, uploaded_entries=None)

Upload a testing payload storage to the conditions database.

Parameters:

• filename (str) – filename of the testing payload storage file that should be uploaded

• global_tage (str) – name of the globaltag to which the data should be uploaded

• normalize (Union[bool, str]) – if True the payload root files will be normalized to have the same checksum for the same content, if normalize is a string in addition the file name in the root file metadata will be set to it

• ignore_existing (bool) – if True do not upload payloads that already exist

• nprocess (int) – maximal number of parallel uploads

• uploaded_entries (list) – the list of successfully uploaded entries

Returns:

True if the upload was successful

class conditions_db.PayloadInformation(payload_id, name, revision, checksum, payload_url, base_url, iov_id=None, iov=None)

Small container class to help compare payload information for efficient comparison between globaltags

base_url

base url

checksum

checksum of the payload

classmethod from_json(payload, iov=None)

Set all internal members from the json information of the payload and the iov.

Parameters:

• payload (dict) – json information of the payload as returned by REST api

• iov (dict) – json information of the iov as returned by REST api

iov

interval of validity

iov_id

iov id in CDB, not used for comparisons

name

name of the payload

payload_id

payload id in CDB, not used for comparisons

payload_url

payload url

readable_iov()

return a human readable name for the IoV

revision

revision, not used for comparisons

property url

Return the full url to the payload on the server

conditions_db.chunks(container, chunk_size)

Cut a container in chunks of max. chunk_size

conditions_db.enable_debugging()

Enable verbose output of python-requests to be able to debug http connections

conditions_db.encode_name(name)

Escape name to be used in an url

conditions_db.file_checksum(filename)

Calculate md5 hash of file

conditions_db.get_cdb_authentication_token(path=None)

Helper function for correctly retrieving the CDB authentication token (either via file or via issuing server).

Parameters:

path – Path to a file containing a CDB authentication token; if None, the function will use a default path (${HOME}/b2cdb_${BELLE2_USER}.token or $TMPFILE/b2cdb_${BELLE2_USER}.token) to look for a token.

conditions_db.require_database_for_test(timeout=60, base_url=None)

Make sure that the database is available and skip the test if not.

This function should be called in test scripts if they are expected to fail if the database is down. It either returns when the database is ok or it will signal test_basf2 that the test should be skipped and exit

conditions_db.set_cdb_authentication_token(cdb_instance, auth_token=None)

Helper function for setting the CDB authentication token.

Parameters:

• cdb_instance – An instance of the ConditionsDB class.

• auth_token – A CDB authentication token: if None, it is automatically retrieved from the issuing server and then set

5.6.6. conditions_db.iov

This module contains classes to work with validity intervals. There’s a class for a single interval, IntervalOfValidity and a class to manage a set of validities, IoVSet, which can be used to manipulate iov ranges

class conditions_db.iov.IntervalOfValidity(*iov)

Interval of validity class to support set operations like union and intersection.

An interval of validity is a set of runs for which something is valid. An IntervalOfValidity consists of a first valid run and a final valid run.

Warning

The final run is inclusive so the the validity is including the final run.

Each run is identified by a experiment number and a run number. Accessing first or final will return a tuple (experiment, run) but the elements can also be accessed separately with first_exp, first_exp, final_exp and final_run.

For final there’s a special case where either the run or both, the run and the experiment number are infinite. This means the validity extends to all values. If only the run number is infinite then it’s valid for all further runs in this experiment. If both are infinite the validity extends to everything.

For simplicity -1 can be passed in instead of infinity when creating objects.

static always()

Return an iov that is valid everywhere

>>> IntervalOfValidity.always()
(0, 0, inf, inf)

contains(exp, run)

Check if a run is part of the validity

property final

Return the final valid experiment,run

property final_exp

Return the final valid experiment

property final_run

Return the final valid run

property first

Return the first valid experiment,run

property first_exp

Return the first valid experiment

property first_run

Return the first valid run

intersect(other)

Intersection with another iov.

Will return None if the payloads don’t overlap

>>> iov1 = IntervalOfValidity(1,0,2,5)
>>> iov2 = IntervalOfValidity(2,0,2,-1)
>>> iov3 = IntervalOfValidity(2,10,5,-1)
>>> iov1.intersect(iov2)
(2, 0, 2, 5)
>>> iov2.intersect(iov3)
(2, 10, 2, inf)
>>> iov3.intersect(iov1) is None
True

property is_open

Check whether the iov is valid until infinity

subtract(other)

Return a new iov with the validity of the other removed. Will return None if everything is removed.

Warning

If the other iov is in the middle of the validity we will return a tuple of two new iovs

>>> iov1 = IntervalOfValidity(0,0,10,-1)
>>> iov2 = IntervalOfValidity(5,0,5,-1)
>>> iov1 - iov2
((0, 0, 4, inf), (6, 0, 10, inf))

property tuple

Return the iov as a tuple with experiment/run numbers replaced with -1

This is mostly helpful where infinity is not supported and is how the intervals are represented in the database.

>>> a = IntervalOfValidity.always()
>>> a
(0, 0, inf, inf)
>>> a.tuple
(0, 0, -1, -1)

union(other, allow_startone=False)

Return the union with another iov.

>>> iov1 = IntervalOfValidity(1,0,1,-1)
>>> iov2 = IntervalOfValidity(2,0,2,-1)
>>> iov3 = IntervalOfValidity(2,10,5,-1)
>>> iov1.union(iov2)
(1, 0, 2, inf)
>>> iov2.union(iov3)
(2, 0, 5, inf)
>>> iov3.union(iov1) is None
True

Warning

This method will return None if the iovs don’t overlap or connect to each other as no union can be formed.

Parameters:

• other (IntervalOfValidity) – IoV to calculate the union with

• allow_startone (bool) –

  If True we will consider run 0 and run 1 the first run in an experiment. This means that if one of the iovs has un unlimited final run it can be joined with the other iov if the experiment number increases and the iov starts at run 0 and 1. If this is False just run 0 is considered the next run.

  >>> iov1 = IntervalOfValidity(0,0,0,-1)
  >>> iov2 = IntervalOfValidity(1,1,1,-1)
  >>> iov1.union(iov2, False) is None
  True
  >>> iov1.union(iov2, True)
  (0, 0, 1, inf)
  

class conditions_db.iov.IoVSet(iterable=None, *, allow_overlaps=False, allow_startone=False)

A set of iovs.

This class allows to combine iovs into a set. New iovs can be added with add() and will be combined with existing iovs if possible.

The final, minimal number of iovs can be obtained with the iovs property

>>> a = IoVSet()
>>> a.add((0,0,0,2))
>>> a.add((0,3,0,5))
>>> a.add((0,8,0,9))
>>> a
{(0, 0, 0, 5), (0, 8, 0, 9)}

add(iov, allow_overlaps=None)

Add a new iov to the set.

The new iov be combined with existing iovs if possible. After the operation the set will contain the minimal amount of separate iovs possible to represent all added iovs

>>> a = IoVSet()
>>> a.add((0, 0, 0, 2))
>>> a.add((0, 3, 0, 5))
>>> a.add((0, 8, 0, 9))
>>> a
{(0, 0, 0, 5), (0, 8, 0, 9)}
>>> a.add(IoVSet([(10, 0, 10, 1), (10, 2, 10, -1)]))
>>> a
{(0, 0, 0, 5), (0, 8, 0, 9), (10, 0, 10, inf)}

Be aware, by default it’s not possible to add overlapping iovs to the set. This can be changed either on construction or per add call using allow_overlap

>>> a.add((0, 2, 0, 3))
Traceback (most recent call last):
    ...
ValueError: Overlap between (0, 0, 0, 5) and (0, 2, 0, 3)
>>> a.add((0, 2, 0, 3), allow_overlaps=True)
>>> a
{(0, 0, 0, 5), (0, 8, 0, 9), (10, 0, 10, inf)}

Parameters:

• iov (Union[IoVSet, IntervalOfValidity, tuple(int)]) – IoV or set of IoVs to add to this set

• allow_overlaps (bool) – Can be used to override global overlap setting of this set to allow/restrict overlaps for a single insertion operation

Warning

This method modifies the set in place

clear()

Clear all iovs from this set

contains(iov)

Check if an iov is fully covered by the set

>>> a = IoVSet([(0,0,2,-1), (5,0,5,-1)])
>>> a.contains((0,0,1,-1))
True
>>> a.contains(IntervalOfValidity(0,0,3,2))
False
>>> a.contains(IoVSet([(0,1,1,23), (5,0,5,23)]))
True
>>> a.contains(IoVSet([(0,1,1,23), (5,0,6,23)]))
False
>>> a.contains((3,0,4,-1))
False

Parameters:

iov (Union[IoVSet, IntervalOfValidity, tuple(int)]) – IoV or set of IoVs to be checked

Returns:

True if the full iov or all the iovs in the given set are fully present in this set

copy()

Return a copy of this set

property final

Return the final run covered by this iov set

>>> a = IoVSet([(3,0,3,10), (10,11,10,23), (0,0,2,-1), (5,0,5,-1)])
>>> a.final
(10, 23)

property first

Return the first run covered by this iov set

>>> a = IoVSet([(3,0,3,10), (10,11,10,23), (0,0,2,-1), (5,0,5,-1)])
>>> a.first
(0, 0)

property gaps

Return the gaps in the set. Any area not covered between the first point of validity and the last

>>> a = IoVSet([(0,0,2,-1)])
>>> a.gaps
{}
>>> b = IoVSet([(0,0,2,-1), (5,0,5,-1)])
>>> b.gaps
{(3, 0, 4, inf)}
>>> c = IoVSet([(0,0,2,-1), (5,0,5,-1), (10,3,10,6)])
>>> c.gaps
{(3, 0, 4, inf), (6, 0, 10, 2)}

intersect(iov)

Intersect this set with another set and return a new set which is valid exactly where both sets have been valid before

>>> a = IoVSet()
>>> a.add((0,0,10,-1))
>>> a.intersect((5,0,20,-1))
{(5, 0, 10, inf)}
>>> a.intersect(IoVSet([(0,0,3,-1), (9,0,20,-1)]))
{(0, 0, 3, inf), (9, 0, 10, inf)}

Parameters:

iov (Union[IoVSet, IntervalOfValidity, tuple(int)]) – IoV or set of IoVs to intersect with this set

property iovs

Return the set of valid iovs

overlaps(iov)

Check if the given iov overlaps with this set.

In contrast to contains this doesn’t require the given iov to be fully covered. It’s enough if the any run covered by the iov is also covered by this set.

>>> a = IoVSet([(0,0,2,-1), (5,0,5,-1)])
>>> a.overlaps((0,0,1,-1))
True
>>> a.overlaps(IntervalOfValidity(0,0,3,2))
True
>>> a.overlaps(IoVSet([(0,1,1,23), (5,0,5,23)]))
True
>>> a.overlaps(IoVSet([(0,1,1,23), (5,0,6,23)]))
True
>>> a.overlaps((3,0,4,-1))
False

Parameters:

iov (Union[IoVSet, IntervalOfValidity, tuple(int)]) – IoV or set of IoVs to be checked

Returns:

True if the iov or any of the iovs in the given set overlap with any iov in this set

remove(iov)

Remove an iov or a set of iovs from this set

After this operation the set will not be valid for the given iov or set of iovs:

>>> a = IoVSet()
>>> a.add((0,0,10,-1))
>>> a.remove((1,0,1,-1))
>>> a.remove((5,0,8,5))
>>> a
{(0, 0, 0, inf), (2, 0, 4, inf), (8, 6, 10, inf)}
>>> a.remove(IoVSet([(3,0,3,10), (3,11,3,-1)]))
>>> a
{(0, 0, 0, inf), (2, 0, 2, inf), (4, 0, 4, inf), (8, 6, 10, inf)}

Parameters:

iov (Union[IoVSet, IntervalOfValidity, tuple(int)]) – IoV or set of IoVs to remove from this set

Warning

This method modifies the set in place