GroupEventStoreToolkit description
GroupEventStoreToolkit is a set of python modules and scripts
grouped together in a single package. It's a prototype of administrative
tools necessary to manage EventStore. Currently is uses three 3d party
python modules: MySQLdb, sqlitem bsddb3 (for Berkeley DB support).
The BerkeleyDB support is abandoned in favor of SQLite.
GroupEventStoreToolkit API
The GroupEventStoreToolkit consists from the following modules:
ESBuilder.py
ESDump.py
fileContent.py
ESManager.py
sql_util.py
convert.py
file_util.py
cgiDBAccess.py
cgiDumpFileContent.py
cgiEventStoreUtils.py
binary_dump.py
binary_reader.py
binary_utils.py
build_binary_location.py
build_key_from_binary.py
build_key_from_pds.py
build_key.py
build_pds_location.py
key_dump.py
lbin_dump.py
lpds_dump.py
pds_dump.py
pds_reader.py
pds_utils.py
Here we will discuss high-level API highlighed in blue color.
The low-level modules are format dependent file utilities with
self-exaplaining names.
The ESManager.py python
module consists of a single class ESManager.
It should be initialized with the following set of parameters:
- data base connector
no default
- location of output key/location files
default: location of input data files
- specific version name
default is composed as:
nodename_sysname_OSrelease_machineName_date_localtime
- administrative version (or grade)
default: physics
- time stamp
default: 0
- view
default: all
- flag to generate key files
default: yes
- verbose flag
default: quiet
- flag indicating which DB to use.
This flag is necessary since SQL-syntax is slightly
different for MySQL and SQLite.
default: use MySQL
The ESManager class has
two high-level API methods:
int updateDB(genMode, fileList):
genMode informs if key file generation is enabled
fileList is a list of files to be added to DB
Returns status upon completion. Failure status allow parent code to
perform necessary actions, such as rollback transaction.
It uses the following low-level API methods:
int updateXXXXDB(args):
update methods for XXXX DB table, e.g. LocationDB. The args is a set
of arguments which form this table
int updateDBUsingGroupList(groupList):
int updateDBUsingFileList(generationMode, fileList):
int updateDBWithoutKeyFile(fileName):
int updateDBFromIDXA(fileName):
run-condition update methods
int moveFilesInES(fileIn,fileOut):
fileIn could be a single file or directory name. In last case
all files from that directory are considered to be moved in EventStore.
fileOut either file or directory name. The following logic has been
applied: check if fileIn exists in DB, check permission for fileOut
to be written, perform DB update, move files. Returns status
upon completion. The status information can be used by main applications
to monitor this transaction and invoke necessary actions.
int deleteGrade(delVer,delTime):
deleteGrade is used to remove data from EventStoreDB.
The process identifies valid data/key/location files for
removal. Key/location files are removed from ESDB and
associative data files are marked as orhpans in OrphanFileIDDB.
The sql_util.py module (SQLUtil class) provides
high-level routines to manipulate EventStore DB tables.
It should be initialized by
- db cursor
- db type
- verbose flag
It has the following API:
list dbNames():
return names of EventStore DB tables
list dbContent():
return a list of table fields in EventStoreDB
void dropTable():
drop table from EventStoreDB
void createTables():
create all EventStore DB tables
void printDBContent(table):
print the content of given table
Main applications (command-line tools):
The ESBuilder.py is in
charge of collecting input information from users,
keeping log of users actions with EventStoreDB and transaction log
(need to be implemented). It uses MySQL or SQLite transactions for
all actions.
The ESDump.py provides
information about EventStore DB tables.
The fileContent.py
prints content of supporting file formats (pds, binary, key,
pds location, binary location). The details of output can be controlled
by verbose option.
The convert.py converts
one DB to another (MySQL<->SQLite)
The file_util.py
provides high-level format independent
API to parse and
build data and key/location files. It consists from the following
methods:
list fileParser(fileName,what=""):
parse input file with optional parameter
what=(run,uid,syncValue,proxies). Return a combined list of
fields.
void build_key(fileIn,fileOut,oFileID):
build key file for given data file and its file ID. Output file
written to fileOut.
void
build_location(fileIn,fileID,fileOut,tagList=[]):
build location file for given data file, its file id and tagList
which vetoes events. Output file written to fileOut.
string fileType(fileName):
return file type (pds, bin, key, lpds, lbin).
USE CASES:
Right now toolkit is capable of doing the following tasks:
- create a new EventStore
- add file and/or set of files to DB based on path/fileName/pattern
- create key and location files for PDS and binary file data formats
- add data from event list based on input IDXA file
- move file inside EventStore
- delete grade from EventStore
- performs necessary updates in EventStore tables
- physically move file
- keeps history of user commands
- print content of underlying DB tables
- print content data/key/location files
- the following formats are supported:
PDS, binary, key, PDS location, binary location
- convert BerkeleyDB<->MySQL<->SQLite
It also has the following features:
- transaction log
- history log, to keep history of users requests
Web interface
The web interface is written in python as CGI module. It consists of
the following modules:
cgiDBAccess.py
cgiDumpFileContent.py
cgiEventStoreUtils.py
The cgiDBAccess.py
module provides the web form interface with
Information/Administration/Expert
pages to perform various tasks.
Information
page
performs the following queries:
- Number of events for give grade/view/runRange
- List of data (proxies) stored for date/grade
- Version details for date/grade/view/run
- Comparison of two date stamps; For each date it summarize
runRange, number of events, specific version, difference in
runRange and data
Administrative page
generates set of arguments for command-line tool
and performs administrative tasks with EventStore DB.
Expert page
allows to look at table contents, location of files, form
event list for given grade/view/run.
Below you can find links to EventStore web interface and user
documentation:
Web
interface
User
documentation