CMS Software Deployment Plan on OSG ( OSG-CSD ) Version 0.2 01 July 2005 Author : USCMS OSG-CSD Group ----------- Content ---------------------------------------------------------------------------- [1] Description of the CMS Software Deployment Activity [2] Required Resources and Target Sites [3] Dependencies and Interaction with Other Services [4] OSG-CSD Tools [5] Testing [6] Deployment Priorities, Policy, and Timeline [7] Monitoring/Bookkeeping Software Deployment [8] Support [9] Documentation ------------------------------------------------------------------------------------------------ ----------- Main ------------------------------------------------------------------------------- [1] Description of the CMS Software Deployment Activity ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Grid can provide computing resources and storages with great flexibility. Because of this flexibility, the same softwares can be installed without publication and notification to the software user community and may be only used by a single software user. Therefore, the flexibility brings the issue of using resources efficiently and preciously. Within a collaboration, if this type of activity could be organized and managed in a consistent manner, one could efficiently and preciously use the grid resources. Such a management of softwares in distributed environment could comprised of (a) No duplicate software installation, (b) Appropriate and timely deployment as well as publication of software (c) A well-defined validation procedure, and (d) Dedicated personnel to perform these activities synchronously and consistently. The well-organized software deployment will reduce the amount of gatekeeper access that is required to purely install the necessary softwares and will use minimal disk space. This might require a database management to book-keep all the software deployment activities. This definitely requires decent tools to download and install the necessary softwares in distributed environment or in local installation. One or two analysis channels can be used to validate the deployment as well as basic software functionality validations. Since the management of the software deployment has to be maintained and updated periodically, it might be nice to have a common user interface to perform the activities. The user interface can be used different software deployment librarians to manage the software deployment persistently if the web portal can be employed. Therefore, the main purpose of this activity is to help physicists analyze data on OSG sites. Furthermore, it would be desirable if this could be managed interoperably, for example with LCG sites. Another important purpose of this activity is to understand the user management in grid environment and to provide inputs to OSG. The following activities will be carried out by a few dedicated personnel. [A] Installation of the CMS softwares on OSG sites ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The installation of the CMS Software is the first step of the activity. Mainly, we will use the CMSI installation tool because the tool is a very resilient tool in installing libraries for the CMS physics analysis environment. [B] Validation of the CMS softwares on OSG sites ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The validation of the CMS Software is to assure the quality of the software installation performed in step [A]. At least one analysis channel will be used to validate the installed software. Very fundamental physics quantities, e.g., cross section, may be compared among different installation sites. A few shapes of fundamental physics quantities may be compared, too. [C] Maintenance of the CMS softwares on OSG sites ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ CMS software updates will be checked periodically. The new updates will go through the cycle of [A] and [B]. No particular continuous monitoring will not be necessary the installed software will be validated as in step [B]. Outdated softwares will be retired. But software preservation could be arranged by providing a mechanism to notify the maintainer such that the particular software should not be retired for certain time period. [D] Publishing of the CMS software installation ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The validated installation will be declared "Deployed" and will be published via the discovery service or ldap-type service. The database should be able to be queried using either direct readonly DB access tool or service's client tools, for example xml-rpc. This activity will last until the activity is dismissed by the DCT authority or by the group's decision. [2] Required Resources and Target Sites ----------------------------------- The required resource for OSG-CSD is a linux machine with a web server and an MySQL installed. At the moment, already existing web server can be used, but later, a dedicated web server might be necessary. The MySQL will be used to maintain the deployment bookkeeping locally and the information stored in the MySQL will have to be consistent with discovery service and ldap-type service. Initially, target sites will be OSG sites with SL3 gatekeeper node and worker node(s) to simplify the deployment. The target sites can be extended per demands. [3] Dependencies and Interaction with Other Activities ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ OSG-CSD will depend on the CMS software releases. Mainly, those are CMKIN, OSCAR, and ORCA. Other CMS softwares also can be installed per request. Since new installation and update can affect jobs that are already running on a site, the new installation or update should be coordinated within the USCMS-DCT. The installed softwares should be published and the publishing might require coordination with the software discovery service ( how can I as a grid user publish this ?) and the ldap-type service. A user should be able to introspect contents of the published software installations. [4] OSG-CSD Tools ^^^^^^^^^^^^^ Although CMSI is written in perl, OSG-CSD tool will be based on autoconf, shell and python scripting. The base tools will be used locally and grid-wise. So these tools are written in a templated way. The grid-wise installation tool is written to carry out the CMSI installation mechanism on OSG sites. The installation can be configured to install softwares through thge fork jobmanager or a batch jobmanager. The python scripting is implemented and is used to interface to a web server so that the maintainer can perform the tasks in a persistent environment. The interface script This also can provide the environment where a few maintainer can work on the same task on a rotation base. There is a demo portal at: https://gdsuf.phys.ufl.edu:8443/gcalog/gcalog [5] Testing ^^^^^^^ OSG-CSD tool is developed and will be further developed. The softwares installed will be tested by validation using at least an analysis channel on a couple of pioneer OSG sites. [6] Deployment Priorities, Policy, and Timeline ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The first deployment priorities will go to ORCA installation and validation. If we have enough requests, We can go and install further CMKIN and OSCAR. The CMS software installed in this management will be typically in the so-called "$app/cmssoft" directory. If some site wants the librarian to install the softwares to a different directory, such site's admin should inform the librarian or the librarian should ask the site admin where to install. Timeline [A] Develop OSG-CSD tool - Phase 1 - 05/20/2005 [B] Validation 1 - Phase 1 - 05/27/2005 [C] Develop OSG-CSD tool - Phase 2 - 06/10/2005 [D] Validation 2 - Phase 2 - 06/17/2005 [E] Start Publishing - Phase 2 - 07/01/2005 [7] Monitoring/Bookkeeping Software Deployment ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Monitoring will occur through the discovery service or ldap-type service. Bookkeeping will be managed by the MySQL database. The MySQL database can be open to anyone with readonly connectivity or xml-rpc service. [8] Support ^^^^^^^ Bug tracking system will be prepared to solve/support problems related with OSG-CSD. DCT email list will be used to respond questions and to solve problems. [9] Documentation ^^^^^^^^^^^^^ This documentation will server as the base documentation for OSG-CSD. Tool's README, INSTALLATION, and OPERATION guide will be prepared. [References] [1] CMSI: CMS Software Installation Toolkit [2] CMS Software Deployment Grid User Interface Portal (CSDGUI) : https://gdsuf.phys.ufl.edu:8443/csdogrid/csdogrid [3] OSG-CSD Operations Guide ( to be written)