Steve at CERN

Next Week EGEE 09

2009-09-17T17:52:00.002+02:00

Next week is of course EGEE 09 in Barcelona. As a warm up the EGEE SA1 OAT sections a sneak preview.

http://www.youtube.com/watch?v=PADq2x8q0kw

Installed Capacity at CHEP

2009-03-23T08:06:00.009+01:00

This week is CHEP 09 proceeded by WLCG workshop. I presented some updates on the roll out of the installed capacity document. It included examples of a few sites that would have zero capacity if considered under the new metrics.
Sites should consider taking the following actions.

Check gridmap. In particular the view obtained by clicking on the more label and selecting the "size by SI00 and LogicalCPUs".
Adjust your published #LogicalCPUS in your SubCluster. It should correspond to the number of computing cores that you have.
Adjust your #Specint2000 settings in the SubCluster. The aim is to make your gridmap box the correct size to represent your total site power.

The followup questions were the following. Now a chance for a a more reflected response.

Will there be any opportunity to run a benchmark within the gcm framework?
I answered that this was not possible since unless it could be executed in under 2 seconds then there was no room for it. Technically there would not be a problem with running something for longer, it could be ran rarely. We should check how the first deployment of GCM goes, longer tests are in no way planned though.
What is GCM collecting and who can see its results?
Currently no one can see on the wire since messages are encrypted. There should be a display at https://gridops.cern.ch/gcm however currently it is down but once there it will be accessible to IGTF CA members. For now there are some test details available.
When should sites start publishing the HEPSpecInt2006 benchmark?
The management contributed "Now" which is of course correct, the procedure is well established. Sites should be in the process of measuring their clusters with the HEPSpec06 bench mark. With the next YAIM release they will be able to publish the value also.
If sites are measuring these benchmarks can they the values be made available on the worker nodes to jobs?
Recently the new glite-wn-info made it as far as the PPS service. This allows the job to find on the WN to which GlueSubCluster it belongs. In principal this should be enough, the Spec benchmarks can be retrieved from the GlueSubClusters. The reality of course is that until some future date when all the WNWG recommendations are deployed along with CREAM also then this is not possible. So for now I will extend glite-
wn-info to also return a HepSpec2006 value as configured by the site administrators.
Do you know how many sites are currently publishing incorrect data?
I did not know the answer nor is an answer easy other than collecting the ones of zero size. Checking now of 498 (468 unique?) SubClusters some 170 of them have zero LogicalCPUs.

On a more random note a member of CMS approached me afterwards to thank me for the support I gave him 3 or so years ago while working at RAL. At the time we both had an interest in making grid work. He got extra queues, reserved resources, process dumps and general job watching from me. It was the fist grid jobs we had approaching something similar to the analysis we now face. Quoting the gentleman from his grid experience and results using RAL he obtained his doctorate and CMS chose to use the grid.

SA1 Nagios Deployment Update

2009-01-29T09:58:00.005+01:00

The EGEE SA1 Nagios bundle was released yesterday with significant updates.

GOCDB Integration: A list of Sites can now be collected using the GOCDB's new API. In particular a list of sites in a ROC or in a Country can be monitored extending the previous LDAP filter on Sites.
GOCDB Downtimes: Downtimes entered in the GOCDB are now also pulled into and inserted as NAGIOS downtimes for your services.
HGSM Integration: HGSM is the SouthEast Europe equivalent to the GOCDB.
NDOUtils Installed: NDOUtils sits behind NAGIOS and fills in a MySQL database with NAGIOS's configuration and metric and test results.
New SRM Tests: These mimic some of the logic of the existing SAM SRM tests. The eventual replacment to the SAM SRM tests. In NAGIOS speak we now have an active check that submits scripts and returns passive results for each of steps of the lcg-cr, lcg-rep, lcg-del seem before.
NSCA Installed: Especially for the case where two nodes are used, a NAGIOS node and NRPE triggered UI then passive test results are submitted back via NSCA from the NRPE-UI. Well almost - Bug.
New BDII Checks: These are the checks taken directly from the gstat2 work but now running against your services.
New msg-to-queue Service: Running on a NAGIOS box this subscribes to externally executed test results for your Site or ROC from the ActiveMQ messaging system. Currently nothing is actually coming in but much of the infastructure is now there.

As before installation can still be done completly via YAIM both for a site or ROC. New packages can be followed for i386 or x86_64. And of course bugs and feedback are always welcome.

The update contains work from Emir, James, Laurence, Konstantin and myself.

Top BDII Publishing Revamp

2008-10-17T11:09:00.003+02:00

I have updated the topbdii summary page.
It now includes an historical element as well so you can see the individual site inclusions in each top level BDII over a period of time. There are clearly some topBDIIs which are flakier than others represented by vertical blue lines and some sites are dropped at times by all BDIIs represented by horizontal blue lines.

Also the tables are broken down by BDII version numbers as well.

WNWG Update at the GDB.

2008-05-13T17:21:00.004+02:00

Tomorrow I present an update of the Worker Node Working Group at the GDB. There is now a complete plan that while complicated with updates to information providers, lcg-tags, YAIM and site configurations of YAIM it will work though add complexity. As planned small sites can ignore everything. It seems it can be deployed at its own speed and once done sites can reconfigure to multiple sub clusters in their own time. The presentation has far to much detail for a GDB audience but it's quite a narrow subject. Mostly of interest to site administrators since it is all work for them. The users just get the benefits, better allocation of jobs to suitable worker nodes.

Interesting GGUS Posts

2008-04-25T22:29:00.002+02:00

In the last two or three weeks GGUS is now serving an experimental rss feed of new posts. Watching the posts there is a whole new view on what is really happening at a low level of 1 to 1 GRID operations.... My aim is to post once a week a summary of the three or so most interesting or relevant tickets....

35783. Tristan Glatard reported that lcg-utils when called with timeout option, -t, rather than timing out gracefully exited with a segfault. In a reality it seems an upgrade fixes this though it is yet to be confirmed.
35798. Earlier in week a gLite release was made that turned out to break certain low level communications between the WN and CE at a globus level. The problem was fixed and released within a working day but with the rush it was never mentioned that the repositories for the different nodes types have been split for the first time. This is transparent if sites follow the install instructions to the letter but I am sure many do not.... I don't. Sites should be told that the glite-WN is currently a break away from everything else.

Plotting the GlueSiteLocation

2008-03-20T23:39:00.006+01:00

I plotted the GlueSiteLocation as advertised in the GlueSites a few days ago. In fact a quick look over the results shows that the field is filled in well in all but a few cases. Most of the plots are centered on a country with a few "World" ones showing everything. Chris and Steve as a result have an action to fix their location some time whenever they fancy it:-)

Filling in the GlueSite

2008-03-14T20:42:00.003+01:00

The last few days I have again wanted to resolve a given site name as belonging to an EGEE ROC. There are no public interfaces to resolve something that would be useful to so many people including GGUS, the GridPP Real Time Monitor, Operations, Accounting and even the Managers. So I have created a review and proposal to fill the GlueSite object in with well defined information. I have spoken to the WLCG coordinators and will follow up in time with ROC managers, GGUS, OSG the WLCG monitoring working group and any one else I can think of who might be interested. Eventually when something
settles down I'll push it through EGEE deployment and YAIM and onto the sites.

Graph the FTS Deployment

2008-02-25T17:56:00.004+01:00

The FTS deployment is of course made up of channels between sites in the EGEE/LCG grid. This plot shows all the channels as queried from the BDII as lines connecting the sites. The grouping of nodes by colours represents all the channels managed by a single FTS instance.

Again it is another plot that is almost impossible to use.

Graphing Glue

2008-02-18T22:56:00.019+01:00

I've been playing with GraphViz and HyperGraph to draw out the relationships between the objects in the GLUE schema. The results are interesting but especially for the complicated sites, e.g. CERN, GRIF, RAL,... they are hard to read. There are massive png's, vrml and svg formats. Also the html pages load a dynamic applet of the data. I'll request that they are worked into gstat which is the natural place for them to live once I have improved them. Warning: Some of the png images are huge and will most likely crash your browser. The image here is RAL-LCG2. The orange blob top right is the FTS.

Combined SLC4 and SL4 CE.

2008-02-14T16:40:00.004+01:00

As part of the WN working group efforts I've been finishing of a vandalised CE which is now part of the CERN_PPS site. It is two node cluster both of which are WNs. One is also a gatekeeper publishing GlueCE, VOViews and CESEBind objects, the other is publishing two GlueCluster and GlueSubClusters objects. ... To cut a long story short I have a single gatekeeper node which can matched to for either SL4 or SLC4 and the jobs route through to the correct node at the back. It is exactly the same for any GlueSubCluster attribute such as memory for instance. Tomorrow we try and thrash how to get the final steps of this into a release.... It is quite possibly not going to be pretty, keeping things backwards compatible may just not be possible or desirable. details.

WN Meeting Kickoff Last Week

2007-10-09T20:19:00.000+02:00

The first EGEE WN working group meeting happened last week. It went well. I had previously thought I knew exactly what the end point would be even if the exact details of migration were not together. The oversight for me was software tags. Namely that for a site publishing non-overlapping SubClusters then a software installation job has to publish software tags in the correct SubCluster. Exactly how you determine which SubCluster you are installing for is far from obvious. No one has suggested a better solution than a configuration file different on every batch worker, there has to be a better way....

From the EGEE conference there was an addition to the group's scope. The group will give help, advice support, approval, something, ... to the WMS wrapper script addition of doing something sensible in grid jobs during the SIGTERM -> SIGKILL window given by the batch system. Francesco's presentation
looks to have considered everything but can check with group if any comments.

One of the consequences of offering users better job matching to WNs is that the sites are possibly going to start killing more excessive jobs dead. A timely addition if the job wrapper now handles the SIGTERMs.

FTS Release Progress

2007-09-19T18:59:00.000+02:00

At last the FTM node has actually made it to be released on the PPS shortly. I really don't want to think how long it has taken. Longer than you can possibly imagine. Also this week I tried the SL4 build of the FTS for first time. Installation has been fine but configuration now looks to stuck and will need some new builds to be solved. Most of the problems are changes between gLite 3.0 and 3.1 rather than the OS upgrade. Having said that most of the changes between 3.0 and 3.1 look sensible and bits of yaim configuration are being removed basically.

Torque Maui Builds from ETICS.

2007-09-14T20:44:00.000+02:00

Finally got around to pushing maui and torque through ETICS. Bang up to date releases for sl3 and sl4 with i386 or X86_64 are all available. Others like SL5 should now be trivial. The torque build was fairly straight forward where as the maui build does some black magic for the torque dependency it needs. Recently there have been many requests for the X86_64 builds and it is needed anyway for the upcoming official gLite X86_64 WN which is around the corner now. Also the Dubliners have been wanting it for ages to build other exotic platforms like MacOS. Releases, CVS Repos. To finish this off I need to close down some of pages hosted at RAL and GridPP with pointers to the new locations. Not bad for a day when I got my notice of contract termination.

Back to School.

2007-09-10T20:35:00.000+02:00

After a two week break without a single keystroke pressed I've been back for my first day. Big outstanding items are starting a TCG inspired group to investigate and conclude on finding and utilizing individual worker node resources. My first thoughts is that it is all done but there are massive gaps in flexibility of configuration or deployment to achieve it..., it is not done in other words. Other than this the FTM node I've been trying to deploy was rejected from certification for a lack of timestamps in it's logfiles. This is fair enough, I thought it at the time but ignored the problem. Finally the SL4 FTS must be installed, the compilation is there but who knows if it works.

SAM test CE-sft-posix now running.

2007-08-16T09:44:00.000+02:00

After deployment mistakes and corrections the CE-sft-posix test is now running again. The significant change is that now new SEs that appear pass for the first two weeks. Previously they were guaranteed to fail for the first two weeks. The results look promising in that out of 287 CEs tested today 69 or so look to fail. This includes the 10 or so CERN CEs where there may be a problem with the configuration of the atlas-durable and lhcb-durable SRMs but perhaps only with respect to the OPS vo which is probably not well tested. Next step will be to get CERN passing. Need to have my own house in order before proceeding. Following that go through the other failiures to look if they seem fair.

Learning Oracle

2007-07-27T21:30:00.000+02:00

This week I was on the Oracle Administrators Course I and II. Apparently because we are CERN and bright young things they compressed the two week course into a one week course. At first it might not be obvious that the course would be directly related what I do day-to-day but I now certainly understand better what the DBAs are doing near by that the FTS uses. In the past they have been easily able to baffle me with the science of blocks, fragmentation, performance tuning, ... The course was given by Lutz Hartmann, a so called Oracle ACE no less. He was genuinely enthusiastic to teach us and it was all pretty effective. I learned as much as I possibly could in one week so am pleased. I now have one year to do my exams to get the Oracle Administrator qualification. It has to be useful for a post CERN life apart from anything else.

Dependents

2007-07-17T23:11:00.000+02:00

This month and either side is the much touted as the gLite restructuring exercise. The aim is chuck out old code and clean up some of what is there especially looking at dependencies and alike. A review basically. Seems I have been given the LFC and DPM to review which is good, I think they should be fairly obvious. The lack of java makes them easier compared to what it could be. There was some suggestions that the dependencies could be determined from the ETICS system, it is good that the final RPM result is now being used. It is this the final product after all which aim to clean up.

Cardinal Sin of Grid Operations

2007-07-04T20:43:00.000+02:00

CERN today committed the cardinal sin of grid operations. They allowed a host certificate to expire for a production service. This took the MyProxy service used by the FTS out for almost 24 hours. The service is a victim of its own success because it had been running itself for the last year without interruption but when it came to it fell between the cracks of responsibility. I've every confidence it will never happen again. At least with this service anyway.

FTM Node is Getting There

2007-07-03T21:09:00.000+02:00

The new named FTM (File Transfer Monitoring) node made reached a significant point today. All the packages are done including the GridView publishing ones. Also the yaim configuration is done within the new glite-fts-yaim module. But there seems to be some dependecy problem which is best resolved by just avoiding the problem and moving to SL4. This will be a pain since in the first instance the FTM monitoring will require a different OS to the FTS itself. The dependecies may be able to improved furthur as well. Currently VDT is needed just to provide an openldap client to the BDII plugin within the glite-sd-query tool. Hopefully this can be removed. I'm sure it does not need to be there.

OS Matching

2007-07-03T21:07:00.000+02:00

I've now published a recipe for matching RHEL3 , RHEL4 OS clones. It's a lot more complicated that it should be and no doubt there will be grumbling but I am happy it is correct.

A Good Match

2007-06-29T22:58:00.000+02:00

A busy week this week where I collected more things to do than I completed. There were some complaints from sites that some VOs have asked them to switch of their resources when their SE is not present for whatever reason. It turns out that CMS already have a good query for finding CEs and SEs but it just needs a bit added to to not match CEs when the SE is not published. This should help sites and users a lot if something sensible can be matched. In the process discovered that there is work going on now to write a more intelligent service information provider in the UK. It looks good in principal but I must give some comments before it gets further. On a similar matching note CERN is about to go live with some SL4 resources which means people are now actually worried about matching to versions of OSes. It seems a bug reported as fixed a while back may in fact not be fixed though I need to check this fix has actually gone into latest WMS that was deployed today.

Writing a New YAIM Component

2007-06-20T17:04:00.000+02:00

Committed and started running a new YAIM component today to configure the new FTM node. It is going to be really trivial to do the code, basically I need to edit some text files and turn some services on so I should be able to handle that. I decided to dig out ed for the purpose. It was over 10 years ago worryingly that someone first showed me ed and tried to convince me that if you wanted to edit a text file from a script it was the perfect tool. At the time it seemed far to backwards, today it does indeed seem to be the perfect tool for the job. The initial testing went okay and uncovered several bugs or improvements to be made to YAIM core which are now submitted. The new modular yaim looks like a very sensible idea. It allows me to work on the FTS part without fear of breaking or holding up the rest of the world.

RPMs not ETICS RPMs

2007-06-19T17:36:00.000+02:00

After generating RPMS with ETICS and being generally dissatisfied with them I investigated what others are doing. It seems the order of the day is just to write your own .spec file and tell ETICS not to attempt to do so. It defeats the object of ETICS in that is meant to be package neutral but it decreases my trivial installation instructions from 10 points to just 2 since the rest is handled by a decent package. It should make the subsequent upgrades trivial as well as opposed to a complete reconfiguration from scratch. In fact ETICS is adding features all the time such as recently it now supports the %config RPM directive that is a step in the right direction to make the autogenerated spec files better.

FTS Upgrade at CERN

2007-06-18T19:02:00.000+02:00

After careful planning the the CERN FTS servers, production T0 export, tiertwo and the pilot were all upgraded today to version 2.0. It basically went okay and much to plan with completion 10 minutes before the announced intervention period end was reached. In fact two things went wrong. The quattor wrapper for yaim was never tested with the new YAIM, the new glite-FTS2 and glite-FTA2 targets had to merged in by hand. That was my mistake. The other problem still needs investigating but basically huge fragmentation to the production database appears to have (a) made the schema update a lot slower than expected and (b) an index has become corrupted. All in all it looks like although things are working some more downtime will be needed now perhaps longer than the upgrade itself to clean the database up. All in all it makes my life a lot easier, there is now a lot less left that I am managing from before my time getting to CERN.

Nagios Sensor gather_sam Now Released

2007-06-14T15:44:00.000+02:00

At the EGEE operations meeting a presentation form the WLCGMG described their first sensor that might be useful especially to sites running or considering to run nagios. Basically a nagios sensor that will raise the SAM alarms on your local hosts in your nagios system. They are currently looking for a few volunteer sites as early adopters.

Stockholm SA1 Meetings

2007-06-12T08:51:00.001+02:00

This week I'm visiting Stockholm for the EGEE SA1 meetings. No doubt something interesting will happen to put here.

FTS Monitoring RPMS

2007-05-31T00:09:00.000+02:00

Have now started a concerted effort to package each of the FTS scripts that are lying around as FtsAddons. One is done now, one of Ron's scripts. Now started on Poalos scripts and by chance there was some interest in this from Taiwan's T1 so looks like I have a tester that is always good. Other than that I seem to be dashing between French and the Hospital for a dodgy thumb that I have. Took the minutes for the EMT as well. This is easily the best meeting to understand what is going in the upcoming software.

COD's Mercy

2007-05-25T16:21:00.000+02:00

I've been CIC-on-Duty this week, lots of problems caused by poorly tested tests such as R-GMA not being on port 8443 at many sites. The good thing though it has given the procedure for new tests a wake up call and hopefully next time it will go better. This is good since the next test may well be the posix test of mine. End of the week has been difficult though, a scheduled GGUS intervention followed by an unscheduled SAM outage and now the GOCDB has fallen over. Of course these three events all happened on consecutive days.

Using GCalDeamon Now

2007-05-20T18:59:00.000+02:00

I used to publish my Apple iCal calendars to a webdav and from there import them is ical format to gcal. I've now configured GCalDaemon to do a two way sync between ical and gcal. Previously I could not update gcal version of my calendar. This is a lot better since I need to work in gcal anyway to update Jenny's calendar. So far it looks to be working perfectly.

Ubuntu on my Mac

2007-05-18T16:38:00.000+02:00

Installed VitualBox on my Mac today followed by Ubuntu. Looks to be much the same as Parallels but it is free to use in binary form for personal use and the underlying code is even opensource so that is good. So far it seems to be working. The reality though is I have not missed linux on my laptop since getting a mac.

I built an RPM

2007-05-18T16:29:00.000+02:00

Building an RPM would not normally be exciting but this was out of the EGEE CVS using ETICS to build and publish an RPM. The RPM is in reality about as simple as it gets and contains basically a cron, some scripts and logrotate to do some monitoring of the FTS. I still have some unanswered questions, for instance I'm currently unable to tag a version nor do I know how to generate a nightly build of particular ETICS tag. For now this is fine and allows Andrey to proceed with doing the rest.

ETICS Progress

2007-05-15T15:48:00.000+02:00

Made some progress today with ETICS and actually have it making a package. However did not manage to do it with out getting expert help in from someone who is far to busy to help my small problems. Turned out I was using an old client, referenced on twiki page all be not an official ETICS page and there was a bit of the web interface I could click that I did not realise I could click.

Eradicating the Hand of the Developer

2007-05-14T18:22:00.000+02:00

After what was meant to be an easy addition to the FTS today of another webserver turns out there was yet another file that has been added to make it work by the developers. It's a classic problem of course. I guess one day that layer of access should be closed down. Hopefully with the upcoming upgrade to the FTS there will be time to wipe the service and start again. The only sure way to check what you think quattor is going to do actually happens at install time and you end up with a working service.

BES, DRMAA and Torque

2007-05-14T18:18:00.000+02:00

Started to look at an OGF standard today BES. Looks to be some standard interface to batch systems that various bits of EGEE will support including cream. However I thought that DRMAA was a similar thing from OGF to give a method for how to interact with batch systems. Need to find out more about both to understand how they are distinct.

Pilot FTS to PPS FTS

2007-05-11T15:00:00.000+02:00

Looked yesterday at the changes that would be applied to the FTS pilot if it was brought in line with the PPS deployment of the FTS. In principal they are the same but the pilot has pre-release versions of the RPMs. In reality it looks like just about every RPM will get an upgrade so hopefully we will end up with the same thing that works the same or at least as well as the pilot. This will definetley be done before the production FTSes get the upgrade.

ETICS Virgin

2007-05-11T14:57:00.000+02:00

Made a first serious attempt today to add a component into ETICS. In terms of software to be added it is very trivial. Just some scripts to be run as a cron to generate some web reports from the FTS. So after reading through quite few python stack traces it turns out I only have read access to the bits of ETICS I needed to operate on. Have requested more access now.

Quattor and RPMS

2007-05-09T17:06:00.000+02:00

Spent all most the whole day writing quattor configuration files for the upcoming FTS release. Very painful, it is basically equivalent to writing an rpm database by hand via trial an error. One of those tasks that in hindsight it would have been easier to write a script but anyway....

Java and Jpackage

2007-05-08T11:55:00.000+02:00

Wrote a page describing howto install JPackage versions of JDK on Scientific Linux. It is lot easier to use the JPackage ones rather than the SUN supplied ones but requires a change in practice probably for many sites who were already doing something else.

RPM cleanup on FTS Servers

2007-05-07T20:11:00.001+02:00

Not very interesting but fixed up the local CERN FTS hacks to comply to the new yaim and ncm-yaim that have been applied across all services at CERN. Also the split in the SetToDesiredState packages has been accommodated. In the process I noticed that the SRM and castorgrid hosts were broken due to this so alerted the relevant people to correct the situation. Also asked about how and when the SetToDesiredState was useful which I now understand. This had been something I had not understood since learning of its existence shortly after arriving at CERN...

Jpackage and Tomcat

2007-05-04T15:35:00.001+02:00

There were some reports from the EMT that installing tomcat5 from jpackage with SUN's JDK 1.5 is a little problematic. Had a look at it today and now I understand it all though how to proceed is unclear. In short tomcat5 requires xml-commons-jaxp-1.3-apis which obsoletes xml-commons-apis. This property is however is provided by SUN's JDK 1.5 and so installing the xml-common api has the result of removing SUN's JDK! Submitted a bug to jpackage and provided a recipe of installation order that works. Bug 266.

FTS Monitoring Spider

2007-05-03T16:18:00.001+02:00

Another meeting about monitoring the FTS and how to create summary of tables of transfers by categorized errors and alike. In fact the stuff already done on the FTS spider looks good. It just needs to be brought closer to the main development and do more in Oracle rather than in PHP.

Voms2gacl Utility Written.

2007-05-03T00:55:00.000+02:00

I've written a short script as an adventure for me python. voms2gacl downloads the xml formatted member of a VOMS server and creates .gacl for use by mod_gridsite. In short it is an easy way to restrict access to webpages by VO for web-browsers. voms2gacl. In fact I wrote it a while ago but I realised I never posted it.

Photos from HEPiX

2007-05-02T23:14:00.000+02:00

Following the HEPiX meeting last week I took some photos of the event but mainly the after hours events. HEPiX Photos

GMOD, ROC and Meetings.

2007-05-02T21:36:00.000+02:00

This week I'm both the GMOD and the CERN ROC which basically results in lots of meetings. The GDB was today as well but I was unable to attend much of it due to other meetings. In the EMT meeting I was surprised that moving to jpackage looks to be non-trivial. The most basic thing of using Java 1.5 and tomcat looks to not be obvious but I think with the correct combination of magic it should work.

French Classes Restart

2007-05-02T21:34:00.000+02:00

French classes restarted on Monday. They are now on Monday, Wednesday and Thursday mornings. So far so good and this time I really want to put some work into get the most out of them. Face it I am never going to get a better chance to learn a language than I do now.

Two HEPiX Talks Archived

2007-05-01T08:01:00.001+02:00

I gave two talks at the DESY HEPiX meeting both of which are now archived for prosperity.

DPM Update.

SRM update.

For both you need real player installed to view them. If I could work out how to convert them to avi or mpeg I would.

Increasing Reliability by Testing

2007-04-26T16:12:00.001+02:00

Interesting talk about the hardware testing that CERN does both at burn in and also routinely. A lot of these things such as fsprobe, SMART, inventory and memory tests are ran routinely on the boxes I run so it is good to here a description of what they are doing.

SL3 Security Updates

2007-04-26T14:36:00.001+02:00

Looks like SL3 will be maintained with vendor's security updates until 2010. This is good news for EGEE who are behind the previous published end of life for the release of the SL4 service nodes.

WLCG System Admin Group

2007-04-26T11:32:00.000+02:00

A summary of the WLCG system admin group was given. Some of the things they are up to include starting a wiki to collect tips and scripts together that people are working on. e.g torque, maui and cfengine recipes. It is a general Cookbook of ideas. There is also a subversion repository and again uses gridsite like the wiki does. There are currently 9 scripts in the repository and more are needed. The need more volunteers which is exactly why publicising at events like HEPiX.

Sharepoint and TWiki

2007-04-23T16:11:00.001+02:00

According to the CERN status report the use of Microsoft's sharepoint instead of the CERN twiki. Not something that I would ever like to see.

HEPiX Goes Green

2007-04-23T14:20:00.001+02:00

Seems there has quite a theme of building new computer centers around the globe at HEP sites. But interesting is that many of them include the use of waste heat from the machine rooms to heat the office space.

Another XEN advantage for Batch Workers

2007-04-23T11:48:00.001+02:00

Brookhaven presented that they were running batch jobs in XEN machines on there compute farm. The aim is that they also serve the storage space within dCache or xrootd and want to protect this storage from batch jobs that crash the machine.

PSI Site Report - Gateway Machine.

2007-04-23T10:26:00.001+02:00

One intresting idea from PSI. Rather than having a single SSH gateway they have a gateway that once you are logged into allows you at the firewall to login to any machine. Much better, going through gateways is always a pain.

HEPiX at DESY

2007-04-23T08:50:00.001+02:00

This week I'm in Hamburg at a HEPiX meeting. The meeting where the sysadmins from the large labs get together. I'll be writing notes on many of the presentations here.

VOMS Groups for the FTS

2007-04-20T15:36:00.001+02:00

Two new roles have now been added to the dteam voms service. These are ftsadmin and ftsmaster. They are associated with regions, the first say /dteam/uki/Role=ftsadmin would contain a list of people able to edit FTS channels around the globe with RAL as endpoint. We still need to check that it really works on the FTS but it is meant to.

FTS Status Monitoring

2007-04-20T11:27:00.001+02:00

A long meeting this morning about improving the reporting that the FTS is able to give. The addition of summary tables from triggers for instance. But also looking at historical information for the states of the system and individual services. The followup for me is to check what the information is that is going to be recorded and then considering what is useful or missing for admins in particular.

FTSreport Yaim Component

2007-04-19T16:33:00.001+02:00

Made some commits today to the a new yaim component for the FTS report server.

In particular this will be the first update I've submitted to an EGEE release for a long time. A good feeling even if it is a trivial change.

IT-GD-OPS Group Meeting

2007-04-19T11:36:00.001+02:00

Attended the grid operations meeting with the new format of groups rather than individuals reporting. It made it a lot better and I was not desperate to get out as has been the case before.

Another FTS CDB Cleanup

2007-04-18T15:31:00.001+02:00

I'm happier about how the FTS prod and pilot services are now represented in CDB. The pilot is no longer a modified prod service and is defined in its own right. This should remove some problems which are not really there for any general FTS installation.

CDB Access Controls

2007-04-18T10:17:00.001+02:00

After a request for the voms admins I looked into how CDB enforces access controls on templates. It is a simple system with its own commands within CDB to edit the ACLs. Very easy once you know how.

FTS and Oracle

2007-04-17T18:00:00.000+02:00

Tried and failed a lot today to install a "package" on the FTS oracle servers to do some statistics collection. It is now done but due to the intervention of others. I need to learn some Oracle and fast if I don't want to feel helpless a few times a week as is the case today.

Ports on WNs

2007-04-17T17:56:00.000+02:00

Big discussion today about some gLite middleware that has started all be it two years ago opening up services in user space on the WN so refreshed proxies can be injected from the CE. I'm convinced myself this is a bad idea and it is something we have to stop. Lots of discussion and some reluctance to actually do it but I think soon it will be okay.

Scribefire an Blogger

2007-04-17T17:55:00.000+02:00

For some reason Scribefire stopped working with Blogger today, don't know why just some odd DOM error.

Sindes and SSH

2007-04-16T14:06:00.001+02:00

Showed another GD admin how to do the upload download ssh key trick in quattor. Its a shame the rest of the world does not do this.

EMT Meetings

2007-04-16T13:58:00.001+02:00

Looks like I will now be attending the EMT meeting twice a week. This is good since I will be able to get a good handle on what is coming up in the release. Of course it is two extra meetings a week but it should be useful information that I listen to.

FTS Reports Setup

2007-04-16T13:56:00.001+02:00

On the FTS I've set up the reporting package, I have to wait really till 08:00 tomorrow to check if it works which is a little boring of course. If it works then I'll look into running three instances of it for the different FTSes but it looks like it is easy to do but only time will tell of course.

Reorganise my Bookmarks

2007-04-13T16:47:00.001+02:00

Not the most productive of tasks but is friday afternoon after all. Have moved all my bookmarks to del.ico.us. Hopefully it pays of, seems like a good idea though. Big advantage is that bookmarks have tag's rather than being in a directory structure.

Twiki Pages for FTS 2.0

2007-04-13T13:29:00.001+02:00

I've now made a complete copy of the FTS1.5 to become the FTS2.0 pages for the upcoming release. Needs the wise men to look over them all now.

Twiki Experiments

2007-04-12T15:51:00.001+02:00

Spent an age trying to mark pages as secret so only a small group can see the pages before we publish them. It is easy to do but an odd thing which makes sense is you then can't search the pages with a  formatted search since the search is anonymous. Of course it makes sense but....

IT-GD Section Meeting

2007-04-12T10:59:00.001+02:00

Much discussion about how to improve the current round table, it will be reported just verbally as group reports. Some discussions about GMOD as well which was inconclusive.

LSF LCG CEs

2007-04-11T17:57:00.001+02:00

Had to work out how to drain an LSF CE today at CERN. A trivial things for me in PBS , in fact the LSF info provider looks to be a lot better written. I guess it was written by a sysadmin rather than some random developer.

ce101 Reported as Faulty

2007-04-11T11:40:00.001+02:00

Had a report in that ce101 was faulty here at CERN. Starting the draining process so someone else can sort it out. It is great not actually having to do the harddisk but my self :-)

VOMS Roles and the FTS

2007-04-11T11:39:00.001+02:00

Wrote a a new page FtsVomsRoles detailing how VOMS roles might be used with the FTS. The FTS already supports this so we just need a schema for actually using them. Needs a bit of central management but still a lot better than what we had before.

Who is Hitting the FTS

2007-04-11T11:36:00.001+02:00

Wrote a small micky mouse script to parse the FTS call.logs. Intrestingly it showed that in the last day for some users they are submitting up to 3000 times as many status requests as they are submitting transfers which seems a little excessive. To be followed up.

High Load on FTS Webserver

2007-04-10T17:52:00.001+02:00

Report of high load of the FTS webservers. It is clear they are busy but there is no script or log file written to do a quick analysis of what the queries are. So it is time to start one. The log format completely changes for release 2.0 so no point spending any time on this.

FTS Documentation for the 20 Release

2007-04-10T17:50:00.001+02:00

Started the empty documentation pages for what will be the FTS 2.0 release. There is a lot of information in the old release  pages. The challenge is making use of it while not including things that are plain wrong.

cx_Oracle Module

2007-04-10T11:40:00.001+02:00

Had a very first go connecting to Oracle with the cx_Oracle python module. It is easy of course like everything once you find the correct level of web page to help you do it. It is needed for some of the FTS monitoring that is being done.

End of GMOD

2007-04-10T09:58:00.001+02:00

I was GMOD last week over the easter period, fairly quiet other than a couple of announcements. The gmod phone did ring but I missed the call, checking the CERN status within SAM it was obvious that something was up since all the queues were in state queuing. I raised a ticket with ce.support and although it was not them who called there was an obvious problem. A cron had been left in place to drain the CERN farm which had been wanted the previous week.

FTS TWiki Pages Review

2007-04-05T14:19:00.001+02:00

Have been working hard with FtsWlcg < LCG < TWiki the homepage of the FTS for the WLCG service. A lot of the pages had fallen way out of date, learnt a lot of twiki in the process including how to do templates, search for particular pages related to a page and many other things. It is looking a lot better but there are still things that need to be done for sure.

Green Plates Progress

2007-04-05T12:50:00.001+02:00

Seven months are starting at CERN there looks to be some progress with my green plates, they should be ready next week or rather I will be able to go to a garage and get them made.

Adding the VOMS manager to Quattor

2007-04-05T12:15:00.001+02:00

Going through the many steps at the moment with Remi so that he can install boxes and configured them with CDB, swrep, etc. Seems to hard to do this a bit and not give access for Remi to do everything.

Small FTS Update

2007-04-05T12:13:00.001+02:00

Updated the FTS T2 service to use a different oracle string, same database different way of accessing. I need to learn some more oracle.

Trying out Scribefire.

2007-04-05T01:08:00.001+02:00

This is a blogging tool integrated into firefox.

First day Back with a Meeting bang.

2007-04-05T00:06:00.000+02:00

As per usual having been away a week for a wedding party a very busy day of catchup. Highlights included.

Attended 9.00 sysadmin meeting.
Attended 10.00 WLCG meeting.
Attended 1.30 meeting with Remi to go over CDB, quattor and things.
Attended 2.30 meeting with LHCb people looking into using MAUI with Dirac.
Attended 3.00 meeting of sysadmins.
Attended 4.30 meeting of EMT.

In between cleared up my inbox and processed 900 new mails.

Crank It Up

2007-04-05T00:04:00.000+02:00

After sometime and following me trying to remember what I did for an appraisal since starting at CERN I must keep this blog up to date.

More XEN and DESY

2007-01-16T09:47:00.000+01:00

Sven gave a talk on use of XEN at DESY. Lots of good things, they consider the problems are that:

XEN is not integrated into the kernel.
MAC Address managment has to be done.
Automatic Installation.
Full virtualisation IO is very slow.

Owen Presents What is Going on at DESY

2007-01-16T09:29:00.000+01:00

DESY using XEN to maintain build and testing boxes. Owen gave a good summary of the other virtualizations tools out there including the new KVM and OpenVZ which I know little about.

Virtulisation at DESY

2007-01-16T09:26:00.000+01:00

Am attending the Virtualisation and GRID meeting at DESY organised by Owen Synge. It is my first time at DESY but seems like a nice place. Staying on a fireboat as well which is pretty smart.

Neglect

2007-01-16T09:25:00.000+01:00

So I've been neglecting this which is a shame. It is useful to look back on.

Return after a Week

2006-12-12T10:54:00.000+01:00

After getting married and being ill for a couple of days I'm back at work. I have 400 emails to work through which is a lot less than I used to have to deal with. This week I am ROC on Duty for CERN and I also want to get my updated ncd-yaim package deployed. If I get them done I'll be happy.

I am a CE Expert!

2006-11-30T10:54:00.000+01:00

Have volunteered to be a CE expert at CERN. I don't really know anything about the gLite CE but this is one very good way to learn of course though not as good as running one completely. Essentially tickets end up with me when they are not easy to solve by the admins. A lot of ticket passing goes on here.

IT-GD-OPS Section Meeting

2006-11-30T10:39:00.000+01:00

Attended the IT-GD-OPS section meeting and reported what I have been up to in the last week:

More clean up of FTS profiles. A bit closer to actually being able to deploy new the new hardware. Just need to understand how the load balancing works.
Starting to impart the knowledge to Yvan about quattor.
Acting as CERN ROC. There really has been nothing interesting that has happened though.
Started to contact people for the WLCG ops meeting. Only one firm offer of a talk at the moment.

Interesting points that were raised here:

Presentation about the utilities that ship the STDOUT, STDERR to an SE for inspection during a job. It runs every now and then GridFTS them off. It appends or creates new files depending on the file size. It all makes sense to me, lots of questions about it though. I guess lots of bright people creates lots of good ideas. Though if the interactive jobs work is this even needed.

FTS Profiles, Just Load Balancing To Go

2006-11-28T19:33:00.000+01:00

Really getting some where now with the FTS CDB profiles. The three clusters are now nicely abstracted from the template layout and there are no longer bits of similar configuration dotted about the different files. However there is still one of configuration still dotted about which is the load balancing magic. This is quite CERN specific and so finding out how it works is going to take a bit of hunting I suspect. The real problem for me though is that the load balance configuration is currently active and live on the real production system, I could really mess things up if not careful.

Tutorials at CERN

2006-11-28T07:21:00.000+01:00

Asked a couple of people if they would give talks with positive responses from one. The other one is my favorite though. Also asked at the WLCG operations meeting for ideas and offers of talks. Of course everyone was quiet as is always the case at the OPS meeting. Must now get in touch Ale' to see if she can help out as well.

ROC On Duty

2006-11-27T16:05:00.000+01:00

Very little happening for the CERN ROC this week though it looks we will be taking on three or four US sites. I've dealt with IOWA before and know they have a quite different setup. For instance GPBOX is deployed and used there. There was some real confusion earlier in the year with them advertising support for the MINOS VO not accepting jobs for them.

Deputy GMOD and CERN ROC on Duty

2006-11-27T09:29:00.000+01:00

This week I have two roles which move around the group within rotation. The GMOD is meant to coordinate between the grid services for the group and dispatch tickets of to the rest of the group. The ROC on duty just deals with GGUS tickets assigned to the CERN ROC. Apart from that my other plans for the week are to look at tutorial session for the LCG meeting and get some concrete decisions from people to give talks.

Learning PAN

2006-11-24T19:00:00.000+01:00

Tried some more complicated things in PAN today in the process of trying to rationalize the templates for FTS service. In particular conditionals and nested nlists are quite cool. What I was pleased to find out was that having come up with a plan of sensible way to handle multiple sub-clusters and nodetypes I looked for some examples to build up my idea. It looks like the CASTOR team is doing exactly the same thing. The CASTOR templates look to be in a much better state than some of the other ones.

Tutorials at LCG Workshop

2006-11-24T10:10:00.000+01:00

I have been given the task of chairing and planning the tutorial portion for the LCG workshop in January at CERN. In fact this looks to be good thing to do, there are several good ideas and plans that have been given to me or I can think off. There are even some volunteers to give presentations which is just great.

CDB is Way Too Slow

2006-11-23T17:27:00.000+01:00

Spent about 30 minutes today putting in an update to CDB. It failed which is fine but it took over an hour to find out. Need to find out what can be done to waste less time when I make trivial mistakes.