The /gbdb fileserver

Introduction

The cluster gods have provided us with a new fileserver that will hold all files referenced by genome browser tables. This is great because when there's trouble with /cluster or /projects, it won't break the browser on hgwbeta or genome. Also, mirror sites won't have to copy our /cluster and /projects hierarchies -- just the new fileserver's tree, /gbdb.

But, it does mean we have to take some additional steps when loading tables that refer to files: axtInfo, chromInfo, and extFile. We used to just load /cluster and /projects paths into those tables, but now here's what we have to do:

link to the /cluster or /projects path from hgwdev's virtual /gbdb
load the table, using the /gbdb path created above
before requesting a push of the table to hgwbeta, request a /gbdb rsync

Step 1. Link to the /cluster or /projects path from hgwdev's virtual /gbdb

hgwdev has a /gbdb on its local disk, with symbolic links to the real locations of files in /cluster and /projects. When hgwdev's /gbdb is rsync'd to the "real" /gbdb, the links are followed and the files are copied over whole. hgwdev's /gbdb has 2 levels of real subdirectories and then links, like this:

/gbdb/$db/$dirname/$link

For example, here are some links that already exist:

/gbdb/hg13/bacends.3/BACends.fa -> /cluster/store1/bacends.3/BACends.fa
/gbdb/mm2/nib/chr14.nib -> /cluster/store2/mm.2002.02/mm2/nib/chr14.nib
/gbdb/hg10/axtBestMm2/chr8.axt -> /cluster/store1/gs.11/build28/bed/blastz.mm2.2002-04-14/axtBest/chr8.axt

Note: since hgwdev's /gbdb just has links, the files can't be removed from /cluster or /projects as long as they are used by the browser! /gbdb isolates hgwbeta and genome from /cluster and /projects, but not hgwdev.

Example:

Say some day in the future, you build the blatFugu track for mm3. The fugu sequence data file is /cluster/store3/fuguSeq/fugu_v3_mask.fasta . When creating the link from /gbdb to the real location, you would use mm3 as $db, fuguSeq as $dirname, and fugu_v3_mask.fasta as $link in the new /gbdb/$db/$dirname/$link:

mkdir -p /gbdb/mm3/fuguSeq ln -s /cluster/store3/fuguSeq/fugu_v3_mask.fasta /gbdb/mm3/fuguSeq/ ls -l /gbdb/mm3/fuguSeq/

Step 2. Load the table, using the /gbdb path created above

You'll still use hgLoadRna, but instead of telling it the /cluster or /projects path, you tell it the /gbdb path. Continuing our example:

hgLoadRna addSeq mm3 /gbdb/mm3/fuguSeq/fugu_v3_mask.fasta

Step 3. Before requesting a push of the table to hgwbeta, request a /gbdb rsync

After testing your track and its detail pages on genome-test, and before asking for a push of mm3.extFile and chr*_blatFugu to hgwbeta, make the following email request:

 To:  push-request AT soe DOE ucsc DOT edu

 [appropriate greeting, propitiation, beer offerings etc]
 Please rsync hgwdev's /gbdb to the real /gbdb -- I added 
 /gbdb/mm3/fuguSeq/ for the blatFugu track.

For additional examples: search for "gbdb" in kent/src/hg/makeDb/makeHg13.doc.

Questions, comments, request for help with adding items to /gbdb...

Talk to angie@soe. ucsc. edu or hiram@soe. ucsc. edu