FAQ  GENERAL iRODSIs there a diagram somewhere of how this stuff is setup?- For NCMIR with CCDB, there is a diagram, here.
What directories store iRODS "stuff"?- /opt/irods
- /etc/irods
- /var/opt/irods
- /var/run/irods
- /etc/init.d
. What files are most important in configuration and troubleshooting of iRODS? - /opt/irods
- /server/bin/irodsctl (binary invoked by initscript)
- /server/config/reConfig (iRODS rules)
- /etc/irods
- /<proj><env>irods (configuration files that tune the binary to a project and environment)
- /var/opt/irods
- /<proj><env>irods/log (log files)
- /var/run/irods
- /<proj><env>irods.pid (process lock file - sticks around if irods dies messily)
How can I increase the logging level?- (iRODS 3.2) export spLogLevel=5, then restart iRODS
- (iRODS 3.2) spLogLevel can be set by editing irodsctl.pl or setting the environment variable spLogLevel and then restarting the server. If set to "5", it is equal to LOG_NOTICE which will log messages of the 'notice' severity and more severe (lower numbers).
What decides the default resource? NOTE: defaultResource does not apply to irodsAdmin users |
- (iRODS 2.5) This and other rules are in the /opt/irods/server/config/reConfig/*.irb files.
- (iRODS 3.2) This and other rules are in the /opt/irods/server/config/reConfig/*.re files.
Are the iRODS configuration files templated?- Yes. See modules/m_irods/templates
Where do the values for the templates get set?- (iRODS 2.5) In the project module. See modules/p_ccdb/manifests/init.pp for an example.
- (iRODS 3.2) In the m_irods module. See modules/m_irods/manifests/params.pp and modules/m_irods/manifests/ccdb/params.pp for an example.
How do I find all the files the iCAT thinks are on a particular resource?- iquest "SELECT RESC_VAULT_PATH, DATA_PATH, DATA_NAME WHERE DATA_RESC_NAME = 'ucsd-ccdb-prod-nas5'"
How do I move data from one resource to another?- In the example below:
- permissions and ownership of the original data is changed
- everything is owned by ccdbprodirods (aka rawdata)
- ccdbprodirods (aka rawdata) has rw to everything
- everything is group owned by gid=100 (aka neuro, aka users)
- the group has read/write to anything in the montage processed_data reconstruction segmentation directories
- any data already in iRODS is iphymv'd to the new resource, then
- any files that have been generated, but not yet sync'd (i.e. cataloged in the iRODS database) are moved to the new resource in order to keep all the data associated with a given MPID together, then
- any symbolic links that have been created are moved to the new resource
[from the directory above the relevant project on the resource that has the data you need to move...]
for i in project_20103/microscopy_46211
do for j in montage processed_data reconstruction segmentation
do echo $i/$j
chown -R 1138 $i
chmod -R u+rw $i
chgrp -R 100 $i
chmod -R g+rw $i/$j
iphymv -r -M -R ucsd-ccdb-prod-nas2 /telescience/home/CCDB_DATA_USER.portal/CCDB_DATA_USER/acquisition/$i |
Before running the next bit, any directories that are missing in the new location need to be created |
cd $i
find . -type f -exec mv -b --suffix=20110815 -u -v {} /ccdbprod/ccdbprod2/home/CCDB_DATA_USER.portal/CCDB_DATA_USER/acquisition/$i/{} \;
find . -type l -exec mv -b --suffix=20110815 -u -v {} /ccdbprod/ccdbprod2/home/CCDB_DATA_USER.portal/CCDB_DATA_USER/acquisition/$i/{} \;
done
done |
I'm getting "rcAuthResponse failed with error -827000 CAT_INVALID_USER" in the log. What's wrong?- The user isn't in the iCAT. You may have misspelled the name, forgotten the extension, used the wrong zone or connected to the wrong environment.
I'm getting "rcAuthResponse failed with error -826000 CAT_INVALID_AUTHENTICATION" in the log. What's wrong?- The password you entered is incorrect.
I'm getting "ERROR: rmUtil: rm error for /telescience/home/CCDB_DATA_USER.portal/irmCDUunreg, status = -528002 status = -528002 UNIX_FILE_RENAME_ERR, No such file or directory" What's wrong?- Check the permissions on the disk. If you don't have permission to the unix file, it acts like the file isn't there.
I'm getting "iadmin rmresc ucsd-ccdb-prod-nas5 ; ERROR: Level 0: resource 'ucsd-ccdb-prod-nas5' contains one or more dataObjects ; ERROR: rcGeneralAdmin failed with error -835000 CAT_RESOURCE_NOT_EMPTY". What's wrong?- Use the iquest command above for finding what the iCAT thinks is on a particular resource.
I'm getting "ERROR: rmUtil: rm error for /telescience/home/ccdbdevirods/irodsupgrade, status = -13000 status = -13000 SYS_NO_API_PRIV" when I try to 'irm' a file that I own. What's wrong?- In iRODS 3.2, the ACL on the top level collection is restricted to the admin user only. Use the -f flag on the 'irm' command, or modify the ACL on the trash collection.
I'm getting "ERROR: connectToRhost: error returned from host ccdb-dev-icat32.crbs.ucsd.edu status = -38000 status = -38000 SYS_AGENT_INIT_ERR ; ERROR: _rcConnect: connectToRhost error, server on ccdb-dev-icat32.crbs.ucsd.edu is probably down status = -38000 SYS_AGENT_INIT_ERR" when I run 'ils' on the iCAT and it looks like the reagent is not starting because 'ps ax' has "10620 ? Ssl 0:00 ./irodsServer PROJECT=ccdbdevirods ; 10629 ? Z 0:00 [irodsReServer] <defunct>" What's wrong?- The reAgent isn't starting. Check both the rodsLog and the reLog for info. It was a parsing problem with the ccdbdev.re file in this case.
" when I try to 'irm' a file that I own. What's wrong?- In iRODS 3.2, the ACL on the top level collection is restricted to the admin user only. Use the -f flag on the 'irm' command, or modify the ACL on the trash collection.
Only rods user can write to dev1 and only rodsadmin can write to dev0. What's wrong?- I moved the dev0 vault from the slave to the iCAT using the 'iadmin modresc' command. When I moved the vault back to the slave, the problem went away.
I'm trying to do Resource Monitoring. How the heck do you get this to work?the rule is loaded in irods cron
```iqstat -v
id name
15249471
msiServerMonPerf("verbose","default");
```
I modified the path in the script because it traversed up one dir too far
```#$path = dirname($0)."/../../config/scriptMonPerf.config";
$path = dirname($0)."/../config/scriptMonPerf.config";
```
when I run the script by hand it runs and generates the scriptMonPerf.config file but gives an error
``` /var/lib/irods/msiExecCmd_bin/irodsServerMonPerf -v
/var/lib/irods/msiExecCmd_bin/irodsServerMonPerf: File system path is missing!
#0#96#98#0##0#0##
=========== Server Name : nyan.crbs.ucsd.edu ===========
=Msc= Operating system => Linux
=Msc= Number of processors => 8
=CPU= CPU utilization rate (%) => 0
=CPU= Run queue load => 0
=Mem= Total memory (Bytes) => 16823705600
=Mem= Fraction of the memory being used (%) => 96
=Mem= Fraction of the swap memory being used (%) => 98
=Mem= Pagging I/O activity => 0
=Dsk= Disk occupation ratio (%) for "" =>
=Dsk= Disk volume available (MB) for "" =>
=Net= Network interface max capacity (Mbits/s) => 1000
=Net= Network input activity (Mbits/s) => 1.368
=Net= Network output activity (Mbits/s) => 0.2736
=Net= Network input activity rate (%) => 0
=Net= Network ouput activity rate (%) => 0
```
I'm guessing that the fs is passed by the rule and is currently "default" what should I set it to? or does it come from somewhere else?
[11:17]
```-bash-4.2$ cat scriptMonPerf.config
systype:Linux
rhost:nyan.crbs.ucsd.edu
proc:8
swap:3
free:4
iswap:7
oswap:8
idle:15
mtu:1
iname:0
ipack:2
opack:6
diskused:0
diskabsavail:0``` |
> Hi Hao,
> thx, this actually what i did. I realized that the core.re file doesn't accept the documented syntax:
> { delay ... }
> input null
> output ...
you have to store this rule in a separate file on your local
file system or on an iRODS resource. It will be kept in the iCAT
as it is a delayed rule being run periodically. It is a little bit like a
cron tab. |
What does the process list look like when I use scp to copy something to iRODS via fuse?18380 pts/0 S+ 0:18 scp -p -C -r crbs-operator@poas.crbs.ucsd.edu:/datapool/ccdb/datajail/ccdbproddj0/gatan/ebushong/CCDBID_68618 .
18381 pts/0 S+ 3:16 /usr/bin/ssh -x -oForwardAgent no -oPermitLocalCommand no -oClearAllForwardings yes -C -lcrbs-operator poas.crbs.ucsd.edu scp -r -p -f /datapool/ccdb/datajail
18388 ? S 0:00 irodsAgent
18389 ? Ss 0:00 postgres: ccdbdevirods ccdbdevicat32 132.239.22.13(35895) idle
18394 ? S 0:03 irodsAgent
18395 ? Ss 0:01 postgres: ccdbdevirods ccdbdevicat32 132.239.22.13(35898) idle
18399 ? S 0:03 irodsAgent
18400 ? Ss 0:01 postgres: ccdbdevirods ccdbdevicat32 132.239.22.13(35901) idle
18401 ? S 0:00 [flush-253:2]
19430 ? S 0:00 irodsAgent
19431 ? Ss 0:00 postgres: ccdbdevirods ccdbdevicat32 132.239.22.13(36124) idle
19434 ? S 0:00 irodsAgent
19435 ? Ss 0:00 postgres: ccdbdevirods ccdbdevicat32 132.239.22.13(36127) idle
19436 ? Z 0:00 [irodsAgent] <defunct>
19448 pts/1 R+ 0:00 ps ax
24099 ? Ssl 0:00 ./irodsServer PROJECT=ccdbdevirods
24108 ? S 0:12 irodsReServer
|
It looks like the iCAT can't talk to the Oracle database. What should I do?If you see the following in the /var/opt/irods/ccdb<env>irods/log/rodslog... file: Apr 27 13:19:29 pid:20623 NOTICE: environment variable set, irodsEnvFile=/var/opt/irods/ccdbdevirods/env
Apr 27 13:19:29 pid:20623 NOTICE: irodsUserName=ccdbdevirods
Apr 27 13:19:29 pid:20623 NOTICE: irodsPort=5630
Apr 27 13:19:29 pid:20623 NOTICE: irodsZone=telescience
Apr 27 13:19:29 pid:20623 NOTICE: irodsHost=armstrong.crbs.ucsd.edu
Apr 27 13:19:29 pid:20623 NOTICE: environment variable set, irodsPort=5630
Apr 27 13:19:29 pid:20623 NOTICE: environment variable set, irodsAuthFileName=/var/opt/irods/ccdbdevirods/auth
Apr 27 13:19:29 pid:20623 NOTICE: created irodsHome=/telescience/home/ccdbdevirods
Apr 27 13:19:29 pid:20623 NOTICE: created irodsCwd=/telescience/home/ccdbdevirods
LocalHostName: fugazi, fugazi.crbs.ucsd.edu, localhost, Port Num: 5630.
Apr 27 13:20:29 pid:20623 ERROR: cllConnect: OCILogon failed: -1
Apr 27 13:20:29 pid:20623 ERROR: OCI_Error: ORA-12170: TNS:Connect timeout occurred
Apr 27 13:20:29 pid:20623 NOTICE: chlOpen cmlOpen failure -803000
Apr 27 13:20:29 pid:20623 NOTICE: connectRcat: chlOpen Error. Status = -803000
Apr 27 13:20:29 pid:20623 SYSTEM FATAL: initServerInfo: no rcatHost error, status = -803000
Apr 27 13:20:29 pid:20623 NOTICE: initServer: initServerInfo error, status = -803000
Apr 27 13:20:29 pid:20623 ERROR: initServerMain: initServer error. status = -803000 |
Try this: sqlplus ccdbdevicat@orcl1 |
or this h5. Quick Database Connectivity Test
for i in ccdbdevicat ccdbdevmcat ccdbstageicat ccdbstagemcat crbsdevicat crbsdevmcat crbsstageicat crbsstagemcat ; do echo $i ; sqlplus $i@$i ; done
|
If that doesn't work, check the status of the database server, including checking disk space, load, and that the database engine is running. CCDB Portal iRODSI want to Archive data that isn't in the data jail. What should I do?On the poas.crbs.ucsd.edu: - mv the directory that contains the data to the appropriate place in the data jail and rename the directory to include the prefix "CCDBID_<MPID>_"
e.g. mv /export/crbs/ncmirdata5/eric/3view/Christine/ONH_506-L /export/crbs/ncmirdata5/rawdata/data/3view/CCDBID_45397_ONH_506-L
mv /export/crbs/ncmirdata5/eric/3view/Christine/507-L /export/crbs/ncmirdata5/rawdata/data/3view/CCDBID_45385_507-L
mv /export/crbs/ncmirdata5/eric/3view/Christine/507-R /export/crbs/ncmirdata5/rawdata/data/3view/CCDBID_45373_507-R
mv /export/crbs/ncmirdata5/eric/3view/Branfman/Mouse81_cortex /export/crbs/ncmirdata5/rawdata/data/3view/CCDBID_43091_Mouse81_cortex |
- recursively change the ownership of the data
The Archive process initiated from the CCDB Portal was interrupted. What should I do?- FASTER OPTION - Use if no data made it to the final archive destination
ALTERNATIVE - SAFER OPTION - Use if some, but not all data made it to the final archive destination
md5sum each file in the data jail and compare to md5sum for corresponding file in final archive destination cd /ccdbprod/ccdbproddj0/<scope> ; for i in `find <MPID_directory> -type f` ; do md5sum $i ; md5sum /ccdbprod/ccdbprod1/home/CCDB_DATA_USER.portal/CCDB_DATA_USER/acquisition/project_####/$i/done |
- If the md5sums do not match, correct it so that the correct file is in both places.
Do an iscan on the archive directory to see if any files were registered with iRODS, but which never made it to the right place on the disk. If files show up as "orphaned" (meaning it's on the disk, but not cataloged in iRODs), they need to be "sync"d via the CCDB Portal. |
[ccdbprodirods@ccdb-irods rawdata]$ iscan -r /ccdbprod/ccdbprod1/home/CCDB_DATA_USER.portal/CCDB_DATA_USER/acquisition/project_20099/microscopy_45249/rawdata/CCDBID_45249
/ccdbprod/ccdbprod1/home/CCDB_DATA_USER.portal/CCDB_DATA_USER/acquisition/project_20099/microscopy_45249/rawdata/CCDBID_45249/CCDB_MetaData/sutterlinGrid2Control_instrumentMetaData.txt tagged as orphan file |
An ils of an orphaned file will says it doesn't exist in iRODS. |
[ccdbprodirods@ccdb-irods rawdata]$ ils -L /telescience/home/CCDB_DATA_USER.portal/CCDB_DATA_USER/acquisition/project_20099/microscopy_45249/rawdata/CCDBID_45249/CCDB_MetaData/sutterlinGrid2Control_instrumentMetaData.txt
ERROR: lsUtil: srcPath /telescience/home/CCDB_DATA_USER.portal/CCDB_DATA_USER/acquisition/project_20099/microscopy_45249/rawdata/CCDBID_45249/CCDB_MetaData/sutterlinGrid2Control_instrumentMetaData.txt does not exist or user lacks access permission |
Copy any missing files from the data jail to the final archive location Be careful not to overwrite data you don't want to be overwritten - copy only the missing files, or try using 'cp -i ...' |
[ccdbprodirods@ccdb-irods rawdata]$ cp -r -i /ccdbprod/ccdbproddj0/3view/CCDBID_45249/<some_dir>/<some_file> /ccdbprod/ccdbprod1/home/CCDB_DATA_USER.portal/CCDB_DATA_USER/acquisition/project_20099/microscopy_45249/rawdata/CCDBID_45249/<some_dir>/<some_file>
|
"sync" the archive directory via the CCDB Portal. The portal will send email telling you the data has been synced. From: ccdbuser@gmail.com
To: vrowley@ncmir.ucsd.edu
Message-ID: <1579866481.17.1302902712555.JavaMail.tomcat@melbourne.crbs.ucsd.edu>
Subject: NCMIR data registration - synchronization
MIME-Version: 1.0
Content-Type: text/html; charset=us-ascii
Content-Transfer-Encoding: 7bit
Your data, /telescience/home/CCDB_DATA_USER.portal/CCDB_DATA_USER/acquisition/project_20099/microscopy_45249/rawdata/CCDBID_45249/CCDB_MetaData has been successfully synchronized with the NCMIR storage.
/////////////////////Debugging//////////////////////////////
.
250 2.0.0 OK 1302902715 i2sm654494pbb.71
QUIT
221 2.0.0 closing connection i2sm654494pbb.71
Finished Sending-------------------------------------- |
How do I know if the service started properly?ON AN ICAT you'll see both an irodsServer ANDand irodsReServer process running. The output of `ps ax | grep irod` should look like this: [root@fugazi ~]# ps ax | grep irod
20609 ? Ss 0:00 ./irodsServer PROJECT=ccdbdevirods
20616 ? S 0:00 irodsReServer
22514 pts/3 R+ 0:00 grep irod
[root@fugazi ~]#
|
ON A SLAVE you'll see onlyan irodsServer process running. The output of `ps ax | grep irod` should look like this: [root@fugazi ~]# ps ax | grep irod
20609 ? Ss 0:00 ./irodsServer PROJECT=ccdbdevirods
22514 pts/3 R+ 0:00 grep irod
[root@fugazi ~]#
|
REGRESSION TESTINGDEV TEST service ccdbdevirods stop ; /opt/irods/server/bin/irodsctl ccdbdevirods start ; tail -f /var/opt/irods/ccdbdevirods/log/rodsLog.2010.6.16 ; ps ax | grep irod |
OUTPUT Jun 20 19:52:54 pid:18609 NOTICE: environment variable set, irodsPort=5630
Jun 20 19:52:54 pid:18609 NOTICE: environment variable set, irodsAuthFileName=/var/opt/irods/ccdbdevirods/auth
Jun 20 19:52:54 pid:18609 NOTICE: created irodsHome=/telescience/home/ccdbdevirods
Jun 20 19:52:54 pid:18609 NOTICE: created irodsCwd=/telescience/home/ccdbdevirods
LocalHostName: fugazi, fugazi.crbs.ucsd.edu, localhost, Port Num: 5630.
Jun 20 19:52:54 pid:18609 ERROR: initZone: zoneName in env telescience does not match CCDBDevZone in icat
Local Resource configuration:
RescName: myresc, VaultPath: /data1/testVault
RescName: SRB-mcat-to-icat-dev, VaultPath: /data1/SRBVault/mcat-to-icat-Vault
RescName: demoResc, VaultPath: /home/ccdbdevirods/iRODS/Vault
RescName: SRB-mcat-to-icat-dev-nas0, VaultPath: /data1/SRBVault/mcat-to-icat-Vault
RescName: ucsd-ccdb-dev-nas0, VaultPath: /nas/nas0/ccdbdevirods/iRODS/Vault
Zone Info:
ZoneName: telescience Type: LOCAL_ICAT HostAddr: fugazi PortNum: 5630
reHost: fugazi
Jun 20 19:52:54 pid:18609 NOTICE: rodsServer Release version rods2.3 - API Version d is up
Jun 20 19:52:54 pid:18609 NOTICE: Starting irodsReServer
|
STAGE TEST service ccdbstageirods stop ; /opt/irods/server/bin/irodsctl ccdbstageirods start ; tail -f /var/opt/irods/ccdbstageirods/log/rodsLog.2010.6.16 ; ps ax | grep irod |
OUTPUT Jun 20 19:48:56 pid:18497 NOTICE: environment variable set, irodsPort=5620
Jun 20 19:48:56 pid:18497 NOTICE: environment variable set, irodsAuthFileName=/var/opt/irods/ccdbstageirods/auth
Jun 20 19:48:56 pid:18497 NOTICE: created irodsHome=/telescience/home/ccdbstageirods
Jun 20 19:48:56 pid:18497 NOTICE: created irodsCwd=/telescience/home/ccdbstageirods
LocalHostName: crockett, crockett.crbs.ucsd.edu, localhost, Port Num: 5620.
Local Resource configuration:
RescName: SRB-z-ucsd-ccdb-nas1, VaultPath: /data1/telescience
RescName: SRB-z-ucsd-ccdb-nas2, VaultPath: /data1/tele_home
RescName: demoResc, VaultPath: /home/ccdbstageirods/iRODS/Vault
RescName: SRB-z-ucsd-ccdb-nas0, VaultPath: /data1/SRBVault
RescName: SRB-z-mcat-to-icat-nas0, VaultPath: /data1/mcat_icat
RescName: SRB-testRes, VaultPath: /data1/SRBTestVault
RescName: SRB-z-ucsd-ccdb-nas3, VaultPath: /data1/tele_old
Zone Info:
ZoneName: telescience Type: LOCAL_ICAT HostAddr: crockett PortNum: 5620
reHost: crockett
Jun 20 19:48:56 pid:18497 NOTICE: rodsServer Release version rods2.3 - API Version d is up
Jun 20 19:48:56 pid:18497 NOTICE: Starting irodsReServer
|
PROD TEST service ccdbprodirods stop ; /opt/irods/server/bin/irodsctl ccdbprodirods start ; tail -f /var/opt/irods/ccdbprodirods/log/rodsLog.2010.6.16 ; ps ax | grep irod |
OUTPUT Jun 20 19:51:07 pid:27192 NOTICE: irodsUserName=ccdbprodirods
Jun 20 19:51:07 pid:27192 NOTICE: irodsPort=5610
Jun 20 19:51:07 pid:27192 NOTICE: irodsZone=CCDBProdZone
Jun 20 19:51:07 pid:27192 NOTICE: irodsHost=localhost
Jun 20 19:51:07 pid:27192 NOTICE: environment variable set, irodsPort=5610
Jun 20 19:51:07 pid:27192 NOTICE: environment variable set, irodsAuthFileName=/var/opt/irods/ccdbprodirods/auth
Jun 20 19:51:07 pid:27192 NOTICE: created irodsHome=/CCDBProdZone/home/ccdbprodirods
Jun 20 19:51:07 pid:27192 NOTICE: created irodsCwd=/CCDBProdZone/home/ccdbprodirods
LocalHostName: lakota, lakota.crbs.ucsd.edu, localhost, Port Num: 5610.
Local Resource configuration:
No Local Resource Configured
Zone Info:
ZoneName: CCDBProdZone Type: LOCAL_ICAT HostAddr: lakota PortNum: 5610
reHost: lakota
Jun 20 19:51:07 pid:27192 NOTICE: rodsServer Release version rods2.3 - API Version d is up
Jun 20 19:51:07 pid:27192 NOTICE: Starting irodsReServer
|
- fugazi, crocket and lakota are all icats
- irods logs are in /var/opt/irods/ccdb<env>irods/log/*
- the pid is in /var/run/ccdb<env>irods.pid
- the exectuables are in /opt/irods/
- the configuration files are in /etc/irods/ccdb<env>irods
|