Loading a Database with Full 201610 Data

Overview

Instructions on loading a full database.

Details - from precomputed

1. Create a separate working dir, e.g. c:/ncifull
   1. Create config/config.properties file for this work -e.g. config.properties
      1. set all directories to c:/ncifull instead of c:/umlsserver
   2. Set up a second "server" in Eclipse to run with this config file.
   3. Set up secondary run configurations that use this config.properties file
2. Truncate your db (or prepare a fresh db)
   1. Make sure you are using collation utf8_bin
   2. ALTER DATABASE ncifulldb DEFAULT CHARACTER SET utf8 COLLATE utf_bin
3. Pull and build the project and make sure your config file is completely up to date
4. Clear your indexes directory of all files
5. Download the data from https://wci1.s3.amazonaws.com/NCI/umls.sql.gz
   1. Data including the NCI scale insertion is at: https://wci1.s3.amazonaws.com/NCI/umls-with-nci.sql.gz 
6. gunzip the file (may require cygwin)
   1. Try "gunzip umls.sql.gz"
   2. Due to the file size being so large, gzip can have some checksum issues, so do this:
      1. "cat umls.sql.gz | gunzip -c > umls.sql"
7. Import the file into your database (with MySQL workbench → see the Management/Data Import tool)
8. Reindex your database (run the Reindex profile on the admin/lucene project)
   1. NOTE: make sure your indexed.objects property is blank.

With that, you should be able to build and deploy a server.

NOTE: on certain unix environments MySQL table names are case sensitive and will need to be corrected (ask BAC if needed)

Details - from RRF data (and identity/unpublished data)

1. Create a separate working dir, e.g. c:/ncifull
   1. Create config/config.properties file for this work -e.g. config.properties
      1. set all directories to c:/ncifull instead of c:/umlsserver
   2. Set up a second "server" in Eclipse to run with this config file.
   3. Set up secondary run configurations that use this config.properties file
2. Truncate your db (or prepare a fresh db)
   1. Make sure you are using collation utf8_bin
   2. ALTER DATABASE ncifulldb DEFAULT CHARACTER SET utf8 COLLATE utf_bin
3. Pull and build the project and make sure your config file is completely up to date
4. Clear your indexes directory of all files
5. Download the 201610 data from https://wci1.s3.amazonaws.com/NCI/NCIM_201610.zip 
   1. Unzip into c:/data/NCIM_201610 (may need to use cygwin to unzip due to large file sizes) 
6. Download the identity/unpublished data from https://wci1.s3.amazonaws.com/NCI/NCIM_201610-identity-unpublished.zip
   1. Unzip into c:/data/NCIM_201610 (may need to use cygwin to unzip due to large file sizes)
7. Run the standard "reset-meta" integration test used to load sample data
   1. See Step 7 of Building and Deploying in Eclipse (use "reset-meta" as the profile instead of "reset")
   2. make sure the "input.dir" is pointing to "c:/data/NCIM_201610" 
   3. and config.properties is pointing to c:/ncifull/config/config.properties

Scale Testing Insertions

Full data for insertions can be downloaded from S3

• Download the "full" testing data - https://wci1.s3.amazonaws.com/NCI/NCI-srcDataDir-full.zip
• Unzip this into c:/umlsserver/data
  • This process will use the contents of the "inv" directory
• Edit config-full.properties so that source.data.dir points to c:/umlsserver/data
• Run a server, and perform NCI, SNOMEDCT_US, or MTH insertion.

Scale Testing Release

• Download the "full" testing data - https://wci1.s3.amazonaws.com/NCI/NCI-srcDataDir-full.zip
• Unzip this into c:/umlsserver/data
  • This process will use the contents of the "mr" directory
• Edit config-full.properties so that source.data.dir points to c:/umlsserver/data
• Copy the "bin" directory from config/prod-nci-meta/src/main/resources to your c:/umlsserver directory
• Run a server, and perform NCI, SNOMEDCT_US, or MTH insertion.

References/Links

• Testing on mini scale  - Loading a Database with Sample 201604 Data