SPRINT 8 - Finish 20160430

Overview

Sprint to extend some of the "read only" APIs, to finalize unit/integration testing for existing functionality, to extend the interface, to develop additional loaders, and to begin developing editing capabilities.

Data Reloads/Code Deployments

• Rebuild data (on backend machine), rebuild and redeploy frontend code
  • DONE: UMLS
  • DONE: VET
  • BAC: SNOMED
  • ICD
  • OWL
• Fix some bugs before rebuilding code (because they don't affect db).Atte

Strategic Directions

• Marketing/SEO - Deploying a demo site with description, screenshots, and video
• Core application enhancement
• Editing features

Marketing/SEO

• PG: Landing and License page
  • User logs in, sees info, screenshots and videos
  • DONE "launch browser" button exists
  • DONE upon clicking, it user is presented with a license agreement (if the cookie for it doesn't already exist)
  • DONE Upon accepting license, a cookie is created that expires 30 days.
  • DONE From accepting license, user enters /content path

  • DONE Handle landing/license/login set up with a new .js file that gets filtered

    appConfig.js
    tsApp
      .value(
        'appConfig',
        { landing : ${deploy.landing.flag},
          license : ${deploy.license.flag},
          ...
    
    

• New front-page website with description, screenshots, video
• "Login" link (with logout link on footer)
• Have the "upload" and "source" tabs from transformer here
  • "Snomed starter kit"
• Create a video demo of the site (camtasia) and post as a link on the header (video glyphicon if there is one)
  • also screenshots that are clickable, etc.
• Verify all entry pages for applications are officially mobile friendly
• Update campaigns for  SNOMED/UMLS/ICD browsers on google adwords.
• Training Video for UMLS browser (need 10 min)
• Training Video for SNOMED browser (simple 5 min)
• Find the SNOMED video that Anil sent, redo with RF2 - same content!
• Other notes (from email)
  • Google adwords campaign
    
    • “free icd-10” browser, “free SNOMED CT US Extension browser”, “free UMLS browser
  • Get webmaster setup for the icd/umls/snomed pages
    
    • Use config setup for that
  • Find east bay healthcare startups
    
    • Pursue like job search (to know they’re hiring, etc)
    • Tech
    • Device
    • Information/informatics 
  • Meetups

Starter Kit - PG

• ValidatoinResult precheckLoadFromSourceData
• Application hardening
  • Error handling during configuration (bad db, username, password)
  • directory already exists?
  • DONE failed load
    • sourceData marked as FAILED -> only support certain actions
  • DONE cancelled load -> allow reload of delta
  • source data missing, some terminology data loaded
• spelling.txt, acronyms.txt
  • package these files into the war. (so they are in WEB-INF/classes)
  • getResourceAsStream("spelling.txt") -> source.data.dir/spelling.txt
• Terminology Starter Kit - ...
  • Run the .war file via Jetty all packaged as an exectuable .jar
  • Single zip file download containing
    • termserver.jar
    • data/ (spelling.txt, acronyms.txt)
    • indexes/ (lucene indexes)
    • hsqldb/ (database files) 
  • config.properties
    • Uses "hsqldb" jdbc (with "file" not "mem" mode)
      • http://stackoverflow.com/questions/3804409/using-hsql-in-memory-database-as-jpa-datasource
    • directories can all be relative (which should work relative to the executable .jar file).
  • StarterKitApplication (e.g. the jersey application)
    • initialization hook to start the hsqldb server - http://stackoverflow.com/questions/3660502/how-can-i-start-and-keep-running-hsqldb-in-server-mode-from-within-my-web-applic
  • Maven process
    • ssk-rest (module)
      • Build .war file
      • .war file must contain a completely ready config.properties file (e.g. in WEB-INF/classes) - put in src/main/resources 
        • no filtering required
        • exact hsqldb jdbc url, user, password
        • indexBase=./indexes
        • spellingFile=./data/spelling.txt
        • security.guest.disabled=false
        • mail server??
        • landing/ login/ intro page configuration
    • ssk-app/ (module)
    • pom.xml
      • Load database  (Rrf2SnapshotSourceDataLoadMojo)
        • specify indexes directory        (${project.build.dir}/app/indexes
        • specify hsqldb directory          (${project.build.dir}/app/db)
        • specify snomed (or input dir)  (-Dinput.dir)
      • Gather all resources into ${project.build.dir}/app
        • packages .war file (from ssk-rest) as an executable .jar that runs jetty, snomed-starter-kit.jar
        • package spelling.txt and acronyms.txt files in ${project.build.dir}/app/data
      • Use assembly plugin to zip everything from ${project.build.dir}/app/* into snomed-starter-kit.zip
• DONE: Get initial starter kit running

Core Application Enhancement - P1

• BAC: fix hibernate properties in config.props (on deployments and in project)

• Validate integration tests again

• PG: Small finishing items

  • DONE Inferred/Stated button not working properly for SNOMED (e.g. "description logic terminology")
  • Tooltips for some things not working in SNOMED (e.g. rel, attribute nmae, etc) (cannot reproduce)

  • DONE siblings/children in a hierarchy should have a default sort order (e.g. alphabetical).  Try looking up "botulism" in snomed.terminology.tools, it looks random.

  • DONE Advanced searching icon having issues - can't click it because of uib-tooltip

  • DONE Highlighting in tree search view not working

  • DONE Switching from non-metathesaurus to metathesaurus improperly shows deep relationships

  • DONE/Duplicate limit deep relationship display to terminology matching current metathesaurus

  • DONE Dummy tree roots should not be clickable
  • DONE Retrieving siblings should not duplicate entries for matching components
  • DONE Tree elements matching search should appear at the top of lists (not sorted)
  • DONE uib-tooltip for advanced search icon should not block clicking the icon

  • Could not retrieve "back" button not working

  • DONE atom relationship link text does not initiate new search

• DONE Once "appConfig.js" exists ,we can stop filtering  footer/header, etc and instead inject the appConfig object into those controllers and just make scope variables for what shows up.

• DONE (BAC): Bring "guest "user stuff over from refset

  • see services.js and also SecurityServiceJpa.

• Implement expression searching based on ECL

  • Replace the "search criteria" mechanism with this.
  • Identify as a search by parsing and confirming it matches.  
  • Build SQL or lucene queries from ECL.
  • See Kai's project on github: https://github.com/IHTSDO/snomed-query-service
  • Also show expressions (like diagrams) for a concept's de

• Compute preferred name enhancements

  • support all modes, including multiple meta
  • tie to user preferences.

• DONE Put all route provider back in routes.js (PG)

• Review overhaul of loader architecture

  • Algorithm -> LoaderAlgorithm
  • SourceDataLoaderAlgorithm (interface)
• Employ a consistent handleLazyIniti strategy
  • enforce this at release time (add to wiki)
  • public handleLazyInit(...)
  • ContentService doesn't need it because of "graph resolver" which is entirely responsible for content objects.
  • applies to "source data" and "project" and "metadata" services.
• Normalize use of sortable tables across the application (search in content, admin service tables) - use ng-table
• Enabling "cancel"/progress of loaders
  • RrfLoaderAlgorithm (can implement at abstract loader algorithm level)
    • cancel() should set a cancelFlag to true
    • commit/log statements should check the cancel flag and throw a CancelException
  • Add CancelException
  • Change content service
    • Add "precondition" checking to loaders (Algorithm)
    • Put all loader logic into "compute"  -> e.g. "open readers", "sort files", etc.
  • Source Data Loader
    • have try/catch
    • support "background" parameter
    • add cancelLoad(...)
    • load -> processId
    • getRunningProcesses(...)
  • Put a ticket on wiki to rewire mojos from content service to sourcedataLoaderService (then remove from content service the load methods
  • BUG Terminology Removerdoes not correctly handle mapsets 

    • Cannot delete or update a parent row: a foreign key constraint fails (`umlsminidb`.`mapsets_attributes`, CONSTRAINT `FK_a1tfp35h17fsbdl07p9xeex2h` FOREIGN KEY (`attributes_id`) REFERENCES `attributes` (`id`))

• Find all <i> within <a> and remove the <a> - e.g. atoms directive
• DSS: Bring over logging stuff from refset
  • objects - LogEntry, LogEntryJpa
    • Add terminology, version, 
    • activity -> LogActivity.LOADER, RELEASE, EDITING
  • RootServiceJpa methods
  • RootServiceRestImpl methods
  • getLog REST API call from ProjectService
    • keep existing method
    • add one with: instead of "projectId": and "objectId" take "terminology" , "version", and "activity"
  • think about what should add log entries (e.g. loaders, terminology removers) 
    • idea; anything that changes the db.
    • addLogEntry statements ONLY go in RestImpl layer. (and maybe the algorithms)
  • Put logging statements for loaders - e.g. Rf2SnapshotLoaderAlgorithm
    • Add three local methods
      • commitClearBegin(...., terminology, version, activity) 
        • super.commitClearBegin(...)

        • if (objectCt % logCt == 0) {
           addLogEntry...
          }

      • logAndCommit(...., terminology, version, activity)
        • super.logAndCommit(...)

        • if (objectCt % logCt == 0) {
           addLogEntry...
          }

      • logInfo(message , terminology, version, activity) - also for warn/error, but add the word "WARNING: " or "ERROR: " to the message
        • Change all Logger.getLogger(...).info(...)  to logInfo(...) calls
        • addLogEnry(...) = use "loader" as username
        • also Logger.getLogger(...).info(... )
• DONE PG: redo header/footer - controllers. (like refset tool)
• PG:  Implement Diagramming
  
  • model transformation
  • Show ONLY for "description logic terminology"
• DONE PG  Add features for "deep" relationships when browsing UMLS.    
  
  • it is definitely only for "concept" and for "metathesaurus" content
  • Add new section to concept report (e.g. "Deep relationships")
  • REMAINING: Update integration tests with actual filter values (currently only tests null values)
• Enhancements to RRF preferred name computer
  • support multiple UMLS's - e.g. have a high-level terminology/version
  • reuse the same default precedence list.
• DONE PG: Use directives for reports
  • DONE Component Report
    • treeComponent
    • semanticTypes
    • definitions
    • atoms
    • attributes
    • subsets
    • relationships
    • deep relationships
      • TODO Relationships and deep relationships are not using callback functions, fix this
  • DONE Search Result Tree directive
• DONE PG Fixed Infinite cycle digest errors on uib-tooltip-html for refset labels in tree search results (extension highlighting)
• DSS; Project/User/UserPrefs stuff
  • User
    • Application role
    • user preferences
    • projectRoleMap (e.g. and project role map adapter)
  • Project
    • Remove "leads", "authors", "administrators"
    • remove "scope" stuff
    • Remove actionWorkflowStatusValues
    • userRoleMap (and userRoleMapAdapter)
  • Bring over UserRole from Refset -> replace term server stuff
  • Align Security Service Rest (and Jpa)
    • Bring over methods from refset that are missing
    • Reconcile differences in the implementation methods.
  • ProjectService Rest (and Jpa)
    • Remove the scope concept calculation
    • Add stuff about user/project (assign,unassign,find...)
  • next step: add an "Admin" tab.
    • basically borrow from refset and make work for this project.
• DONE PG Use route params to dynamically load a "report" (and be able to bookmark URLs)
  
  • DONE Implement routing for terminologyId/terminology/version/type so we can preserve URLs.  can even include ?query=... for the query
    • CANCELLED Also clean up the way routes work first so we have
      • https://umls.terminology.tools/content
      • instead of
        • https://umls.terminology.tools/#/content
      • This is likely related to the starting URL redirected to https://umls.terminology.tools/#/
      • See http://stackoverflow.com/questions/14319967/angularjs-routing-without-the-hash
      • e.g. https://umls.terminology.tools/content/CUI/SNOMEDCT/2015_01_31/12738006
• DONE Support opening a concept in a new window (e.g. there's a pointing off arrow icon that opens a new window with a routing URL that  shows exactly that concept - then drag/drop between windows can be editing mechanism).
  • https://umls.terminology.tools/content/simple/cui/SNOMEDCT/2015_01_31/12738006
  • Improvement: Fix header offset in simple report
• User Preferences stuff
  • Bring over model from Refset tool.
  • Add "last query" (e.g. "brain" and whether it's "list" or "tree" mode).
  • Add "last report" (e.g. type/ui/terminology/version)
• Remove "void addXXX" and "void removeXXX" methods from model objects. Use getObjects().add/remove(...) only
  • e.g. AtomClass doesn't need addAtom or removeAtom
  • Rewire any uses of them  
  • "addXXX" -> "getXXX().add"
  • "removeXXX" -> "getXXX().remove"
  • same if there is "clear"
• DONE Show concept and atom-level subset information (e.g. language refset entires in SNOMED. may require a change to the graph resolver).
  • DefaultGraphResolver now resolves subset members for both Atom and Concept
• DONE Migration of upload and loading and removal of sources from transformer to term server
  • Separate tabCtl, tabService
  • Get all workgin,
  • Then configure/use in the Starter kit
• TODO: Restore functionality of SourceDataRemoverMojo -> should call a service.
• TODO: precondition checking for SoruceData load 
  • (e.g. verify terminology/version doesn't already exist) - e.g. for Rf2Snap, but not Rf2Delta, there it's vice versa.
  • Each loader probably  needs its own rules - revisit all of them.

Core Application Enhancement - P2

•  Ronald Cornet ideas for starter kit: 
  

  • Add some capability to annotate concepts (and save those annotations, and then recover annotated concepts and export them for later use - e.g. SIRS integration).  
    Also could support some basic refset maintenance (extensional, plus

  • "favorites list"
• User interface for mappings content
  • Have a separate tab for this
  • search by concept id
  • restrict to map set
  • search by target id
  • Show full mappings in a table
  • Need methods for searching mappings.
• Review all the static config info from the config file
  
  • determine which have a single option and which have multiple options
  • determine which can have a single implementation or need to be copied (e.g. search handler)
  • determine how API calls (at JPA) layer work, and how they work at REST layer - make the same.
  • etc
• Refactor loader mojos and service calls to all work the same way and use source data loader.
• Refactor Content Service (Rest)
  • Separate out concept functions so that they can be deployed without CODE and DUI  services
• Release criteria integration tests
  • Verify no *java classes have System.out.println
  • Verify no *java classes have TODO
  • ...
• Implement precedence list saving in updateUserPreferences - currently it just sets user precedence list to null
• Implement glass pane for an individual component so that say "deep relationships" can load in its own time, but other elements can be interactive and ready to go when ready.
• DONE PG: Support RF2 "atom association refset members" as atom relationships.
• Support RF2 association refset members for non-concept members  (e.g. descriptions)
• Enable glass pane while switching tabs (between content/metadata)
  • UNNECESSARY, as any calls will glass pane immediately, and if no calls, rendering is immediate
• DONE PG Bring "source data file" and "source data" over from transformer application, along with other things.
  • DONE Put the objects into the package structure of terminology server (e.g. put with Project and User)
  • Need to update transformer too so it continues to work.
  • Leave mojos behind.
  • Uploading issues
    • Uploaded file count is restricted to 10 for some reason
    • Subsequent uploads do not add to the queue
    • NOTE: May want to switch to ng-file-upload or flow.js, as they allow folder uploads (angular-file-upload does not)
      • In meantime, added flatFileStructure flag to RF2 loader 
  • Loading notes:
    • Added flat file structure setter to RF2 loader to determine whether a webapp load from flat files or normal folder structure load
      • TODO Need to do this for other loaders as well
      • Must also delete the sorted temp folder for loaders if it exists (otherwise false flag on directory structure)
    •  
       
• Advanced search
  • LATER: Expression. - e..g. "Search Criteria" - for "descriptionLogicTerminology"  only
    • descendants of
    • has relationship -> xx
• Show "atom" subset information in the report.  Only concept (or component) subset (or refset) information is being shown.  this may involve a change in the graph resolver to return the data.  i.e. for Snomed you should be able to tell what is just british.

Editing Features

• Basic metathesaurus editing
  • Add/remove STY
  • Add/remove atom
  • Move atom
  • Split atoms from concept
  • Merge concepts
  • Add/remove concept relationship
• Publication Process
  • RRF
• Project
  • Figure out how to capture "project scope" for SNOMED and for UMLS in a generalized way.  Update project objects to be able to properly capture (and compute) project scope.  NOTE: the scope definition may involve concepts/terminologies/semantic types.  IN that event, the scope computer gets a little bit more complicated.
• Test loading a DB with envers auditing disabled and then making changes in a DB while it is enabled. Does it properly create audit entries?
  • for the old edition of the component?
  • for the new edition?
• Metathesaurus editing actions
  • MetathesaurusContentEditingRest
    • methods for each "edit action"
    • Create a RestImpl
    • Create a client
    • Create integration tests to run against a "stock" dev  database
  • Add a semantic type component, Remove a semantic type component
    • Have a pencil icon by  the STYs section
    • clicking gives you a list of available STYs assigned, in tree order with a filter box for typing characters of the STY you want.
      • See the metadata "semantic type" option
    • User may want to choose multiple ones (so have a "finished" button)
    • Dont allow user to choose STYs already assigned to the concept.
    • Final action is to call "contentService.addSemanticTypeComponent"
    • Consider what happens to workflow status
    • Consider how to show "NEEDS_REVIEW" content in the browser
    • Consider how to support "Undo". - perhaps an action log (atomic/molecular) is a good idea still for that
  • Implement this completely including testing before moving on to other actions (each which requires a UI enhancement)
    • Approve a concept (e.g. set workflow status values).
    • Add an atom (e.g. MTH/PN - that needs to be generalized somehow)
    • Merge two concepts (consider the "workflow status "when this happens).
    • Move an atom (or atoms) from one concept to another
    • Split an atom (or atoms) out of a concept and specify a relationship type between the two concept
• Terminology Editing (first use case)
  • Add a concept (as a child of an existing concept) with one or more atoms and a PAR/CHD relationship.
  • Run the classifier
  • Show classifier results (e.g. new inferred relationships, etc)
  • NOTE: this only works with a description logic based source that tracks inferred relationships.
  • PREREQ: SNOMEDCT RF2 loader.

Other Things

• Create initial framework for JS unit testing
• Verify that ClaML loader makes asterisk/dagger attributes for modifier concepts - e.g. M01.4 and M01.41
• Integration tests for static code analysis (e.g. *.code.XXTest.java)
  • REST Service
    • opened sessions are closed in finally blocks.
• For "semantic type" mechanism - if there are tree numbers, order by that and indent"
• Labels – filtering out LABELFOR labels should happen in getPagedArray function in labels.js, and not in the html via ng-show
• OWL - NCIt

User Interface Enhancements

• Do something useful with websocket
• Expression-based searching (get harold's parser?)
• Consider adding "LABELFOR" all subsets and making the star pop up a picklist of the things to highlight (ordered by type with extensions first, subsets later)
• RECURRING: Mobile-friendly and other style issues

Additional/Enhanced  Loaders

• Owl loader - have a snorocket (2) reasoner for role relationships.

Services

• Action Service
  • Implement classification.
  • Need to go to/from Owl so do Owl loader FIRST.

Testing 

• Implement additional unit tests for model objects (PrecedenceListJpa, LabelSet, etc)
• RRF loader -> create label set for SNOMED (both "single" and "umls")
• Handler002Test for normal use
• Implement Handler003/008Test - for ID assignment algorithms.  Borrow code from other project (though there may be differences).  The uuidHash algorithm is implemented properly for UMLS and may be different than for SNOMED.

Admin Tools

• Test QA queries and flesh them out for 100% coverage.

Optimizations

• TBD

Future Stuff

• Test conditional envers auditing: http://stackoverflow.com/questions/14250612/conditional-envers-auditing
• escg (expression grammar - research)
• Use Lucene SynonymFilter with synonym table
• Component-Component relationships (between any two components).
• Value set definitions (and marking of subset.isValueSet()) and linking to definition? via attribute?
• Owl loader, Owl export of DL terminologies (e.g. RF2-loaded SNOMED)

  • http://owlapi.sourceforge.net/owled2011_tutorial.pdf

• Rdf export (flat)
• Classifier (owl interface)
• Expression language (based on SNOMED expression constraint grammar)
• Sub-branching
  • branchResolutionHandler - figures out how to copy and mark branched objects  and update indexes  - for different branching strategies.
• Handle mappings - may be not worth it
• Implement an RF2 loader (use DL features)
• Implement a ClaML loader
• Support semantic network (e.g. sty rels, etc).  - probably want to wait for a real ontology - and maybe even load it as such.