SPRINT 4 - Finish 7/31/2015

Owned by bcarlsen (Unlicensed)
Last updated: Aug 05, 2015
5 min read

Overview

Sprint to extend some of the "read only" APIs, to finalize unit/integration testing for existing functionality, to extend the interface, to develop additional loaders, and to begin developing editing capabilities.

Priority things

  • Model: GCI
  • DONE: Add precedence list to metadata service rest
    • Add a "getDefaultPrecedenceList(terminology,version)" method to MetadataServiceRest (and implement all connected stuff)
    • Add integration test for this method as well (MetadataServiceRestNormalUse, DegenerateUse, EdgeCases,...)
    • On application side
      • Load the precedence list after "get all metadata" is called and set a scope variable equal to it.
      • In the "Metadata" tab, show the precedence list in its own section (separate from the other metadata - because it is an ordered list.)
      • Put in its own bullet before the "general metadata entries" stuff.
  • Implement NewConceptMinRequirementsCheck and "validation service"
    • DONE: Validation check
      • validate(Concept), validate(Atom), validate(Code), validate(Descriptor), validateMerge(Concept,Concept)
    • Validation service - See ValidationService.java
      • public ValidationResult validateConcept 
      • public ValidationResult validateAtom 
      • public ValidationResult validateDescriptor ...
      • public ValidationResult validateCode  ...
      • public ValidationResult validateMerge(Strng cui1, String cui2, String branch)
    • Implement ValidationService (e.g. ValidationServiceJpa)
      • pre-cache validation handlers (see how this works in MetadataService, etc).
        • static List<ValidationCheck> validationHandlers  = ...
      • Implement the interface methods
        • Call the corresponding validation check method
        • accumulate validation results
        • return them
    • ValidationServiceRest/RestImpl/ClientRest
      • validateConcept  /validate/cui/ (PUT) ConceptJpa
        • load all validation handlers and call each validate Concept) method.
      • validateAtom  /validate/aui (PUT) AtomJpa
      • validateDescriptor ... /validate/dui (PUT)DescriptorJpa
      • validateCode  .../validate/code (PUT)DescriptorJpa
      • validateMerge /validate/cui/merge/{cui1}/{cui2} - GET
    • Integration tests - create infrastructure and implement for "default validation check"
      • Assume that validation always passes.
      • normal use
        • validate a concept
        • validate an atom
        • validate a descriptor
        • validate a code
        • validate a merge between two concepts
      • degenerate use
      • edge cases
    • For individual tests, we need unit tests
      • implement a unit test for DefaultValidationCheck
      • in server-jpa-services
      • com.wci.umls.server.test.validation...
    • DONE: DefaultValidationCheck
      • validate atom whitespace
      • validate that a concept has at least one atom and at least one "hierarchical" relationship (to something that exists)
      • update config.properties to use this check
    • DONE: Organize validation checks into packages and create classes
      • com.wci.umls.server.jpa.services.validation (DefaultValidationCheck)
      • com.wci.umls.server.jpa.services.validation.umls (NewConceptMinRequirementsCheck)
        • concept has at least one atom and at least one hierarchical rel 
      • com.wci.umls.server.jpa.services.validation.snomed (NewConceptMinRequirementsCheck)
        • see term server project
  • DONE: Support roman-numeral sort order for ICD10
  • DONE: Test solutions for exact matches showing up at the top for auto complete
    • the edge ngram boost should already be working for words starting with those things
  • DONE: Need improvements to filtering (e.g. misaligned quotes, extra punctuation, lots of things don't quite work right)
    • test for atoms - should work cleanly
    • test for relationships - consider escaping the query and handling the "*" at the end differently
  • DONE: Autocomplete suggestions should always be valid 
    • i.e. the value when searched should always produce a result
    • one idea: remove lucene special characters when setting the value
    • another idea is to use QueryParser to escape the query
    • Make sure text is urlencoded
    • Handled by making query parser escape if there's a parse exception
  • DONE: Finalize REST service for a general concept query (e.g. lucene query, HQL query, or SQL query)
    • DONE: Must start with "SELECT " and not contain ";"
      • provide appropriate error message via exception if not
    • DONE: Execute the query but check the object type of the first result
      • if doesn't match expected type, fail with an exception.
    • query.setHint("javax.persistence.query.timeout", timeout);

      Appears to be not supported in MySQL 5.6.  should be in 5.7.4

Marketing/SEO

  • Create a video demo of the site (camtasia) and post as a link on the header (video glyphicon if there is one)
  • Promote "ICD10" browser on various lists
  • Ensure all entry pages for applications contain SEO text for browsers and are indexed by google
  • Verify all entry pages for applications are officially mobile friendly
  • Consider advertising SNOMED/UMLS/ICD browsers on google adwords.
  • FB?

User Interface Enhancements

  • Upon logging in or changing terminology, focus on the "query" field
  • Add features for "deep" relationships when browsing UMLS.    
    • Need a generalized way to know when to use this
    • it is definitely only for "concept"
    • It may be that if any "atoms" of the concept don't match the terminology, then we show it.
  • Websocket (for a WebsocketWorkflowListener)
  • Advanced search (uses "search criteria" or "general query" mechanisms)
    • Create "stock" query examples that can be run by clicking a button to demonstrate how it works.
      • UMLS concepts containing both SNOMED and LOINC atoms
      • SNOMED US Extension concepts added in 20140901
      • Can generalize this by using the "general_metadata_entries" mechanism.
  • Consider adding "LABELFOR" all subsets and making the star pop up a picklist of the things to highlight (ordered by type with extensions first, subsets later)
  • RECURRING: Mobile-friendly and other style issues

Additional/Enhanced  Loaders

Services

  • Action Service
    • Implement classification.
    • Need to go to/from Owl so do Owl loader FIRST.

 

Testing 

  • ClaML Testing

    • Look at a concept with an X in the code
      Look at the child of a concept with an X code
      Make sure modifier classes don't have their own codes
      test each concept has a "self" row in transitive closure

  • Finish implementing content service test suites
  • Validate all mojo integration tests
    • RRF-umls
    • RRF-single
    • RF2-snapshot
    • ClaML
  • Implement additional unit tests for model objects (PrecedenceListJpa, LabelSet, etc)
  • RRF loader -> create marker set for SNOMED (both "single" and "umls")
  • Handler002Test for normal use
  • Implement Handler003/008Test - for ID assignment algorithms.  Borrow code from other project (though there may be differences).  The uuidHash algorithm is implemented properly for UMLS and may be different than for SNOMED.

Editing Features

  • Project
    • Figure out how to capture "project scope" for SNOMED and for UMLS in a generalized way.  Update project objects to be able to properly capture (and compute) project scope.  NOTE: the scope definition may involve concepts/terminologies/semantic types.  IN that event, the scope computer gets a little bit more complicated.
  • Test loading a DB with envers auditing disabled and then making changes in a DB while it is enabled. Does it properly create audit entries?
    • for the old edition of the component?
    • for the new edition?
  • Metathesaurus editing actions
    • MetathesaurusContentEditingRest
      • methods for each "edit action"
      • Create a RestImpl
      • Create a client
      • Create integration tests to run against a "stock" dev  database
    • Add a semantic type component, Remove a semantic type component
      • Have a pencil icon by  the STYs section
      • clicking gives you a list of available STYs assigned, in tree order with a filter box for typing characters of the STY you want.
        • See the metadata "semantic type" option
      • User may want to choose multiple ones (so have a "finished" button)
      • Dont allow user to choose STYs already assigned to the concept.
      • Final action is to call "contentService.addSemanticTypeComponent"
      • Consider what happens to workflow status
      • Consider how to show "NEEDS_REVIEW" content in the browser
      • Consider how to support "Undo". - perhaps an action log (atomic/molecular) is a good idea still for that
    • Implement this completely including testing before moving on to other actions (each which requires a UI enhancement)
      • Approve a concept (e.g. set workflow status values).
      • Add an atom (e.g. MTH/PN - that needs to be generalized somehow)
      • Merge two concepts (consider the "workflow status "when this happens).
      • Move an atom (or atoms) from one concept to another
      • Split an atom (or atoms) out of a concept and specify a relationship type between the two concept
  • Terminology Editing (first use case)
    • Add a concept (as a child of an existing concept) with one or more atoms and a PAR/CHD relationship.
    • Run the classifier
    • Show classifier results (e.g. new inferred relationships, etc)
    • NOTE: this only works with a description logic based source that tracks inferred relationships.
    • PREREQ: SNOMEDCT RF2 loader.

Admin Tools

  • Test QA queries and flesh them out for 100% coverage.

Optimizations

  • TBD

 

Future Stuff

  • Test conditional envers auditing: http://stackoverflow.com/questions/14250612/conditional-envers-auditing
  • escg (expression grammar - research)
  • Use Lucene SynonymFilter with synonym table
  • Component-Component relationships (between any two components).
  • Value set definitions (and marking of subset.isValueSet()) and linking to definition? via attribute?
  • Owl loader, Owl export of DL terminologies (e.g. RF2-loaded SNOMED)
  • Rdf export (flat)
  • Classifier (owl interface)
  • Expression language (based on SNOMED expression constraint grammar)
  • Sub-branching
    • branchResolutionHandler - figures out how to copy and mark branched objects  and update indexes  - for different branching strategies.
  • Handle mappings - may be not worth it
  • Implement an RF2 loader (use DL features)
  • Implement a ClaML loader
  • Support semantic network (e.g. sty rels, etc).  - probably want to wait for a real ontology - and maybe even load it as such.