SPRINT 7 - Finish 20160131

Owned by bcarlsen (Unlicensed)
Last updated: Mar 21, 2016 by Patrick Granvold (Unlicensed)
9 min read

Overview

Sprint to extend some of the "read only" APIs, to finalize unit/integration testing for existing functionality, to extend the interface, to develop additional loaders, and to begin developing editing capabilities.

Strategic Directions

  • Marketing/SEO - Deploying a demo site with description, screenshots, and video
  • Core application enhancement
  • Editing features

Marketing/SEO

  • DONE Intro page that doesn't have as login
    • DONE Initial page now routes to "landing" view, with landing.html. Initial version includes only placeholder text and link to #/login
    • Note: May want to introduce route bypass if user hits login page while logged in (i.e. content -> landing or landing -> content). Or may not.
  • New front-page website with description, screenshots, video
  • "Login" link (with logout link on header)
  • Have the "upload" and "source" tabs from transformer here
    • "Snomed starter kit"
  • Create a video demo of the site (camtasia) and post as a link on the header (video glyphicon if there is one)
    • also screenshots that are clickable, etc.
  • Verify all entry pages for applications are officially mobile friendly
  • Update campaigns for  SNOMED/UMLS/ICD browsers on google adwords.
  • Training Video for UMLS browser (need 10 min)
  • Training Video for SNOMED browser (simple 5 min)
  • Find the SNOMED video that Anil sent, redo with RF2 - same content!
  • Other notes (from email)
    • Google adwords campaign
      • “free icd-10” browser, “free SNOMED CT US Extension browser”, “free UMLS browser
    • Get webmaster setup for the icd/umls/snomed pages
      • Use config setup for that
    • Find east bay healthcare startups
      • Pursue like job search (to know they’re hiring, etc)
      • Tech
      • Device
      • Information/informatics 
    • Meetups

Core Application Enhancement - P1

  • DONE (BC): Fix long error header - attach icon not showing, where it displays error is not in the error area.
  • DONE PG: Need to index atoms as lowercase string, punctuation replaced by whitespace, no trailing, duplicate, or leading whitespace.
    • Create a new tokenizer factory for this and define in @AnalyzerDefs in AbstractAtomClass
    • "Cholsterol.in HDL/Cholesterol.in LDL" => " cholesterol in hdl cholesterol in ldl"
    • "ABC def----ghi" => "abc def ghi"

    • "ABC     ////df3948123,.3482932–..xx abc" => "abc df3948123 3482932 xx abc"

    • Add an AtomJpa method called getNameNorm() that isn't analyzed but returns a lowercased string split on punctuation

    • Put algorigthm into ConfigUtility - getIndexerNormalizedString
      • String.join(" ","hiv antibody".toLowerCase().split("[ \t-({[)}]_!@#%&*\\:;\"',.?/~+=|<>$`^]")).replaceAll(" +"," ");
    • Default search handler can include as a boosted "or" clause in the query, e.g. by 10
      • if literalField.endsWith("Sort"), create normField = literalField.replace("Sort","Norm"),
      • add an OR clause with this field and the normalized value.
    • "HIV Antibody" => 
      • (HIV Antibody OR nameNorm:\"hiv antibody\"^10.0 OR nameSort:\"HIV Antibody\"^20.0) AND ...
    • "1-2,Anydroxy" =>
      • (1-2,Anhydroxy OR nameNorm:\"1 2 anhydroxy\"^10.0 OR nameSort:\"1-2,Anhydroxy\"^20.0) AND ...
  • DONE PG Clear filters on new searches or navigating (e.g. atom filters stick around)
    • Added function resetPaging(), called on load (instantiates filters and paging)
    • getComponentFromTree and getComponentFromType now clear paging on successful component retrieval
  • DONE PG Relationship sort order - allow sorting on REL, RELA, or TO(terminology)ID
  • DONE PG Color code search results based on log base value -> GREEN for high, YELLOW, RED for low.  Based on score indexed to a 1 value.
    • Normalize scores with a "log base", e.g. take the raw score (may be 12.5, may be 0.83) Choose a log base, e.g. "5", if the score is > 5, it gets a value of 1, otherwise it is the log of the score with the base (e.g. "5").
    • when getScoreMap is called -> every score should be in a range from 0 - 1
  • DONE PG: Advanced search - like mapping
    • Semantic type - [ picklist ]
      • semanticTypes.semanticType:"Clinical Drug"
    • term type  - [ picklist ]
      • atoms.termType:...
    • match terminology - [ picklist ] (only for "metathesaurus")
      • atoms.terminology:..
    • language  - [ picklist ]
      • atoms.language:...
  • DONE PG: support multiple sort fields
    • keep pfsParameter sortField
    • pfsParameter.get/setSortFields(List<String> fields)
    • sortFields overrides sortField
    • Handle in IndexUtility where new Sort(...) is.  Make sure to handle date & id fields as LONG and other fields as STRING.
  • DONE PG Relationship Filter needs to search more efficiently
    • DONE e.g. only single word matching currently performed ("600" does not return "600mg" terms)
      • Fixed by adding wildcard searches directly from webapp
    • DONE Misc. improvements:
      • Return to page 1 on change of filter or sort field/sort order
      • Add tooltip to sort order icon
    • DONE Relationships should be loaded with blank value instead of 0 (RRF loader, line 2151)
    • DONE Index toFromName on ConceptRelationshipJpa, CodeRelationshipJpa, DescriptorRelationshipJpa
    • QUESTION: Relationship searching is set up to handle inverse cases, and forces a fielded search in DefaultSearchHandler. Is this really what we want from the webapp?
  • DONE PG Implement atom class search handler
    • DONE Should search name and id fields 
    • DONE Search id fields if single term contains at least one number
    • DONE Should throw exception if invoked on a non-AbstractAtomClass
    • DONE Find reasonable weights
      • Terminology Id -> 4.0 NOTE This is across all fields (boosted expression)
        • Includes atom.conceptId, atom.descriptorId, atom.codeId
      • Name (Sort) -> 5.0
      • Name (Norm) -> 5.0 
      • Name (by split term) -> 1.0
    • DONE Split term needs to respect quoted terms, e.g. "a b", "a b" c "d e f"
  • DONE PG: Scale down DefaultSearchHandler
    • Get rid of acronym expansion
    • Get rid of spelling checks
    • Get rid of normalized field checking
    • Goal:  text in, results out, basic filtering/searching features
      • Fielded search
      • If no results literal search
  • PG: redo header/footer - controllers. (like refset tool)
  • PG:  Implement Diagramming
    • model transformation
    • Show ONLY for "description logic terminology"
  • Enhancements to RRF preferred name computer
    • support multiple UMLS's - e.g. have a high-level terminology/version
    • reuse the same default precedence list.
  • IN PROGRESS PG: Use directives for reports
    • Move all http calls to services.
  • DONE PG: Paging controls are no longer showing up for subcategories (e.g. atoms, relationships)
  • IN PROGRESS Use route params to dynamically load a "report" (and be able to bookmark URLs)
  • Support opening a concept in a new window (e.g. there's a pointing off arrow icon that opens a new window with a routing URL that  shows exactly that concept - then drag/drop between windows can be editing mechanism).
  • User Preferences stuff
    • Bring over model from Refset tool.
    • Add "last query" (e.g. "brain" and whether it's "list" or "tree" mode).
    • Add "last report" (e.g. type/ui/terminology/version)
  • Remove "void addXXX" and "void removeXXX" methods from model objects. Use getObjects().add/remove(...) only
    • e.g. AtomClass doesn't need addAtom or removeAtom
    • Rewire any uses of them  
    • "addXXX" -> "getXXX().add"
    • "removeXXX" -> "getXXX().remove"
    • same if there is "clear"
  • DSS: Model
    • Mapping, MapRecord, MapEntry (each with attributes) - 
      • similar to Subset, SubsetMember except with one more layer.
    • MapSet extends ComponentHasAttributes
      • name (String)
      • type (String)
      • fromTerminology/Version: String
      • toTerminology/Version: String
      • to/fromComplexity (String)
      • to/fromExhaustive (String)
      • get/setMappings -> use mappedFrom in @OneToMany
    • Mapping extends ComponentHasAttributes
      • fromTerminologyId, fromIdType
      • toTerminologyId, toIdType
      • mapSet
      • relationshipType, additionalRelationshipType
      • subsetId => group
      • rank
      • rule
      • restriction -> rename to 'advice'
      • In Jpa class
        • Add  @XmlTransient methods with @Field
        • getFromTermionlogy(), getFromVersion() - call getMapSet().getXXX
        • getToTermionlogy(), getToVersion() - call getMapSet().getXXX
    • Remove MapObject
    • MappingList/Jpa 
    • ContentService - Jpa
      • add/remove/update MapSet/Mapping
      • getMapping(Long id)
      • getMapping(String terminologyid, terminology, version, branch)
      • findMappingsForMapSet(Long mapSetId, query, pfs)
      • findMappingsForConcept(Long conceptId, query, pfs)
        • call getConcept(conceptId).
        • then perform a "from" search based on the terminologyId, terminology, version, and idType (e.g. CONCEPT)
      • findMappingsForCode(Long codetId, query, pfs)
      • findMappingsForDescriptor(Long descriptorId, query, pfs)
    • REST layer - ContentServiceRest (Impl,Client)
      • getMapsets
        • /mapset/all/{terminology}/{version}
      • getMapset
        • /mapset/{terminologyId}/{terminology}/{version}
      • findMappingsForMapSet(Long mapSetId, query, pfs)
        • /mapset/{terminologyId}/{terminology}/{version}/mappings
      • findMappingsForXXX(Long xxxId, query, pfs)
        • /cui/{terminology}/{version}/{terminologyId}/mappings
        • /code/...
        • /dui/...
    • Integration tests
      • jpa
      • REST
        • basic test of client layer - e.g. make sure the calls to the server actually work and return something
    • Unit Tests
      • MapSetJpa, MappingJpa, MappingListJpa
    • Loaders
      • Then implement this for RRF loader
        • MRSAT

          • Handle in loadMrsat

            • Have a class-level   Map<String, MapSet> mapSetMap variable
            • When encountering a "mapping" attribute, create an entry in this map if it doesn't exist
              • use the CUI as the key
            • Extrac the mapset from the map, then set the corresponding field
            • At the end of loadMrsat, for each mapSet in the variable, call service.addMapSet(...)

          • MTH_MAPFROMEXHAUSTIVE => mapSet.fromExhaustive

          • MTH_MAPTOEXHAUSTIVE => mapSet.toExhaustive

          • MTH_MAPSETCOMPLEXITY => mapSet.complexity

          • MTH_MAPSETFROMCOMPLEXITY => mapSet.fromComplexity
          • MTH_MAPSETTOCOMPLEXITY => mapSet.toComplexity

          • MAPSETSID = mapSet.terminologyId

          • MAPSETNAME = mapSet.name

          • MAPSETVERSION = mapSet.mapVersion

          • MAPSETFROMVERSION = mapSet.mapVersion
          • MAPSETTOVERSION = mapSet.mapVersion
          MRMAP:
          • Add a loadMrmap method
            • use mapSetMap when processing mapping entries.
            • skip entries where the FROMTYPE or TOTYPE fields cannot be resolved to an IdType enum (IdType.valueOf(...) fails)
            • Look up the MapSetCui in mapSetMap, use this for mapping.setMapSet
          • MAPSETCUI - used to look up in mapSetMap
          • MAPSETSAB - mapSet.terminology
          • MAPSUBSETID = mapping.group
          • MAPRANK = mapping.rank
          • MAPID, MAPSID = mapping attribute, name="MAPID" value = "value of the MAPID field"
          • FROMID, FROMSID = mapping attribute
          • FROMEXPR = mapping.fromTerminologyId
          • FROMTYPE = mapping.fromIdType
          • FROMRULE, FROMRES = mapping attribute
          • REL,RELA = mapping.relationshipType, additionalRelationshipType
          • TOID,TOSID = mapping attribute
          • TOEXPR = mapping.toTerminologyId
          • TOTYPE = mapping.toIdType
          • TORULE,TORES = mapping attribute
          • MAPRULE = mapping.rule
          • MAPRES = mapping.advice
          • MAPTYPE = mapping.type
          • MAPATN,MAPATV = mapping attribute
          • CVF - n/a
      • Then implement this for RF2 loader (snapshot)
      • Then implement this for RF2 loader (delta)
    • Show mappings in UI.
  • Show atom-level subset information (e.g. language refset entires in SNOMED. may require a change to the graph resolver).

Core Application Enhancement - P2

  • Release criteria integration tests
    • Verify no *java classes have System.out.println
    • Verify no *java classes have TODO
    • ...
  • PG: Support RF2 "atom association refset members" as atom relationships.
    • * REMAINING: Need to decide how to handle non-concept association reference refset members (e.g. descriptions)
    Enable glass pane while switching tabs (between content/metadata)
  • Bring "source data file" and "source data" over from transformer application, along with other things.
    • Put the objects into the package structure of terminology server (e.g. put with Project and User)
    • Need to update transformer too so it continues to work.
    • Leave mojos behind. 
  • Add features for "deep" relationships when browsing UMLS.    
    • it is definitely only for "concept" and for "metathesaurus" content
    • Add new section to concept report (e.g. "Deep relationships")
  • Advanced search
    • LATER: Expression. - e..g. "Search Criteria" - for "descriptionLogicTerminology"  only
      • descendants of
      • has relationship -> xx
  • Show "atom" subset information in the report.  Only concept (or component) subset (or refset) information is being shown.  this may involve a change in the graph resolver to return the data.  i.e. for Snomed you should be able to tell what is just british.

Editing Features

  • Basic metathesaurus editing
    • Add/remove STY
    • Add/remove atom
    • Move atom
    • Split atoms from concept
    • Merge concepts
    • Add/remove concept relationship
  • Publication Process
    • RRF
  • Project
    • Figure out how to capture "project scope" for SNOMED and for UMLS in a generalized way.  Update project objects to be able to properly capture (and compute) project scope.  NOTE: the scope definition may involve concepts/terminologies/semantic types.  IN that event, the scope computer gets a little bit more complicated.
  • Test loading a DB with envers auditing disabled and then making changes in a DB while it is enabled. Does it properly create audit entries?
    • for the old edition of the component?
    • for the new edition?
  • Metathesaurus editing actions
    • MetathesaurusContentEditingRest
      • methods for each "edit action"
      • Create a RestImpl
      • Create a client
      • Create integration tests to run against a "stock" dev  database
    • Add a semantic type component, Remove a semantic type component
      • Have a pencil icon by  the STYs section
      • clicking gives you a list of available STYs assigned, in tree order with a filter box for typing characters of the STY you want.
        • See the metadata "semantic type" option
      • User may want to choose multiple ones (so have a "finished" button)
      • Dont allow user to choose STYs already assigned to the concept.
      • Final action is to call "contentService.addSemanticTypeComponent"
      • Consider what happens to workflow status
      • Consider how to show "NEEDS_REVIEW" content in the browser
      • Consider how to support "Undo". - perhaps an action log (atomic/molecular) is a good idea still for that
    • Implement this completely including testing before moving on to other actions (each which requires a UI enhancement)
      • Approve a concept (e.g. set workflow status values).
      • Add an atom (e.g. MTH/PN - that needs to be generalized somehow)
      • Merge two concepts (consider the "workflow status "when this happens).
      • Move an atom (or atoms) from one concept to another
      • Split an atom (or atoms) out of a concept and specify a relationship type between the two concept
  • Terminology Editing (first use case)
    • Add a concept (as a child of an existing concept) with one or more atoms and a PAR/CHD relationship.
    • Run the classifier
    • Show classifier results (e.g. new inferred relationships, etc)
    • NOTE: this only works with a description logic based source that tracks inferred relationships.
    • PREREQ: SNOMEDCT RF2 loader.

Other Things

  • Create initial framework for JS unit testing
  • Verify that ClaML loader makes asterisk/dagger attributes for modifier concepts - e.g. M01.4 and M01.41
  • Integration tests for static code analysis (e.g. *.code.XXTest.java)
    • REST Service
      • opened sessions are closed in finally blocks.
  • For "semantic type" mechanism - if there are tree numbers, order by that and indent"
  • OWL - NCIt

User Interface Enhancements

  • Do something useful with websocket
  • Expression-based searching (get harold's parser?)
  • Consider adding "LABELFOR" all subsets and making the star pop up a picklist of the things to highlight (ordered by type with extensions first, subsets later)
  • RECURRING: Mobile-friendly and other style issues

Additional/Enhanced  Loaders

  • Owl loader - have a snorocket (2) reasoner for role relationships.

Services

  • Action Service
    • Implement classification.
    • Need to go to/from Owl so do Owl loader FIRST.

Testing 

  • Implement additional unit tests for model objects (PrecedenceListJpa, LabelSet, etc)
  • RRF loader -> create label set for SNOMED (both "single" and "umls")
  • Handler002Test for normal use
  • Implement Handler003/008Test - for ID assignment algorithms.  Borrow code from other project (though there may be differences).  The uuidHash algorithm is implemented properly for UMLS and may be different than for SNOMED.

Admin Tools

  • Test QA queries and flesh them out for 100% coverage.

Optimizations

  • TBD

 

Future Stuff

  • Test conditional envers auditing: http://stackoverflow.com/questions/14250612/conditional-envers-auditing
  • escg (expression grammar - research)
  • Use Lucene SynonymFilter with synonym table
  • Component-Component relationships (between any two components).
  • Value set definitions (and marking of subset.isValueSet()) and linking to definition? via attribute?
  • Owl loader, Owl export of DL terminologies (e.g. RF2-loaded SNOMED)
  • Rdf export (flat)
  • Classifier (owl interface)
  • Expression language (based on SNOMED expression constraint grammar)
  • Sub-branching
    • branchResolutionHandler - figures out how to copy and mark branched objects  and update indexes  - for different branching strategies.
  • Handle mappings - may be not worth it
  • Implement an RF2 loader (use DL features)
  • Implement a ClaML loader
  • Support semantic network (e.g. sty rels, etc).  - probably want to wait for a real ontology - and maybe even load it as such.

 

DONE

DONE: Small Changes/Fixes

  • DONE Attribute abbr tooltips not working for MSH but working for SNOMEDCT, see D006321 for example

  • DONE Relationship tooltips not working for MSH, but working for SNOMEDCT, see D006321 for example

    • Apparently no problem, probably a caching issue

  • DONE View All History button nonfunctional (navigation forward/back works, but no list)

  • DONE Show a suppressible indicator (an obsolete indicator is showing, but not a surppessible one)

    • suppressibleLabel and suppressibleIndicator not set in webapp metadata

  • DONE Combine "suppressible" and "obsolete" buttons into one -> e.g. the either both flags are on, or both flags are off.
    • Merged into showSOElements, with function toggleShowSOElements
  • DONE BUT TEST If "Atoms (0)", and "show suppressible/obsolete" is OFF, then show a message to click the "show" button to see the obsolete content.
    • Only shown if atoms.length > 0 (i.e. suppressible/obsolete content actually present)
  • Application should scroll to top on click of concept links
  • NEEDS TESTING Load association reference data as relationships (mark them as suppressible?)
    • INFO id effectiveTime active moduleId refsetId referencedComponentId targetComponentId
    • DONE Fail if the referencedComponentId is not a concept terminology id.
    • DONE Make a ConceptRelationshipJpa
      • id = terminologyId
      • timestamp/lastModified = effectiveTime
      • !active = obsolete
      • published = true, publishable = true, ... 
      • moduleId = becomes an attribute, see how it works in loadRelationships
      • referencedComponentId, look up the concept -> setFrom
      • targetComponentId, look up the concept -> setTo
      • relationshipType = "RO"
      • additionalRelationshipType = refsetId
        • Need to make sure a metadata additionalRelationshipType gets added
          • abbreviation = refsetId, expandedForm = name of that concept (e.g. the part in capitals at the beginning, "POSSIBLY EQUIVALENT TO")
        • make sure to not add it more than once.
      • Outstanding Questions
        • CONFIRMED NO NEED Need to preserve default of AtomSubsetMember or AtomRelationshipMember?
        • No unexpected consequences for SNOMEDCT display/processing of:
          • No group
          • Not stated, not inferred
        • Possible case of Suppressible meaning something different than Obsolete?
        • Worth doublechecking existence of refset id for additional relationship types in loadAssociationReference, or just let exception thrown if refset id is not a concept?
  • NEEDS TESTING Delta loader for association reference data
    • see loadRelationships for how to compute diffs with a relationships
  • Other small fixes - found by PG.
    • test...
    • update copyright to 2016
  • "Release"