Versions Compared

Version Old Version 35 New Version Current
Changes made by

bcarlsen

Patrick Granvold (Unlicensed)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • DONE Intro page that doesn't have as login
    • DONE Initial page now routes to "landing" view, with landing.html. Initial version includes only placeholder text and link to #/login
    • Note: May want to introduce route bypass if user hits login page while logged in (i.e. content -> landing or landing -> content). Or may not.
  • New front-page website with description, screenshots, video
  • "Login" link (with logout link on header)
  • Have the "upload" and "source" tabs from transformer here
    • "Snomed starter kit"
  • Create a video demo of the site (camtasia) and post as a link on the header (video glyphicon if there is one)
    • also screenshots that are clickable, etc.
  • Verify all entry pages for applications are officially mobile friendly
  • Update campaigns for  SNOMED/UMLS/ICD browsers on google adwords.
  • Training Video for UMLS browser (need 10 min)
  • Training Video for SNOMED browser (simple 5 min)
  • Find the SNOMED video that Anil sent, redo with RF2 - same content!
  • Other notes (from email)
    • Google adwords campaign
      • “free icd-10” browser, “free SNOMED CT US Extension browser”, “free UMLS browser
    • Get webmaster setup for the icd/umls/snomed pages
      • Use config setup for that
    • Find east bay healthcare startups
      • Pursue like job search (to know they’re hiring, etc)
      • Tech
      • Device
      • Information/informatics 
    • Meetups

Core Application Enhancement - P1

  • DONE (BC): Fix long error header - attach icon not showing, where it displays error is not in the error area.
  • DONE PG: Need to index atoms as lowercase string, punctuation replaced by whitespace, no trailing, duplicate, or leading whitespace.
    • Create a new tokenizer factory for this and define in @AnalyzerDefs in AbstractAtomClass
    • "Cholsterol.in HDL/Cholesterol.in LDL" => " cholesterol in hdl cholesterol in ldl"
    • "ABC def----ghi" => "abc def ghi"

    • "ABC     ////df3948123,.3482932–..xx abc" => "abc df3948123 3482932 xx abc"

    • Add an AtomJpa method called getNameNorm() that isn't analyzed but returns a lowercased string split on punctuation

    • Put algorigthm into ConfigUtility - getIndexerNormalizedString
      • String.join(" ","hiv antibody".toLowerCase().split("[ \t-({[)}]_!@#%&*\\:;\"',.?/~+=|<>$`^]")).replaceAll(" +"," ");
    • Default search handler can include as a boosted "or" clause in the query, e.g. by 10
      • if literalField.endsWith("Sort"), create normField = literalField.replace("Sort","Norm"),
      • add an OR clause with this field and the normalized value.
    • "HIV Antibody" => 
      • (HIV Antibody OR nameNorm:\"hiv antibody\"^10.0 OR nameSort:\"HIV Antibody\"^20.0) AND ...
    • "1-2,Anydroxy" =>
      • (1-2,Anhydroxy OR nameNorm:\"1 2 anhydroxy\"^10.0 OR nameSort:\"1-2,Anhydroxy\"^20.0) AND ...
  • PG: Support RF2 "atom association refset members" as atom relationships.
    • REMAINING: Need to decide how to handle non-concept association reference refset members (e.g. descriptions)
  • DONE PG Clear filters on new searches or navigating (e.g. atom filters stick around)
    • Relationship sort order - allow sorting on REL, RELA, or TO(terminology)ID
      • Added function resetPaging(), called on load (instantiates filters and paging)
      • getComponentFromTree and getComponentFromType now clear paging on successful component retrieval
    • DONE PG Relationship sort order - allow sorting on REL, RELA, or TO(terminology)ID
    • DONE PG Color code search results based on log base value -> GREEN for high, YELLOW, RED for low.  Based on score indexed to a 1 value.
      • Advanced search - like mapping
        • Semantic type
        • terminology
        • term type
        • others?
      • PG: redo header/footer - controllers. (like refset tool)
      • PG:  Implement Diagramming
        • model transformation
        • Show ONLY for "description logic terminology"
        • Enhancements to RRF preferred name computer
        • support multiple UMLS's - e.g. have a high-level terminology/version
        • reuse the same default precedence list.
      • PG: Use directives for reports
        • Move all http calls to services.
      • DONE PG: Paging controls are no longer showing up for subcategories (e.g. atoms, relationships)
      • Use route params to dynamically load a "report" (and be able to bookmark URLs)
      • Support opening a concept in a new window (e.g. there's a pointing off arrow icon that opens a new window with a routing URL that  shows exactly that concept - then drag/drop between windows can be editing mechanism).
      • User Preferences stuff
        • Bring over model from Refset tool.
        • Add "last query" (e.g. "brain" and whether it's "list" or "tree" mode).
        • Add "last report" (e.g. type/ui/terminology/version)
      • Remove "void addXXX" and "void removeXXX" methods from model objects. Use getObjects().add/remove(...) only
        • e.g. AtomClass doesn't need addAtom or removeAtom
        • Rewire any uses of them  
        • "addXXX" -> "getXXX().add"
        • "removeXXX" -> "getXXX().remove"
        • same if there is "clear"
      • DSS: Model
        • Mapping, MapRecord, MapEntry (each with attributes) - 
          • similar to Subset, SubsetMember except with one more layer.
        • MapSet extends ComponentHasAttributes
          • name (String)
          • type (String)
          • fromTerminology/Version: String
          • toTerminology/Version: String
          • to/fromComplexity (String)
          • to/fromExhaustive (String)
          • get/setMappings -> use mappedFrom in @OneToMany
        • Mapping extends ComponentHasAttributes
          • fromTerminologyId, fromIdType
          • toTerminologyId, toIdType
          • mapSet
          • relationshipType, additionalRelationshipType
          • subsetId => group
          • rank
          • rule
          • restriction -> rename to 'advice'
          • In Jpa class
            • Add  @XmlTransient methods with @Field
            • getFromTermionlogy(), getFromVersion() - call getMapSet().getXXX
            • getToTermionlogy(), getToVersion() - call getMapSet().getXXX
        • Remove MapObject
        • MappingList/Jpa 
        • ContentService - Jpa
          • add/remove/update MapSet/Mapping
          • getMapping(Long id)
          • getMapping(String terminologyid, terminology, version, branch)
          • findMappingsForMapSet(Long mapSetId, query, pfs)
          • findMappingsForConcept(Long conceptId, query, pfs)
            • call getConcept(conceptId).
            • then perform a "from" search based on the terminologyId, terminology, version, and idType (e.g. CONCEPT)
          • findMappingsForCode(Long codetId, query, pfs)
          • findMappingsForDescriptor(Long descriptorId, query, pfs)
        • REST layer - ContentServiceRest (Impl,Client)
          • getMapsets
            • /mapset/all/{terminology}/{version}
          • getMapset
            • /mapset/{terminologyId}/{terminology}/{version}
          • findMappingsForMapSet(Long mapSetId, query, pfs)
            • /mapset/{terminologyId}/{terminology}/{version}/mappings
          • findMappingsForXXX(Long xxxId, query, pfs)
            • /cui/{terminology}/{version}/{terminologyId}/mappings
            • /code/...
            • /dui/...
        • Integration tests
          • jpa
          • REST
            • basic test of client layer - e.g. make sure the calls to the server actually work and return something
        • Unit Tests
          • MapSetJpa, MappingJpa, MappingListJpa
        • Loaders
          • Then implement this for RRF loader
            • MRSAT

              • Handle in loadMrsat

                • Have a class-level   Map<String, MapSet> mapSetMap variable
                • When encountering a "mapping" attribute, create an entry in this map if it doesn't exist
                  • use the CUI as the key
                • Extrac the mapset from the map, then set the corresponding field
                • At the end of loadMrsat, for each mapSet in the variable, call service.addMapSet(...)

              • MTH_MAPFROMEXHAUSTIVE => mapSet.fromExhaustive

              • MTH_MAPTOEXHAUSTIVE => mapSet.toExhaustive

              • MTH_MAPSETCOMPLEXITY => mapSet.complexity

              • MTH_MAPSETFROMCOMPLEXITY => mapSet.fromComplexity
              • MTH_MAPSETTOCOMPLEXITY => mapSet.toComplexity

              • MAPSETSID = mapSet.terminologyId

              • MAPSETNAME = mapSet.name

              • MAPSETVERSION = mapSet.mapVersion

              • MAPSETFROMVERSION = mapSet.mapVersion
              • MAPSETTOVERSION = mapSet.mapVersion
              MRMAP:
              • Add a loadMrmap method
                • use mapSetMap when processing mapping entries.
                • skip entries where the FROMTYPE or TOTYPE fields cannot be resolved to an IdType enum (IdType.valueOf(...) fails)
                • Look up the MapSetCui in mapSetMap, use this for mapping.setMapSet
              • MAPSETCUI - used to look up in mapSetMap
              • MAPSETSAB - mapSet.terminology
              • MAPSUBSETID = mapping.group
              • MAPRANK = mapping.rank
              • MAPID, MAPSID = mapping attribute, name="MAPID" value = "value of the MAPID field"
              • FROMID, FROMSID = mapping attribute
              • FROMEXPR = mapping.fromTerminologyId
              • FROMTYPE = mapping.fromIdType
              • FROMRULE, FROMRES = mapping attribute
              • REL,RELA = mapping.relationshipType, additionalRelationshipType
              • TOID,TOSID = mapping attribute
              • TOEXPR = mapping.toTerminologyId
              • TOTYPE = mapping.toIdType
              • TORULE,TORES = mapping attribute
              • MAPRULE = mapping.rule
              • MAPRES = mapping.advice
              • MAPTYPE = mapping.type
              • MAPATN,MAPATV = mapping attribute
              • CVF - n/a
          • Then implement this for RF2 loader (snapshot)
          • Then implement this for RF2 loader (delta)
        • Show mappings in UI.
      • Enable glass pane while switching tabs (between content/metadata)
      • Bring "source data file" and "source data" over from transformer application, along with other things.
        • Put the objects into the package structure of terminology server (e.g. put with Project and User)
        • Need to update transformer too so it continues to work.
        • Leave mojos behind. 
      • Add features for "deep" relationships when browsing UMLS.    
        • it is definitely only for "concept" and for "metathesaurus" content
        • Add new section to concept report (e.g. "Deep relationships")Normalize scores with a "log base", e.g. take the raw score (may be 12.5, may be 0.83) Choose a log base, e.g. "5", if the score is > 5, it gets a value of 1, otherwise it is the log of the score with the base (e.g. "5").
        • when getScoreMap is called -> every score should be in a range from 0 - 1
      • DONE PG: Advanced search - like mapping
        • Semantic type - [ picklist ]
          • semanticTypes.semanticType:"Clinical Drug"
        • term type  - [ picklist ]
          • atoms.termType:...
        • match terminology - [ picklist ] (only for "metathesaurus")
          • atoms.terminology:..
        • language  - [ picklist ]
          • atoms.language:...
      • DONE PG: support multiple sort fields
        • keep pfsParameter sortField
        • pfsParameter.get/setSortFields(List<String> fields)
        • sortFields overrides sortField
        • Handle in IndexUtility where new Sort(...) is.  Make sure to handle date & id fields as LONG and other fields as STRING.
      • DONE PG Relationship Filter needs to search more efficiently
        • DONE e.g. only single word matching currently performed ("600" does not return "600mg" terms)
          • Fixed by adding wildcard searches directly from webapp
        • DONE Misc. improvements:
          • Return to page 1 on change of filter or sort field/sort order
          • Add tooltip to sort order icon
        • DONE Relationships should be loaded with blank value instead of 0 (RRF loader, line 2151)
        • DONE Index toFromName on ConceptRelationshipJpa, CodeRelationshipJpa, DescriptorRelationshipJpa
        • QUESTION: Relationship searching is set up to handle inverse cases, and forces a fielded search in DefaultSearchHandler. Is this really what we want from the webapp?
      • DONE PG Implement atom class search handler
        • DONE Should search name and id fields 
        • DONE Search id fields if single term contains at least one number
        • DONE Should throw exception if invoked on a non-AbstractAtomClass
        • DONE Find reasonable weights
          • Terminology Id -> 4.0 NOTE This is across all fields (boosted expression)
            • Includes atom.conceptId, atom.descriptorId, atom.codeId
          • Name (Sort) -> 5.0
          • Name (Norm) -> 5.0 
          • Name (by split term) -> 1.0
        • DONE Split term needs to respect quoted terms, e.g. "a b", "a b" c "d e f"
      • DONE PG: Scale down DefaultSearchHandler
        • Get rid of acronym expansion
        • Get rid of spelling checks
        • Get rid of normalized field checking
        • Goal:  text in, results out, basic filtering/searching features
          • Fielded search
          • If no results literal search
      • PG: redo header/footer - controllers. (like refset tool)
      • PG:  Implement Diagramming
        • model transformation
        • Show ONLY for "description logic terminology"
      • Enhancements to RRF preferred name computer
        • support multiple UMLS's - e.g. have a high-level terminology/version
        • reuse the same default precedence list.
      • IN PROGRESS PG: Use directives for reports
        • Move all http calls to services.
      • DONE PG: Paging controls are no longer showing up for subcategories (e.g. atoms, relationships)
      • IN PROGRESS Use route params to dynamically load a "report" (and be able to bookmark URLs)
      • Support opening a concept in a new window (e.g. there's a pointing off arrow icon that opens a new window with a routing URL that  shows exactly that concept - then drag/drop between windows can be editing mechanism).
      • User Preferences stuff
        • Bring over model from Refset tool.
        • Add "last query" (e.g. "brain" and whether it's "list" or "tree" mode).
        • Add "last report" (e.g. type/ui/terminology/version)
      • Remove "void addXXX" and "void removeXXX" methods from model objects. Use getObjects().add/remove(...) only
        • e.g. AtomClass doesn't need addAtom or removeAtom
        • Rewire any uses of them  
        • "addXXX" -> "getXXX().add"
        • "removeXXX" -> "getXXX().remove"
        • same if there is "clear"
      • DSS: Model
        • Mapping, MapRecord, MapEntry (each with attributes) - 
          • similar to Subset, SubsetMember except with one more layer.
        • MapSet extends ComponentHasAttributes
          • name (String)
          • type (String)
          • fromTerminology/Version: String
          • toTerminology/Version: String
          • to/fromComplexity (String)
          • to/fromExhaustive (String)
          • get/setMappings -> use mappedFrom in @OneToMany
        • Mapping extends ComponentHasAttributes
          • fromTerminologyId, fromIdType
          • toTerminologyId, toIdType
          • mapSet
          • relationshipType, additionalRelationshipType
          • subsetId => group
          • rank
          • rule
          • restriction -> rename to 'advice'
          • In Jpa class
            • Add  @XmlTransient methods with @Field
            • getFromTermionlogy(), getFromVersion() - call getMapSet().getXXX
            • getToTermionlogy(), getToVersion() - call getMapSet().getXXX
        • Remove MapObject
        • MappingList/Jpa 
        • ContentService - Jpa
          • add/remove/update MapSet/Mapping
          • getMapping(Long id)
          • getMapping(String terminologyid, terminology, version, branch)
          • findMappingsForMapSet(Long mapSetId, query, pfs)
          • findMappingsForConcept(Long conceptId, query, pfs)
            • call getConcept(conceptId).
            • then perform a "from" search based on the terminologyId, terminology, version, and idType (e.g. CONCEPT)
          • findMappingsForCode(Long codetId, query, pfs)
          • findMappingsForDescriptor(Long descriptorId, query, pfs)
        • REST layer - ContentServiceRest (Impl,Client)
          • getMapsets
            • /mapset/all/{terminology}/{version}
          • getMapset
            • /mapset/{terminologyId}/{terminology}/{version}
          • findMappingsForMapSet(Long mapSetId, query, pfs)
            • /mapset/{terminologyId}/{terminology}/{version}/mappings
          • findMappingsForXXX(Long xxxId, query, pfs)
            • /cui/{terminology}/{version}/{terminologyId}/mappings
            • /code/...
            • /dui/...
        • Integration tests
          • jpa
          • REST
            • basic test of client layer - e.g. make sure the calls to the server actually work and return something
        • Unit Tests
          • MapSetJpa, MappingJpa, MappingListJpa
        • Loaders
          • Then implement this for RRF loader
            • MRSAT

              • Handle in loadMrsat

                • Have a class-level   Map<String, MapSet> mapSetMap variable
                • When encountering a "mapping" attribute, create an entry in this map if it doesn't exist
                  • use the CUI as the key
                • Extrac the mapset from the map, then set the corresponding field
                • At the end of loadMrsat, for each mapSet in the variable, call service.addMapSet(...)

              • MTH_MAPFROMEXHAUSTIVE => mapSet.fromExhaustive

              • MTH_MAPTOEXHAUSTIVE => mapSet.toExhaustive

              • MTH_MAPSETCOMPLEXITY => mapSet.complexity

              • MTH_MAPSETFROMCOMPLEXITY => mapSet.fromComplexity
              • MTH_MAPSETTOCOMPLEXITY => mapSet.toComplexity

              • MAPSETSID = mapSet.terminologyId

              • MAPSETNAME = mapSet.name

              • MAPSETVERSION = mapSet.mapVersion

              • MAPSETFROMVERSION = mapSet.mapVersion
              • MAPSETTOVERSION = mapSet.mapVersion
              MRMAP:
              • Add a loadMrmap method
                • use mapSetMap when processing mapping entries.
                • skip entries where the FROMTYPE or TOTYPE fields cannot be resolved to an IdType enum (IdType.valueOf(...) fails)
                • Look up the MapSetCui in mapSetMap, use this for mapping.setMapSet
              • MAPSETCUI - used to look up in mapSetMap
              • MAPSETSAB - mapSet.terminology
              • MAPSUBSETID = mapping.group
              • MAPRANK = mapping.rank
              • MAPID, MAPSID = mapping attribute, name="MAPID" value = "value of the MAPID field"
              • FROMID, FROMSID = mapping attribute
              • FROMEXPR = mapping.fromTerminologyId
              • FROMTYPE = mapping.fromIdType
              • FROMRULE, FROMRES = mapping attribute
              • REL,RELA = mapping.relationshipType, additionalRelationshipType
              • TOID,TOSID = mapping attribute
              • TOEXPR = mapping.toTerminologyId
              • TOTYPE = mapping.toIdType
              • TORULE,TORES = mapping attribute
              • MAPRULE = mapping.rule
              • MAPRES = mapping.advice
              • MAPTYPE = mapping.type
              • MAPATN,MAPATV = mapping attribute
              • CVF - n/a
          • Then implement this for RF2 loader (snapshot)
          • Then implement this for RF2 loader (delta)
        • Show mappings in UI.
      • Show atom-level subset information (e.g. language refset entires in SNOMED. may require a change to the graph resolver).

      Core Application Enhancement - P2

      • Release criteria integration tests
        • Verify no *java classes have System.out.println
        • Verify no *java classes have TODO
        • ...
      • PG: Support RF2 "atom association refset members" as atom relationships.
        • * REMAINING: Need to decide how to handle non-concept association reference refset members (e.g. descriptions)
        Enable glass pane while switching tabs (between content/metadata)
      • Bring "source data file" and "source data" over from transformer application, along with other things.
        • Put the objects into the package structure of terminology server (e.g. put with Project and User)
        • Need to update transformer too so it continues to work.
        • Leave mojos behind. 
      • Add features for "deep" relationships when browsing UMLS.    
        • it is definitely only for "concept" and for "metathesaurus" content
        • Add new section to concept report (e.g. "Deep relationships")
      • Advanced search
        • LATER: Expression. - e..g. "Search Criteria" - for "descriptionLogicTerminology"  only
          • descendants of
          • has relationship -> xx
      • Show "atom" subset information in the report.  Only concept (or component) subset (or refset) information is being shown.  this may involve a change in the graph resolver to return the data.  i.e. for Snomed you should be able to tell what is just british.

      Editing Features

      • Basic metathesaurus editing
        • Add/remove STY
        • Add/remove atom
        • Move atom
        • Split atoms from concept
        • Merge concepts
        • Add/remove concept relationship
      • Publication Process
        • RRF
      • Project
        • Figure out how to capture "project scope" for SNOMED and for UMLS in a generalized way.  Update project objects to be able to properly capture (and compute) project scope.  NOTE: the scope definition may involve concepts/terminologies/semantic types.  IN that event, the scope computer gets a little bit more complicated.
      • Test loading a DB with envers auditing disabled and then making changes in a DB while it is enabled. Does it properly create audit entries?
        • for the old edition of the component?
        • for the new edition?
      • Metathesaurus editing actions
        • MetathesaurusContentEditingRest
          • methods for each "edit action"
          • Create a RestImpl
          • Create a client
          • Create integration tests to run against a "stock" dev  database
        • Add a semantic type component, Remove a semantic type component
          • Have a pencil icon by  the STYs section
          • clicking gives you a list of available STYs assigned, in tree order with a filter box for typing characters of the STY you want.
            • See the metadata "semantic type" option
          • User may want to choose multiple ones (so have a "finished" button)
          • Dont allow user to choose STYs already assigned to the concept.
          • Final action is to call "contentService.addSemanticTypeComponent"
          • Consider what happens to workflow status
          • Consider how to show "NEEDS_REVIEW" content in the browser
          • Consider how to support "Undo". - perhaps an action log (atomic/molecular) is a good idea still for that
        • Implement this completely including testing before moving on to other actions (each which requires a UI enhancement)
          • Approve a concept (e.g. set workflow status values).
          • Add an atom (e.g. MTH/PN - that needs to be generalized somehow)
          • Merge two concepts (consider the "workflow status "when this happens).
          • Move an atom (or atoms) from one concept to another
          • Split an atom (or atoms) out of a concept and specify a relationship type between the two concept
      • Terminology Editing (first use case)
        • Add a concept (as a child of an existing concept) with one or more atoms and a PAR/CHD relationship.
        • Run the classifier
        • Show classifier results (e.g. new inferred relationships, etc)
        • NOTE: this only works with a description logic based source that tracks inferred relationships.
        • PREREQ: SNOMEDCT RF2 loader.

      ...