MID Validation


Implementation

MID Validation checks are implemented as bins in the Workflow tab. When the check passes, it should return 0 results. The bins drive the action of the MIDValidationReport algorithm which runs as a cron job each Saturday.

  • MID_VALIDATION: Queries that can additionally be made into checklists (i.e., in format clusterId, conceptId OR conceptId OR conceptId1, conceptId2)
  • MID_VALIDATION_OTHER: Queries that cannot be made into checklists (i.e., the query doesn't not return a conceptId). These are not intended to be made into bins, and will not run correctly as bins. However, they still drive the action of MIDValidationReport.

To test either type of query, implement as a bin, and then use "Test" in the Edit Bin screen.

To implement "exceptions" to the checks, add them to the query.

Parameters

  • terminology - e.g. NCIMTH
  • version - e.g. latest

Query Patterns

Finding atoms for a concept (also works for a code or a descriptor)

select a.* from atoms a, concepts_atoms b, concepts c
where a.id = b.atoms_id and b.concepts_id = c.id ...;

Find semantic types for a concept

select sty.* from concepts a, concepts_semantic_type_components b, semantic_type_components sty
where a.id = b.concepts_id and b.semanticTypes_id = sty.id ...;

Find AUIs for an atom

select b.alternateTerminologyIds aui
from atoms a, AtomJpa_alternateTerminologyIds b
where a.id = b.AtomJpa_id and alternateTerminologyIds_KEY = :terminology ...;

Find SRC_ATOM_ID for an atom

select a.alternateTerminologyIds srcAtomId
from atoms a, AtomJpa_alternateTerminologyIds b
where a.id = b.AtomJpa_id and alternateTerminologyIds_KEY = concat(:terminology,'-SRC') ...;

Find last release CUI for an atom

select a.conceptTerminologyIds srcAtomId
from atoms a, AtomJpa_conceptTerminologyIds b
where a.id = b.AtomJpa_id and conceptTerminologyIds_KEY = :terminology ...;

Find the CUI for a concept (note these are not fully assigned until release time)

select c.terminologyId cui from concepts c where terminology = :terminology ...;

Find the ATUI for a semantic type (note these are not fully assigned until release time)

select sty.terminologyId atui from semantic_type_components where terminology = :terminology...;

Find the RUI for C level relationships (note these are not fully assigned until release time)

select r.terminologyId rui from concept_relationships r where terminology = :terminology ...;

Find the RUI for a SNOMED SCUI-SCUI relationship

select r.alternateTerminologyIds rui
from concept_relationships r, ConceptRelationshipJpa_alternateTerminologyIds b
where r.id = b.ConceptRelationshipJpa_id and alternateTerminologyIds_KEY = :terminology ...;

Find the sab/tty termgroups

-- default precedence list
select a.id, sab.terminologies, tty.termTypes
from precedence_lists a, precedence_list_terminologies sab, precedence_list_term_types tty
where sab.terminologies_ORDER = tty.termTypes_ORDER
  and a.id = sab.PrecedenceListJpa_id and a.id = tty.PrecedenceListJpa_id
  and a.terminology = :terminology;
 
-- project precedence list
select a.id, sab.terminologies, tty.termTypes
from precedence_lists a, precedence_list_terminologies sab, precedence_list_term_types tty
where sab.terminologies_ORDER = tty.termTypes_ORDER
  and a.id = sab.PrecedenceListJpa_id and a.id = tty.PrecedenceListJpa_id
  and a.id in (select precedenceList_id from projects where terminology = :terminology);


Find inverse of a relationship

select * from additional_relationship_types....


Existing Checks

(to derive queries from or for)

Approved C rel matching demotion - DONE

select cr.from_id conceptId1, cr.to_id conceptId2
from atom_relationships ar, concept_relationships cr,
     concepts_atoms ca1, concepts_atoms ca2
where ar.terminology = :terminology and cr.terminology = :terminology
  and ar.from_id = ca1.atoms_id and ar.to_id = ca2.atoms_id
  and cr.from_id = ca1.concepts_id and cr.to_id = ca2.concepts_id
  and ar.workflowStatus = 'DEMOTION'
  and cr.workflowStatus in ('READY_FOR_PUBLICATION','PUBLISHED');


Atoms with leading/trailing junk - DONE, OTHER

select c.id, a.id conceptId from concepts c, concepts_atoms ca,
    (select id from atoms where name like ' %' or name like '% ') a
where c.terminology=:terminology and c.id = ca.concepts_id
and ca.atoms_id = a.id;


Atom with blank language value - DONE, OTHER

select c.id conceptId, a.id
from concepts c, concepts_atoms ca,
    (select id from atoms where language='') a
where ca.atoms_id = a.id and ca.concepts_id = c.id
and c.terminology = :terminology;

 
N/A - Attribute value with multiple hashcodes 0 Sat Jan 7 09:02:19 2017 - n/a hascodes are internally compute
N/A - C level atts with concept_id,sg_id out of sync 0 Sat Jan 7 09:02:52 2017 - n/a cannot happen
N/A - C level rels with concept_id,sg_id out of sync 0 Sat Jan 7 09:03:01 2017 - n/a cannot happen

Codes with leading/trailing spaces - DONE, OTHER

select codeId, id as atomId, terminology from atoms a where codeId like ' %' or codeId like '% '


N/A - Conflicting context relationships 0 Sat Jan 7 09:03:18 2017

N/A - Conflicting relationships 0* Sat Jan 7 09:03:35 2017

Current sources without releasable atoms - DONE, OTHER

select terminology,version
from terminologies b
where current = true
 and not exists
 (select * from atoms a
 where a.terminology = b.terminology
 and a.version=b.version
 and a.publishable = true);


N/A - Current ver attribute connected to old ver object
N/A - Current ver relationoship connected to old ver object
N/A: Current ver attribute connected to old ver cxt rels

Current ver attribute connected to old ver concept rels - DONE, OTHER - *** add other types of rels

-- concept_relationships
select at.id as attribute_id, c.id as concept_id, c.terminology from
concept_relationships c, concept_relationships_attributes c_at, attributes at
where c.id = c_at.concept_relationships_id and at.id = c_at.attributes_id
and (c.terminology,c.version) in (select terminology,version from terminologies where current = false)
and (at.terminology,at.version) in (select terminology,version from terminologies where current = true)
and at.publishable = true;


Current ver relationship connected to old ver atom - DONE, OTHER

select a_r.id, a_r.from_id, a.terminology, a_r.to_id
from atoms a, atom_relationships a_r
where a.id = a_r.from_id
and (a.terminology,a.version) in (select terminology,version from terminologies where current = false)
and (a_r.terminology,a_r.version) in (select terminology,version from terminologies where current = true)


N/A - Dangling XMAP (no XMAPFROM) 0 Sat Jan 7 09:05:46 2017
N/A - Dangling XMAP (no XMAPTO) 0 Sat Jan 7 09:05:54 2017
N/A - Dangling XMAPFROM 0 Sat Jan 7 09:05:59 2017
N/A - Dangling XMAPTO 0 Sat Jan 7 09:06:02 2017

Mapset without mappings - DONE, OTHER

select a.id, a.name
from mapsets a
where a.publishable = true
  and not exists
   (select * from mappings b where a.id = b.mapset_id)

Subset without members - DONE, OTHER

select a.id, a.name
from atom_subsets a
where a.publishable = true
  and not exists
   (select * from atom_subset_members b where a.id = b.subset_id)
union all
select a.id, a.name
from concept_subsets a
where a.publishable = true
  and not exists
   (select * from concept_subset_members b where a.id = b.subset_id);


Duplicate relationship type inverse - DONE, OTHER

select inverse_id from relationship_types group by inverse_id having count(*)>1;

Duplicate additional relationship type inverse - DONE, OTHER

select inverse_id from additional_relationship_types group by inverse_id having count(*)>1;

N/A - Hashcode with multiple attribute values 0 Sat Jan 7 09:06:16 2017
Illegal C level relationship (rn=RT?,LK,status=R)  - DONE, OTHER

select a.id from concept_relationships a
where terminology = :terminology
  and relationshipType not in ('XR','RO','RB','RN','BRO','BRB','BRN');

N/A - llegal C level rels (status=U or rel=LK) 0 Sat Jan 7 09:06:20 2017

Source relationship with status=N - DONE, OTHER

select a.id from concept_relationships a
where terminology != :terminology
  and workflowStatus = 'NEEDS_REVIEW'
union all
select a.id from atom_relationships a
where workflowStatus = 'NEEDS_REVIEW'
union all
select a.id from code_relationships a
where workflowStatus = 'NEEDS_REVIEW'
union all
select a.id from descriptor_relationships a
where workflowStatus = 'NEEDS_REVIEW';


?? Inconsistent hierarchical code 0* Sat Jan 7 09:06:30 2017

N/A - Incorrect text_total in stringtab 0 Sat Jan 7 09:06:36 2017
N/A - Inverted context relationship attributes 0 Sat Jan 7 09:06:43 2017
Multiple C-level relationships - DONE, OTHER

select r.to_id conceptId1, r.from_id conceptId2
from concept_relationships r
where terminology = :terminology
group by r.to_id, r.from_id having count(*)>1;


N/A - Non-UTF8 attributes rows 0 Sat Jan 7 09:07:44 2017
N/A - Non-UTF8 string_ui rows 0 Sat Jan 7 09:07:47 2017
N/A - Non-UTF8 stringtab rows 0 Sat Jan 7 09:07:55 2017
N/A - Preferred atoms without contexts 0 Sat Jan 7 09:08:14 2017
N/A - Releasable QA concepts 0 Sat Jan 7 09:08:14 2017
N/A - Releasable old version rels 0 Sat Jan 7 09:08:14 2017
N/A - Releasable old version foreign sources 0 Sat Jan 7 09:08:14 2017
N/A - S level atts with aui,sg_id out of sync 0 Sat Jan 7 09:09:31 2017
N/A - S level rels with aui,sg_id out of sync 0 Sat Jan 7 09:11:30 2017\
N/A - SUBSET_MEMBER atts without corresponding SB atom 0 Sat Jan 7 09:11:49 2017

Self-referential C level relationships 0 Sat Jan 7 09:11:51 2017 - DONE

select r.from_id conceptId
from concept_relationships r
where terminology = :terminology
  and from_id = to_id;


N/A - Strings with junk characters 0 Sat Jan 7 09:11:55 2017
N/A - Termgroup (termgroup_rank) does not match tty 0 Sat Jan 7 09:11:55 2017
N/A - Termgroup does not match tty 0 Sat Jan 7 09:11:57 2017


Unexpected source with suppress = E -1* Sat Jan 7 09:12:04 2017 - DONE, OTHER

select distinct a.terminology
from atoms a, term_types b
where a.termType = b.abbreviation
  and a.suppressible = true and b.suppressible = false
  and a.terminology not in ('CSP','CST','DXP','FMA','MTH','NCBI','RXNORM','UWDA'); 


Unexpected source with suppress = O 0* Sat Jan 7 09:12:09 2017 - DONE, OTHER

select distinct a.terminology
from atoms a
where a.obsolete = true
  and a.terminology not in ('CVX', 'FMA', 'GO','HCPCS','HPO','HL7V3.0','LNC','MDR','MGED','MVX','MTHSPL','NCI','NICHD','PDQ','RXNORM','VANDF'); 


Checks To Implement

Checking new src integrities.........Sat Jan 7 09:19:58 2017

Bad termgroup in versioned SRC concept - (Implemented on MEME-Edit 04/08/2022)

select distinct c1.id conceptId
from atoms a1, concepts_atoms b1, concepts c1, 
     atoms a2, concepts_atoms b2, concepts c2
where a1.id = b1.atoms_id and b1.concepts_id = c1.id
  and a2.id = b2.atoms_id and b2.concepts_id = c2.id
  and c1.terminology = :terminology
  and c2.terminology = :terminology
  and c1.id = c2.id
  and a1.publishable = true
  and a1.terminology = 'SRC' and a1.termType = 'VPT'
  and a2.publishable = true
  and a2.terminology = 'SRC' AND a2.termType not in ('VAB','VPT','VSY');


Code does not match SRC/RAB name  - (Implemented on MEME-Edit 04/08/2022)

select c1.id conceptId
from atoms a1, concepts_atoms b1, concepts c1, 
     atoms a2, concepts_atoms b2, concepts c2
where a1.id = b1.atoms_id and b1.concepts_id = c1.id
  and a2.id = b2.atoms_id and b2.concepts_id = c2.id
  and c1.terminology = :terminology
  and c2.terminology = :terminology
  and c1.id = c2.id
  and a1.terminology = 'SRC'
  and a1.termType = 'RAB'
  and a2.terminology = 'SRC'
  and a2.code not like concat('V-',a1.codeId);


Code does not match SRC/VAB name 28 Sat Jan 7 09:20:00 2017 (OUTPUT CONTAINS COMPLICATIONS)  SEE ATTACHMENT HERE

select c1.id conceptId
from atoms a1, concepts_atoms b1, concepts c1, 
     atoms a2, concepts_atoms b2, concepts c2
where a1.id = b1.atoms_id and b1.concepts_id = c1.id
  and a2.id = b2.atoms_id and b2.concepts_id = c2.id
  and c1.terminology = :terminology
  and c2.terminology = :terminology
  and c1.id = c2.id
  and a1.terminology = 'SRC'
  and a1.termType = 'VAB'
  and a2.terminology = 'SRC'
  and a2.code not like concat('V-',a1.codeId);


Concepts with duplicate atoms (Implemented on MEME-Edit 05/11/2022)

select concepts_id as conceptId from concepts_atoms group by conceptId, atoms_id having count(*) >1;


Non-code-based SRC version_of rels : where terminology='SRC' and additionalRelationshipType='version_of' and publishable=true) (Implemented on MEME-Edit 07/11/2022)
(atom_relationship and component_info_relationship versions not implemented)

SELECT from_id AS conceptId1, to_id AS conceptId2
FROM concept_relationships
WHERE terminology = 'SRC'AND additionalRelationshipType = 'version_of'AND publishable = TRUE;

SELECT c1.id as conceptId1, c2.id as conceptId2 
FROM concepts c1, concepts c2, descriptor_relationships b, descriptors_atoms da1, descriptors_atoms da2, concepts_atoms ca1, concepts_atoms ca2 
WHERE b.from_id = da1.descriptors_id AND b.to_id = da2.descriptors_id 
AND da1.atoms_id = ca1.atoms_id AND da2.atoms_id = ca2.atoms_id 
AND c1.id = ca1.concepts_id AND c2.id = ca2.concepts_id 
AND c1.terminology = 'NCIMTH' AND c2.terminology = 'NCIMTH' AND
b.terminology='SRC' AND b.additionalRelationshipType='version_of' AND b.publishable=true;


#ATOM RELATIONSHIPS ARE ALLOWED TO HAVE THIS CONDITION. QUERY NOT IMPLEMENTED

SELECT ca1.concepts_id as conceptId1, ca2.concepts_id as conceptId2 
FROM atom_relationships ar,concepts_atoms ca1, concepts_atoms ca2
WHERE ar.from_id = ca1.atoms_id
AND ar.to_id = ca2.atoms_id 
AND ar.terminology='SRC'
AND ar.additionalRelationshipType = 'version_of' 
AND ar.publishable=true;

N/A - Invalid type/qualifier for SRC translation_of rels 0 Sat Jan 7 09:21:04 2017

N/A - Current source without citation 0 Sat Jan 7 09:20:00 2017
Current sources without releasable SRC/VABs

select terminology
from terminologies t where current = true
and not exists
 (select * from atoms a
  where a.terminology = 'SRC' and a.publishable = true
    and a.termType = 'VAB' and a.codeId = concat('V-',t.terminology));


N/A - HCPT has CPT source family 0 Sat Jan 7 09:20:00 2017
Hierarchy treetop not in same concept as RHT 0 Sat Jan 7 09:20:48 2017

how to find treetop?



N/A - Missing sims_info data 0 Sat Jan 7 09:21:18 2017
N/A - Non-ENG root SRC without translation_of rel 0 Sat Jan 7 09:21:19 2017
Non-SRC version_of rels 0 Sat Jan 7 09:21:27 2017  (Implemented on MEME-Edit 08/04/2022)

select id from code_relationships where terminology!='SRC' and additionalRelationshipType='version_of' and publishable=true;


Obsolete source w/o bequeathal rel to root 1 Sat Jan 7 09:21:27 2017  (Implemented on MEME-Edit 08/04/2022)

SELECT c_a.concepts_id FROM atoms a, concepts_atoms c_a
WHERE a.publishable = false
    AND a.terminology = 'SRC' AND a.termType = 'VAB' 
    AND a.id = c_a.atoms_id
    AND SUBSTRING(a.codeId,3) IN
    (SELECT concat(terminology,'_',version)
     FROM terminologies t WHERE current = false)
AND NOT EXISTS
(SELECT * FROM concept_relationships r, concepts_atoms c_a2, atoms a2
 WHERE a2.publishable=true
 AND a2.terminology='SRC' AND a2.termType='RAB' 
 AND a2.id = c_a2.atoms_id
 AND c_a.concepts_id = r.from_id
 AND c_a2.concepts_id = r.to_id
 AND r.relationshipType='BRO'
);


Obsolete sources with releasable SRC/VABs 16 Sat Jan 7 09:21:28 2017 (Implemented on MEME-Edit 08/04/2022)

SELECT c_a.concepts_id, SUBSTRING(a.codeId,3) obsolete_source FROM atoms a, concepts_atoms c_a
WHERE a.publishable = true
    AND a.terminology = 'SRC' AND a.termType = 'VAB'
    AND a.id = c_a.atoms_id
    AND SUBSTRING(a.codeId,3) IN
    (SELECT concat(terminology,'_',version)
     FROM terminologies t WHERE current = false);


Official Name does not match SRC/VPT 0 Sat Jan 7 09:21:28 2017 (Implemented on MEME-Edit 08/03/2022)

--official name is terminology.preferredName
SELECT c_a.concepts_id, a.name
FROM atoms a, concepts_atoms c_a
WHERE a.publishable=true
AND a.terminology='SRC' AND a.termType='VPT' 
AND a.id = c_a.atoms_id
AND NOT EXISTS
    (SELECT * from terminologies t
     where t.preferredName=a.name);


N/A - Releasable SRC/RHT not used as tree-top 0* Sat Jan 7 09:22:10 2017


Root SRC exists for terminology without any atoms 0* Sat Jan 7 09:22:20 2017 (Implemented on MEME-Edit 08/04/2022)

SELECT * FROM atoms a
WHERE a.terminology='SRC' AND a.termType='RPT' AND a.publishable=true
AND NOT EXISTS
   (SELECT * FROM atoms b
    WHERE b.terminology=SUBSTRING(a.codeId,3) AND b.publishable=true);


SRC concept no Intellectual Product STY 0 Sat Jan 7 09:22:25 2017 (Implemented on MEME-Edit 08/04/2022)

SELECT c.id FROM atoms a, concepts_atoms c_a, concepts c
WHERE a.id = c_a.atoms_id AND c_a.concepts_id = c.id
AND a.terminology='SRC' AND a.publishable=true
AND NOT EXISTS
    (select sty.* from concepts_semantic_type_components cstc, semantic_type_components sty
     where c.id = cstc.concepts_id and cstc.semanticTypes_id = sty.id
     and sty.semanticType='Intellectual Product');


SRC rels must be owned by SRC 0 Sat Jan 7 09:22:40 2017 

atom/code relationships connected to SRC atoms or codes must be owned by SRC.

SRC/RAB missing from root_terminologies 0 Sat Jan 7 09:22:40 2017
SRC/RHT without contexts 0 Sat Jan 7 09:23:06 2017

no atom_tree_positions or concept_tree_positions or code/descriptor tree positions

SRC/VAB missing from terminologies 31 Sat Jan 7 09:23:06 2017
Source family is not RSAB of current source 0 Sat Jan 7 09:23:06 2017

there is a source family value in root_terminologies that does not match the abbreviation of a root terminology that has a current terminology version


N/A - Source version not set 0 Sat Jan 7 09:23:06 2017
Source with contexts and bad SRC/RHT 0 Sat Jan 7 09:23:06 2017
Source without exactly one SRC/RAB 0 Sat Jan 7 09:23:06 2017
Source without exactly one SRC/RPT 0 Sat Jan 7 09:23:06 2017
Source without exactly one SRC/SSN 0 Sat Jan 7 09:23:06 2017
Source without exactly one SRC/VAB 27* Sat Jan 7 09:23:06 2017
Source without exactly one SRC/VPT 0* Sat Jan 7 09:23:06 2017
Source with blank or null language/character_set/source_family 0* Sat Jan 7 09:23:06 2017
Source without name/short name 1 Sat Jan 7 09:23:06 2017
N/A - Verify TTY order has not changed 0 Sat Jan 7 09:23:06 2017
Versioned SRC without version_of rel 1 Sat Jan 7 09:23:06 2017
N/A - XM atoms in NCI Meta -1* Sat Jan 7 09:23:12 2017
N/A - source_rank.stripped_source minus source_version 0 Sat Jan 7 09:23:12 2017
N/A - source_version minus SRC/RAB 0 Sat Jan 7 09:23:12 2017
N/A - source_version.current_name minus source_rank 0 Sat Jan 7 09:23:12 2017
N/A - source_version.source minus source_rank 0 Sat Jan 7 09:23:12 2017
N/A - ~ MRDOC.EXPL with more than 1 value 8* Sat Jan 7 09:23:12 2017



Unimplemented Checks




Unexpected source with suppressible atts 0* Sat Jan 7 09:12:43 2017  (Implemented on MEME-Edit 10/18/2022)

select distinct a.id conceptId
from attributes a
where suppressible = true
and terminology not in ('VANDF','MTHSPL');


N/A - Unexpected source with suppressible cxt rels 0 Sat Jan 7 09:13:08 2017


Unexpected source with suppressible relationships 0 Sat Jan 7 09:13:14 2017  (Implemented on MEME-Edit 10/18/2022)

select a.from_id as conceptId from atom_relationships a
where a.suppressible = true
union
select a.from_id from concept_relationships a
where a.suppressible = true
  and a.terminology != 'MTH'
union
select a.from_id from code_relationships a
where a.suppressible = true
union
select a.from_id from descriptor_relationships a
where a.suppressible = true;

N/A - Unreleasable current version rels 2924 Sat Jan 7 09:13:16 2017
N/A - Unreleasable parent in current source 0 Sat Jan 7 09:13:32 2017
Update sources with only zero last release rank 0 Sat Jan 7 09:13:48 2017 (NOT DOING)

select terminology,version from atoms
where (terminology,version) in (select terminology,version from terminologies where current = false)
group by terminology having max(lastPublishedRank) = '0'; 

Updated source now with CODE=NOCODE 0 Sat Jan 7 09:14:06 2017  (Implemented on MEME-Edit 11/09/2022)

select distinct terminology as conceptId
from atoms
where codeId = 'NOCODE' 
and terminology not in 
('COSTAR','DXP','MCM','MTH','MTHCMSFRF','MTHMST','MTHSPL','NCIMTH');

#Original Query Below
/* select distinct terminology,version
from atoms 
where (terminology,version) in (select terminology,version from terminologies where current = false)
 and codeId = 'NOCODE'; */

N/A - Uppercase words 0 Sat Jan 7 09:14:07 2017

Validate CUI format 0 Sat Jan 7 09:14:15 2017  (Implemented on MEME-Edit 11/14/2022)

select id conceptId
from concepts
where terminology = :terminology
  and terminologyId != id
  and terminologyId not like 'C_______' and terminologyId not like 'CL_______';


N/A - ^Rels represented in both directions 91076* Sat Jan 7 09:15:35 2017
N/A - bad ATOM_NOTE attributes 0 Sat Jan 7 09:15:35 2017
N/A - 
bad CONCEPT_NOTE attributes 0 Sat Jan 7 09:15:35 2017
N/A - bad CONTEXT attributes 0 Sat Jan 7 09:15:47 2017
N/A - bad DEFINITION attributes 0 Sat Jan 7 09:19:03 2017
N/A - bad LEXICAL_TAG (TRD) attributes 0 Sat Jan 7 09:19:05 2017
N/A - bad LEXICAL_TAG (non-TRD) attributes 0 Sat Jan 7 09:19:06 2017
N/A - bad NON_HUMAN attributes 0 Sat Jan 7 09:19:06 2017
N/A - bad SEMANTIC_TYPE attributes 0 Sat Jan 7 09:19:18 2017
N/A - bad SFO/LFO relationships 0 Sat Jan 7 09:19:20 2017
N/A - bad SYNTACTIC_CATEGORY attributes 0 Sat Jan 7 09:19:20 2017
N/A - bad source level attributes 0 Sat Jan 7 09:19:32 2017
N/A - obsolete sources with releasable atoms 0 Sat Jan 7 09:19:41 2017

Incorrect publishable for concept level relationships 

select a.from_id conceptId1, a.to_id conceptId2
from concept_relationships a
where terminology = :terminology
  and ((relationshipType in ('XR','BRB','BRN','BRO') and publishable = true) OR 
       (relationshipType not in ('XR','BRB','BRN','BRO') AND publishable  = false));


N/A - ~ ATX mappings connected to bad CUIs 0 Sat Jan 7 09:19:43 2017
N/A - ~ Mixed status concepts 196 Sat Jan 7 09:19:47 2017
N/A - ~ Suppressible preferred name (level 0,9) 0 Sat Jan 7 09:19:58 2017
N/A - ~Required Editing Bins 0 Sat Jan 7 09:19:58 2017



Checking other referential integrities.........Sat Jan 7 09:23:12 2017

Metadata referential integrity (atom.tty, attribute.name, relationship.rel/rela, etc.  atom.terminology/tty → precedence list - default or project, stycomponent->sty) 

(Implemented on MEME-Edit 11/17/2022)

-- atom.tty
select termType from atoms a where not exists (select * from term_types b where a.termType = b.abbreviation)
and a.publishable = true;

-- attribute.name
select name from attributes a where not exists (select * from attribute_names b where a.name = b.abbreviation)
and a.publishable = true;

-- relationship.rel
select relationshipType from concept_relationships a 
where not exists (select * from relationship_types b where a.relationshipType= b.abbreviation) 
and a.publishable = true
union
select relationshipType from atom_relationships a 
where not exists (select * from relationship_types b where a.relationshipType= b.abbreviation) 
and a.publishable = true
union
select relationshipType from code_relationships a 
where not exists (select * from relationship_types b where a.relationshipType= b.abbreviation) 
and a.publishable = true
union
select relationshipType from descriptor_relationships a 
where not exists (select * from relationship_types b where a.relationshipType= b.abbreviation) 
and a.publishable = true;

-- relationship.rela
-- same as above but "additionalRelationshipType" instead of "relationshipType"
-- also include "atom_tree_positions", "concept_tree_positions", ...

-- atom.terminology,tty
select terminology, termType from atoms a 
where not exists 
  (select * from precedence_lists b, precedence_list_terminologies c, precedence_list_termtypes d
   where b.id = c.PrecedenceListJpa_id and b.id = d.PrecedenceListJpa_id
     and b.terminology = 'NCIMTH'
     and a.terminology = c.terminologies and a.termType = d.termTypes
     and c.terminologies_ORDER = d.termTypes_ORDER)
and a.publishable = true;

  • terminology fields for all objects  OR make a foreign key constraint from each of the tables to "terminologies.terminology" (Implemented on MEME-Edit 11/14/2022)
select terminology from
(
select terminology from atoms where publishable = true
union
select terminology from codes where publishable = true
union
select terminology from descriptors where publishable = true
union
select terminology from concepts where publishable = true
union
select terminology from atom_relationships where publishable = true
union
select terminology from concept_relationships where publishable = true
union
select terminology from code_relationships where publishable = true
union
select terminology from descriptor_relationships where publishable = true
union
select terminology from atom_subset_members where publishable = true
union
select terminology from atom_subsets where publishable = true
union
select terminology from atom_transitive_rels where publishable = true
union
select terminology from atom_tree_positions where publishable = true
union
select terminology from attribute_identity
union
select terminology from attribute_names where publishable = true
union
select terminology from attributes where publishable = true
union
select terminology from code_transitive_rels where publishable = true
union
select terminology from code_tree_positions where publishable = true
union
select terminology from component_histories where publishable = true
union
select terminology from component_info_relationships where publishable = true
union
select terminology from concept_subset_members where publishable = true
union
select terminology from concept_subsets where publishable = true
union
select terminology from concept_transitive_rels where publishable = true
union
select terminology from concept_tree_positions where publishable = true
union
select terminology from deep_atom_relationships where publishable = true
union
select terminology from deep_concept_relationships where publishable = true
union
select terminology from definitions where publishable = true
union
select terminology from descriptor_tree_positions where publishable = true
union
select terminology from general_concept_axioms where publishable = true
union
select terminology from general_metadata_entries where publishable = true
union
select terminology from label_sets where publishable = true
union
select terminology from languages where publishable = true
union
select terminology from lexical_classes where publishable = true
union
select terminology from mappings where publishable = true
union
select terminology from mapsets where publishable = true
union
select terminology from relationship_types where publishable = true
union
select terminology from semantic_type_components where publishable = true
union
select terminology from semantic_types where publishable = true
union
select terminology from string_classes where publishable = true
union
select terminology from term_types where publishable = true
) a
where  not exists (select * from terminologies b where a.terminology = b.terminology);


Invalid language in atoms 0 Sat Jan 7 09:27:07 2017 (Implemented on MEME-Edit 11/14/2022)

select id from atoms where language != 'ENG';


Invalid semantic types (A) 0 Sat Jan 7 09:28:13 2017 (Implemented on MEME-Edit 11/14/2022)

select id,semanticType from semantic_type_components 
where semanticType not in
(select expandedForm from semantic_types);

N/A - Max CL CUI should match value in max_tab 0 Sat Jan 7 09:29:08 2017  - n/a

Mismatched inverse_rel_attributes 0 Sat Jan 7 09:29:08 2017

additional_relationship_types.abbreviation - additional_relationship_types.inverse (Implemented on MEME-Edit 11/14/2022)

select abbreviation, inverse_id from additional_relationship_types where inverse_id is null
union
select b.abbreviation, b.id from additional_relationship_types a, additional_relationship_types b, additional_relationship_types c
where a.inverse_id = b.id and b.inverse_id = c.id and a.id != c.id and a.publishable = true and c.publishable = true;


Mismatched inverse_relationships 0 Sat Jan 7 09:29:08 2017 (Implemented on MEME-Edit 11/14/2022)

select abbreviation, inverse_id from relationship_types where inverse_id is null
union
select b.abbreviation, b.id from relationship_types a, relationship_types b, relationship_types c
where a.inverse_id = b.id and b.inverse_id = c.id and a.id != c.id and a.publishable = true and c.publishable = true;

N/A relationship_types.abbreviation -  relationship_types.inverse
Missing ancestor node 0 Sat Jan 7 09:32:12 2017

(Brian is trying to optimize this one)

-- play around with "1~2~3~4" and try to get '4' and '1~2~3'

select a.ancestorPath
from atom_tree_positions a, atoms where a.node_id = atoms.id and not exists
  (select * from atom_tree_positions b where SUBSTRING_INDEX(a.ancestorPath, '~', -1) = atoms.terminologyId
   and a.ancestorPath like '%~%'
   and LEFT(a.ancestorPath, LENGTH(a.ancestorPath) - LENGTH(SUBSTRING_INDEX(a.ancestorPath, '~', -1)) - 1) = b.ancestorPath
   and a.terminology = b.terminology
   and a.additionalRelationshipType = b.additionalRelationshipType)

-- repeat for concept_tree_positions, code_tree_positions, descriptor_tree_positions


N/A - handled above - Rows in classes with invalid sources 0 Sat Jan 7 09:35:10 2017

N/A - handled above Rows in classes with invalid termgroups 0 Sat Jan 7 09:35:18 2017

N/A - handled above - attributes with invalid source 0 Sat Jan 7 09:35:30 2017
N/A - handled above - relationships with invalid source 0 Sat Jan 7 09:35:30 2017
N/A - handled above - attribute wit invalid atn 0* Sat Jan 7 09:35:43 2017
N/A - handled above - atoms with invalid tty 0 Sat Jan 7 09:35:52 2017

N/A - handled above - relationsips with invalid rel/rela 0* Sat Jan 7 09:36:18 2017
N/A - handled above - atoms with invalid language 0 Sat Jan 7 09:36:18 2017

Checking source specific integrities.........Sat Jan 7 09:36:22 2017

TEST RESULTS
------------------------------------------------------ ----------
N/A - Check for GFG edits 0 Sat Jan 7 09:36:23 2017
N/A - Context TTYs check 0 Sat Jan 7 09:39:02 2017
Duplicate MTH CUI relationships 1166 Sat Aug  5 20:49:02 2017  (Seems like Multiple C-Level Relationships)

-- NOTE: we may already have a check like this, so it could be redundant
--   check first
select c1.id as conceptId1, c2.id as conceptId2
from concept_relationships cr, concepts c1, concepts c2
where cr.from_id = c1.id and cr.to_id = c2.id
AND from_id<to_id
and cr.terminology='NCIMTH'
and c1.terminology='NCIMTH' and c2.terminology='NCIMTH'
and cr.publishable = true and c1.publishable = true and c2.publishable = true
GROUP BY cui1,cui2 HAVING COUNT(*)>1;


N/A - MSH foreign split MUIs 0 Sat Jan 7 09:39:02 2017
N/A - already checked - MTH self-referential relationships 3752 Sat Jan 7 09:39:04 2017 
MTH-only concepts 0 Sat Jan 7 09:39:21 2017 (Implemented on MEME-Edit 12/02/2022)

-- NOTE: this may already exist, check first  (OR see the fix for mth-only concepts, it likely has a query to find this)
-- see query action process -> turn of MTH/PN-only concepts
select c1.id as conceptId
from concepts c, concepts_atoms ca, atoms a
where c.id = ca.concepts_id and ca.atoms_id = a.id
and c.publishable = true
and a.publishable = true
and (a.terminology = 'MTH' OR a.terminology = 'NCIMTH')
and not exists
  (select * from concepts_atoms ca2, atoms a2
   where ca2.atoms_id = a2.id
     and a2.publishable = true
     and ca1.concepts_id = ca2.concepts_id
     and a.terminology != 'MTH' and a.terminology != 'NCIMTH');


N/A - NOCODE MTH/{PT,MM} atoms 0 Sat Jan 7 09:39:21 2017

Duplicate RUI 0 Sat Jan 7 09:39:24 2017
Duplicate AUI
Duplicate ATUI
Duplicate SUI
Duplicate LUI

-- "duplicate" RUI
select id from relationship_identity
group by other fields,... having count(*)>1;

-- duplicate ATUI
... same thing from attribute_identity ...

-- same for AUI - atom_identity
-- same for LUI - lexical_class_identity
-- same for SUI - string_class_identity


-- NOT for CUI


N/A - SIB rel with non null RELA (not UWDA) 0 Sat Jan 7 09:39:45 2017
N/A - Unreleasable MTH relationships 0 Sat Jan 7 09:39:47 2017
N/A - ~ MTH rels to split CUIs 43861 Sat Jan 7 09:40:34 2017
N/A - ~ concepts with multiple MTH/PN atoms 0 Sat Jan 7 09:40:36 2017

Checking string unique identifier integrity.........Sat Jan 7 09:40:36 2017

TEST RESULTS
------------------------------------------------------ ----------

use string_class_identity and lexical_class_identity

N/A - (norm_string,language)-LUI mismatch 0 Sat Jan 7 09:41:14 2017 (lexical_class_identity, atoms)
N/A - ENG strings with foreign language SUIs 0 Sat Jan 7 09:41:29 2017
N/A - Erroneous norm string 0 Sat Jan 7 09:41:40 2017
N/A - LUI-norm_string mismatch 0 Sat Jan 7 09:42:11 2017

Mismatches in string_ui,classes,atoms 0 Sat Jan 7 09:42:59 2017 (Implemented on MEME-Edit 11/18/2022)

-- NOTE: if the atoms stringClassId starts with an S you have to remove it
select a.stringClassId, a.name atoms_name, s_c_i.name sci_name
from atoms a, string_class_identity s_c_i
where (substr(a.stringClassId,2))+0) = s_c_i.id and a.name != s_c_i.name;

Mismatches between atoms and string_class_identity, lexical_class_identity (Implemented on MEME-Edit 11/18/2022)

select id from atoms where terminology != 'SRC'
and (name,(substr(stringClassId,2))+0) not in
    (select name,id from string_class_identity);

select id from atoms where terminology != 'SRC'
and (substr(lexicalClassId,2)+0) not in
    (select id from lexical_class_identity);

N/A - handled by above - Mismatches in string_ui,classes,normstr 0 Sat Jan 7 09:43:43 2017
Mismatches in string_ui,d_classes,d_atoms 0 Sat Jan 7 09:43:43 2017
Non-unique SUI in string_ui 0 Sat Jan 7 09:43:58 2017
Non-unique string,language in string_ui 0 Sat Jan 7 09:44:25 2017
Rows in classes minus string_ui (SUI) 0 Sat Jan 7 09:45:00 2017

Rows in classes minus string_ui(LUI) 0 Sat Jan 7 09:45:16 2017 (Already Implemented)

select id from atoms a where not exists (select * from string_class_identity b where b.id = (substr(a.stringClassId,2))+0) );
select id from atoms a where not exists (select * from lexical_class_identity b where b.id = (substr(a.lexicalClassId,2))+0) );


N/A - Rows in dead_classes minus string_ui (LUI) 0 Sat Jan 7 09:45:21 2017
N/A - Rows in dead_classes minus string_ui(SUI) 0 Sat Jan 7 09:45:50 2017
N/A - Rows in foreign_classes minus string_ui (LUI,lang) 0 Sat Jan 7 09:46:16 2017
N/A - Rows in foreign_classes minus string_ui (SUI,lang) 0 Sat Jan 7 09:46:49 2017
N/A - Rows w/ non-null LUI & null norm_string 0 Sat Jan 7 09:46:49 2017
N/A - Rows w/ null LUI & non-null norm_string 0 Sat Jan 7 09:46:49 2017

leading spaces 0 Sat Jan 7 09:46:57 2017 (Whitespace already implemented.  Special chars implemented on MEME-Edit 12/07/2022)

-- leading, trailing or duplicate whitespace in names (if there are duplicate whitespace things, just remove that part)
select id from atoms where (name like ' %' OR name like '% ' OR name like '%  %') and publishable = true; 

-- check for \t, \r, or \n (look up ascii numbers for these 3 chars)
SELECT * FROM atoms WHERE INSTR(name, char(10)) > 0 and publishable = true
union
SELECT * FROM atoms WHERE INSTR(name, char(09)) > 0 and publishable = true
union
SELECT * FROM atoms WHERE INSTR(name, char(13)) > 0 and publishable = true;



Checking unique identifier (ui) integrity.........Sat Jan 7 09:46:57 2017

TEST RESULTS
------------------------------------------------------ ----------
ATUI with bad source 0 Sat Jan 7 09:47:18 2017

TODO: edit the large query that checks terminology to include "atom_identity" and "relationship_identity"

n/a- AUI with bad root source 0 Sat Jan 7 09:47:21 2017
done - Bad ATUI (C rels - attributes_ui) 0 Sat Jan 7 09:48:35 2017 
n/a Bad ATUI (S rels - attributes_ui) 0 Sat Jan 7 09:48:40 2017

Bad AUI (classes-atoms_ui) 0 Sat Jan 7 09:49:40 2017 (Implemented on MEME-Edit 12/07/2022)

select * from atomjpa_alternateterminologyids where alternateTerminologyIds_KEY = 'NCIMTH'
  AND not exists (select * from atom_identity b where b.id = (substr(a.alternateTerminologyIds,2)+0) );


Bad RUI (C rels - relationships_ui) 0 Sat Jan 7 09:50:14 2017 (Implemented on MEME-Edit 12/07/2022)

select * from atomrelationshipjpa_alternateterminologyids a where alternateTerminologyIds_KEY = 'NCIMTH'
  AND not exists (select * from relationship_identity b where b.id = (substr(a.alternateTerminologyIds,2)+0) )
UNION ALL
select * from coderelationshipjpa_alternateterminologyids a where alternateTerminologyIds_KEY = 'NCIMTH'
  AND not exists (select * from relationship_identity b where b.id = (substr(a.alternateTerminologyIds,2)+0) )
UNION ALL
select * from componentinforelationshipjpa_alternateterminologyids a where alternateTerminologyIds_KEY = 'NCIMTH'
  AND not exists (select * from relationship_identity b where b.id = (substr(a.alternateTerminologyIds,2)+0) )
UNION ALL
select * from conceptrelationshipjpa_alternateterminologyids a where alternateTerminologyIds_KEY = 'NCIMTH'
  AND not exists (select * from relationship_identity b where b.id = (substr(a.alternateTerminologyIds,2)+0) )
UNION ALL
select * from descriptorrelationshipjpa_alternateterminologyids a where alternateTerminologyIds_KEY = 'NCIMTH'
  AND not exists (select * from relationship_identity b where b.id = (substr(a.alternateTerminologyIds,2)+0) )


N/A - Bad RUI (NLM03 rels - relationships_ui) 0 Sat Jan 7 09:50:50 2017
N/A - Bad RUI (S rels - relationships_ui) 0 Sat Jan 7 09:54:43 2017
N/A - Bad RUI (context rels - relationships_ui) 0 Sat Jan 7 09:55:24 2017
N/A - Duplicate ATUI (attributes_ui) 0 Sat Jan 7 09:57:19 2017
N/A - Duplicate ATUI values (attributes_ui) 0 Sat Jan 7 09:57:53 2017
N/A - Duplicate AUI values 0 Sat Jan 7 09:58:30 2017
N/A - Duplicate RUI (inverse_relationships_ui) 0 Sat Jan 7 10:01:33 2017
N/A - Duplicate RUI (relationships_ui) 0 Sat Jan 7 10:03:47 2017
N/A - Duplicate RUI values (relationships_ui) 67 Sat Jan 7 10:04:26 2017
N/A - Duplicate atom_id's (C-dead_C) 0 Sat Jan 7 10:04:31 2017
N/A - Duplicate atom_id's (atoms-dead_atoms) 0 Sat Jan 7 10:04:40 2017
N/A - Duplicate atomic_action_id's 0 Sat Jan 7 10:04:41 2017
N/A - Duplicate attribute_id's (A-dead_A) 0 Sat Jan 7 10:04:55 2017
N/A - Duplicate concept_id's (CS-dead_CS) 0 Sat Jan 7 10:05:00 2017
Duplicate ids in source_id_map (C) 0 Sat Jan 7 10:05:11 2017 - alternate terminology ids

select alternateTerminologyIds from atomjpa_alternateterminologyids where alternateTerminologyIds_KEY = 'NCIMTH-SRC' group by alternateTerminologyIds having count(*)>1;

-- if above has exceptions try
select alternateTerminologyIds from atomjpa_alternateterminologyids where alternateTerminologyIds_KEY = 'NCIMTH-SRC' group by alternateTerminologyIds having count(distinct AtomJpa_id)>1;


Duplicate inverse RUI 0 Sat Jan 7 10:05:29 2017

-- ADD a unique index to relationship_identity inverseId

select * from atomrelationshipjpa_alternatetermiologyids a, atomrelationshipjpa_alternatetermiologyids b, relationship_identity c
where a.alternateTerminologyIds_KEY  = 'NCIMTH' and b.alternateTerminologyIds_KEY  = 'NCIMTH'
  and (substr(a.alternateTerminologyIds,2))+0) ) = c.id
  and (substr(b.alternateTerminologyIds,2))+0) ) = c.inverseId
group by b.alternateTerminologyIds having count(distinct a.alternateTerminologyIds)>1;

... do this also for the other relationship alternate id tables.


N/A - Duplicate molecule_id's 0 Sat Jan 7 10:05:29 2017
N/A - Duplicate relationship_id's (CR-dead_CR) 0 Sat Jan 7 10:06:00 2017
N/A - Duplicate relationship_id's (R-dead_R) 0 Sat Jan 7 10:06:11 2017
N/A - Duplicate string_id's (stringtab-dead_stringtab) 0 Sat Jan 7 10:06:21 2017
N/A - Duplicate string_ids (A) 0 Sat Jan 7 10:07:01 2017

Inverse RUI check 237 Sat Jan 7 10:07:36 2017

-- ADD a unique index to relationship_identity inverseId

select * from atomrelationshipjpa_alternatetermiologyids a
where a.alternateTerminologyIds_KEY  = 'NCIMTH'
  and not exists
  (select * from atomrelationshipjpa_alternatetermiologyids b, relationship_identity c
   where b.alternateTerminologyIds_KEY  = 'NCIMTH'
     and (substr(a.alternateTerminologyIds,2))+0) ) = c.id
     and (substr(b.alternateTerminologyIds,2))+0) ) = c.inverseId);

... do this also for the other relationship alternate id tables.

N/A - Non-unique ATUI 0 Sat Jan 7 10:08:02 2017
N/A - Non-unique AUI 0 Sat Jan 7 10:08:38 2017
N/A - Non-unique AUI (foreign) 0 Sat Jan 7 10:08:38 2017
N/A - Non-unique RUI (CR) 0 Sat Jan 7 10:10:04 2017
N/A - Non-unique RUI (R union inverse R) 0 Sat Jan 7 10:11:38 2017
N/A - Non-unique RUI (R with relationship_group) 0 Sat Jan 7 10:14:06 2017
N/A - Non-unique RUI (R) 0 Sat Jan 7 10:14:12 2017
N/A - Non-unique RUI (R) - selfref 0 Sat Jan 7 10:14:14 2017
N/A - Overlapping relationship_id's (R-CR) 0 Sat Jan 7 10:14:24 2017
N/A - Overlapping relationship_id's (R-dead_CR) 0 Sat Jan 7 10:14:24 2017
N/A - Overlapping relationship_id's (dead_R-CR) 0 Sat Jan 7 10:14:26 2017
N/A - RUI with bad source 0 Sat Jan 7 10:14:49 2017


PICKUP HERE


Releasable atom with null AUI 0 Sat Jan 7 10:14:56 2017

atoms entry that doesn't have an alternateTerminologyIds entry for 'NCIMTH'

N/A - Releasable atom with null AUI (foreign) 0 Sat Jan 7 10:14:56 2017
Releasable attribute with null ATUI 0 Sat Jan 7 10:15:08 2017
Releasable rel with null RUI 0 Sat Jan 7 10:15:17 2017
Source rui without rel directionality 0 Sat Jan 7 10:15:26 2017
atom_id out of range (C-dead_C) 0 Sat Jan 7 10:15:27 2017
atom_id out of range (atoms-dead_atoms) 0 Sat Jan 7 10:15:27 2017
atomic_action_id out of range(atomic_actions) 0 Sat Jan 7 10:15:28 2017
attribute_id out of range (A-dead_A) 0 Sat Jan 7 10:15:40 2017
aui out of range 0 Sat Jan 7 10:15:48 2017
concept_id out of range (CS-dead_CS) 0 Sat Jan 7 10:15:49 2017
last_release_cui out of range (C) 0 Sat Jan 7 10:15:49 2017
molecule_id out of range (molecular_actions) 0 Sat Jan 7 10:15:49 2017
relationship_id out of range (CR-dead_CR) 0 Sat Jan 7 10:15:51 2017
relationship_id out of range (R-dead_R) 0 Sat Jan 7 10:15:51 2017
string_id out of range (A-dead_A) 0 Sat Jan 7 10:16:01 2017
string_id out of range ({dead_,}stringtab) 0 Sat Jan 7 10:16:03 2017
~ AUI split across CUI 0 Sat Jan 7 10:16:29 2017
~ Non-unique AUI (NLM02) 0 Sat Jan 7 10:16:29 2017
~ATUI with multiple attribute_ids 0 Sat Jan 7 10:17:42 2017
~AUI with multiple atom_ids 50 Sat Jan 7 10:17:55 2017
~RUI with multiple relationship_ids 0 Sat Jan 7 10:18:12 2017

Checking unique identifier (ui) referential integrity.........Sat Jan 7 10:18:12 2017

TEST RESULTS
------------------------------------------------------ ----------
C level attributes minus classes 0 Sat Jan 7 10:18:53 2017
C level relationships minus classes 0 Sat Jan 7 10:19:11 2017
Rows in CR minus classes 0 Sat Jan 7 10:19:37 2017
Rows in CS minus classes 0 Sat Jan 7 10:19:41 2017
Rows in atomic_actions (A) minus A 0 Sat Jan 7 10:20:42 2017
Rows in atomic_actions (C) minus classes 0 Sat Jan 7 10:20:51 2017
Rows in atomic_actions (CR) minus CR 0 Sat Jan 7 10:21:06 2017
Rows in atomic_actions (CS) minus CS 0 Sat Jan 7 10:21:12 2017
Rows in atomic_actions (R) minus R 0 Sat Jan 7 10:21:22 2017
Rows in atoms minus classes 0 Sat Jan 7 10:21:32 2017
Rows in attributes minus stringtab 0 Sat Jan 7 10:21:55 2017
Rows in classes minus CS 0 Sat Jan 7 10:22:00 2017
Rows in classes minus atoms 0 Sat Jan 7 10:22:09 2017
Rows in classes minus normstr 0 Sat Jan 7 10:22:46 2017
Rows in classes minus normwrd 0* Sat Jan 7 10:23:25 2017
Rows in classes minus source_id_map 66202 Sat Jan 7 10:24:02 2017
Rows in classes minus word_index 0* Sat Jan 7 10:24:41 2017
Rows in foreign_classes minus classes 0 Sat Jan 7 10:24:46 2017
Rows in normstr minus classes 0 Sat Jan 7 10:25:08 2017
Rows in normwrd minus classes 0 Sat Jan 7 10:25:21 2017
Rows in source_id_map minus classes 0 Sat Jan 7 10:25:32 2017
Rows in stringtab minus attributes 0 Sat Jan 7 10:25:55 2017
Rows in word_index minus classes 0 Sat Jan 7 10:26:10 2017
S level attributes minus classes 0 Sat Jan 7 10:27:31 2017
S level relationships minus classes 0 Sat Jan 7 10:28:25 2017


New Checks

ComponentInfoRels

  • there are only certain combinations used in this table - see what they are and make a check to verify those are the only patterns
  • e.g. source,from type, to type

Orphaned tracking records

select id from tracking_records tr 
left join checklists_tracking_records ctr on tr.id=ctr.trackingRecords_id 
left join worklists_tracking_records wtr on tr.id=wtr.trackingRecords_id 
left join workflow_bins_tracking_records wbtr on tr.id=wbtr.trackingRecords_id 
where ctr.trackingRecords_id is null 
and wtr.trackingRecords_id is null 
and wbtr.trackingRecords_id is null;


Verify SNOMED SRC concept structure (see snomed.csh recipe at the end)
Atoms without AUIs

SELECT id FROM atoms a 
WHERE NOT EXISTS
    (SELECT * from atom_identity a_i
     WHERE
     a.codeId = a_i.codeId AND 
     a.conceptId = a_i.conceptId AND
     a.descriptorId = a_i.descriptorId AND
     a.stringClassId = a_i.stringClassId AND
     a.termType = a_i.termType AND
     a.terminology = a_i.terminology AND
     a.terminologyId = a_i.terminologyId
    );
    
SELECT id FROM atoms a
WHERE NOT EXISTS
    (SELECT alternateTerminologyIds from AtomJpa_alternateTerminologyIds b
     WHERE a.id = b.AtomJpa_id and alternateTerminologyIds_KEY = :terminology);



Relationships without RUIs (non-NCIMTH)

SELECT id FROM atom_relationships r
WHERE NOT EXISTS
    (SELECT alternateTerminologyIds FROM AtomRelationshipJpa_alternateTerminologyIds b
     WHERE r.id = b.AtomRelationshipJpa_id AND alternateTerminologyIds_KEY != :terminology);
     
-- does not apply for concepts because NCIMTH

SELECT id FROM code_relationships r
WHERE NOT EXISTS
    (SELECT alternateTerminologyIds FROM CodeRelationshipJpa_alternateTerminologyIds b
     WHERE r.id = b.CodeRelationshipJpa_id AND alternateTerminologyIds_KEY != :terminology);
     
SELECT id FROM descriptor_relationships r
WHERE NOT EXISTS
    (SELECT alternateTerminologyIds FROM DescriptorRelationshipJpa_alternateTerminologyIds b
     WHERE r.id = b.DescriptorRelationshipJpa_id AND alternateTerminologyIds_KEY != :terminology);
     
SELECT id FROM component_info_relationships r
WHERE NOT EXISTS
    (SELECT alternateTerminologyIds FROM ComponentInfoRelationshipJpa_alternateTerminologyIds b
     WHERE r.id = b.ComponentInfoRelationshipJpa_id AND alternateTerminologyIds_KEY != :terminology);


Attributes without ATUIs

SELECT a.id FROM attributes a, atoms_attributes a_a, atoms b, AtomJpa_alternateTerminologyIds b_a
WHERE a.id = a_a.attributes_id and b.id = a_a.atoms_id
AND b.id = b_a.AtomJpa_id AND b_a.alternateTerminologyIds_KEY = :terminology
AND (b_a.alternateTerminologyIds,b.terminology,'ATOM',a.name,a.terminology, a.terminologyId)
NOT IN
    (SELECT componentId,componentTerminology, componentType, name, terminology, terminologyId
     FROM attribute_identity a_i
     WHERE
     a_i.componentType = 'ATOM');
     
SELECT a.id FROM attributes a, codes_attributes c_a, codes b
WHERE a.id = c_a.attributes_id and b.id = c_a.codes_id
AND (b.id, b.terminology,'CODE',a.name,a.terminology, a.terminologyId)
NOT IN
    (SELECT componentId, componentTerminology, componentType, name, terminology, terminologyId
     FROM attribute_identity a_i
     WHERE
     a_i.componentType = 'CODE');
     
SELECT a.id FROM attributes a, concepts_attributes c_a, concepts b
WHERE a.id = c_a.attributes_id and b.id = c_a.concepts_id
AND (b.id,b.terminology,'CONCEPT',a.name,a.terminology, a.terminologyId)
NOT IN
    (SELECT componentId, componentTerminology, componentType, name, terminology, terminologyId
     FROM attribute_identity a_i
     WHERE
     a_i.componentType = 'CONCEPT');
    
SELECT a.id FROM attributes a, descriptors_attributes d_a, concepts b
WHERE a.id = d_a.attributes_id and b.id = d_a.descriptors_id
AND (b.id,b.terminology,'DESCRIPTOR',a.name,a.terminology, a.terminologyId)
NOT IN
    (SELECT componentId,componentTerminology, componentType, name, terminology, terminologyId
     FROM attribute_identity a_i
     WHERE
     a_i.componentType = 'DESCRIPTOR');


Relationships without inverses

-- Invalid inverseId in relationship_identity

select id,inverseId from relationship_identity
WHERE inverseId is null 
OR inverseId NOT IN
    (select id from relationship_identity);

-- Invalid inverse_id values in relationship_identity

SELECT id FROM relationship_identity WHERE NOT EXISTS
(SELECT ri1.id, ri2.id
FROM relationship_identity ri1, relationship_identity ri2
WHERE ri1.inverseId=ri2.id
AND ri1.additionalRelationshipType=
    (SELECT art1.abbreviation
     FROM additional_relationship_types art1, additional_relationship_types art2
     WHERE ri1.additionalRelationshipType = art1.abbreviation
       AND art1.terminology = :terminology
       AND art1.inverse_id = art2.id
       AND art2.terminology = :terminology
       AND art2.abbreviation = ri2.additionalRelationshipType
     )
AND ri1.terminology = ri2.terminology AND ri1.terminologyId=ri2.terminologyId
AND ri1.fromId = ri2.toId AND ri1.fromTerminology = ri2.toTerminology AND ri1.fromType = ri2.toType
AND ri1.relationshipType =
    (SELECT rt1.abbreviation
     FROM relationship_types rt1, relationship_types rt2
     WHERE ri1.relationshipType = rt1.abbreviation 
       AND rt1.terminology = :terminology
       AND rt1.inverse_id=rt2.id
       AND rt2.terminology = :terminology
       AND rt2.abbreviation = ri2.relationshipType
     )
AND ri1.terminology = ri2.terminology AND ri1.terminologyId=ri2.terminologyId
AND ri1.toId = ri2.fromId AND ri1.toTerminology=ri2.fromTerminology AND ri1.toType = ri2.fromType
);


DEFINITIONs not connected to core data

select id from definitions d
WHERE publishable=true
AND NOT EXISTS
(SELECT * FROM
    (SELECT definitions_id FROM atoms_definitions
     UNION ALL
     SELECT definitions_id FROM concepts_definitions
     UNION ALL
     SELECT definitions_id FROM descriptors_definitions) a
 WHERE d.id = a.definitions_id);


Publishable old version content (atoms, attributes, atom classes, relationships, definitions, subset, mapset, subset member, mapping)
Publishable content (attribute, relationship, atom, subset member, mapping, definitions) connected to unpublishable object

atom "concept terminology ids" should exist for MTH$VER where VER is the current version of MTH.

Tree positions should exist only for "current" version terminologies.

mapset.toTerminology and toVersion must exist in terminologies