instructions to load dataset to Amazon Athena

[yuvallb]

No description provided.

[Nintorac]

Hey, thanks for writing this! Saved me some time :D

I didn't verify the rest of the schemas yet but these came up as issue for us. When I have some time I will verify the rest and update if necessary.

[yuvallb]

Thanks Nintorac!
Let me know if there are other issues, and also if all went well.

[Nintorac]

There ended up being issues on most of the tables, I pretty much just set every varchar as a string so there might be some more improvements to make but here is the current schema I've come up with

schema.tar.gz (this outdated now, will upload again later)

Also seeing some weird results for some tables. eg

SELECT * FROM "mimic_iv_raw"."d_labitems" limit 10;

row 3 has a column that looks like "something, else" and athena is breaking on the , within the quotes

[alistairewj]

Would be happy to merge (a working version of) this - my initial thoughts are:

1. Not familiar with loading data into Athena, but my hunch is you need to specify that the fields are quoted "
   • Could be that you need to specify quotes in the info here:

     mimic-code/mimic-iv/buildmimic/athena/schema.sql

     Lines 18 to 19 in 3c91000

     	ROW FORMAT DELIMITED
     	FIELDS TERMINATED BY ','

   • Or it could be that this is misspecified, since the text columns are quoted:

     mimic-code/mimic-iv/buildmimic/athena/schema.sql

     Lines 26 to 33 in 3c91000

     	TBLPROPERTIES (
     	'ColumnsQuoted'='false',
     	'classification'='csv',
     	'columnsOrdered'='true',
     	'compressionType'='gzip',
     	'delimiter'=',',
     	'skip.header.line.count'='1',
     	'typeOfData'='file');

2. Needs an update for v2.2. The tables in core have been moved to hosp, and there are new columns.
3. Not sure what the distinction is between the mimiciv_parquet and mimiciv_csv schemas. In general it would be great to keep the mimiciv_hosp, mimiciv_icu schema names as that makes concept code much easier to transfer.