Guerrilla analytics

Guerrilla Analytics is data analytics performed in a very dynamic project environment that presents the team with varied and frequent disruptions and constrains the team in terms of the resources they can bring to bear on their analytics problem.
From: Guerrilla Analytics, 2015

Data science projects can get big, and keeping all files and folders tidy makes sure that it will stay possible to go through all the files and see how the analyses are performed. Keeping a well managed structure also allows for easier reproduction of the performed analyses when the folders are sent to other data scientist.

First we take a look at what a general folder structure might look like after a few weeks of assignments:

Old folder structure:

fs::dir_tree(path = “/Users/Alex/Documents/daur2/daurII_students_OLD”)
/Users/Alex/Documents/daur2/daurII_students_OLD
├── Alex_Groot copy.csv
├── Alex_Groot_1730201_Capstone_Exam_ABDS_DAUR2_BIOC.Rmd
├── Alex_Groot_1730201_Capstone_Exam_ABDS_DAUR2_BIOC.html
├── Andy.csv
├── David.csv
├── EXAM
│ ├── capstone_exam.Rmd
│ ├── capstone_exam.html
│ └── capstone_exam.md
├── GDS858.soft.gz
├── GPL11154.soft
├── GPL96.soft
├── GSE116936
│ └── GSE116936_RAW.tar
├── GSE116936_series_matrix.txt.gz
├── GSE116936_supp_files
│ ├── GSM3264576_A1-Ctrl.txt.gz
│ ├── GSM3264577_A2-Ctrl.txt.gz
│ ├── GSM3264578_A3-Ctrl.txt.gz
│ ├── GSM3264579_B1-PGE2.txt.gz
│ ├── GSM3264580_B2-PGE2.txt.gz
│ ├── GSM3264581_B3-PGE2.txt.gz
│ ├── GSM3264582_I1-TNF.txt.gz
│ ├── GSM3264583_I2-TNF.txt.gz
│ ├── GSM3264584_I3-TNF.txt.gz
│ ├── GSM3264585_J1-TNF+PGE2.txt.gz
│ ├── GSM3264586_J2-TNF+PGE2.txt.gz
│ └── GSM3264587_J3-TNF+PGE2.txt.gz
├── GSE148829
│ ├── GSE148829_BEAS_Basal_Pops_TPM.txt.gz
│ ├── GSE148829_Human1_Basal_Pops_TPM.txt.gz
│ ├── GSE148829_Human2_Basal_Pops_TPM.txt.gz
│ ├── GSE148829_Human_Ileum_absorptiveAndCryptentero_dge.csv.gz
│ ├── GSE148829_Human_lung_epithelial_cell_raw_counts.txt.gz
│ ├── GSE148829_Mouse_Basal_Pops_TPM.txt.gz
│ ├── GSE148829_NHP_TB_Grans_and_Lung_dge.txt.gz
│ ├── GSE148829_NHP_ileum_epithelial_raw_expression_counts.csv.gz
│ ├── GSE148829_NHP_lung_epithelial_raw_expression_counts.csv.gz
│ ├── GSE148829_NasalSSS_cleaned_raw_expression_counts.csv.gz
│ ├── GSE148829_RAW
│ │ ├── GSM4487938_Mouse_Nasal_IFNa_NH11_dge.txt.gz
│ │ ├── GSM4487939_Mouse_Nasal_IFNa_NH12_dge.txt.gz
│ │ ├── GSM4487940_Mouse_Nasal_IFNa_NH21_dge.txt.gz
│ │ ├── GSM4487941_Mouse_Nasal_IFNa_NH22_dge.txt.gz
│ │ ├── GSM4487942_Mouse_Nasal_IFNa_NS11_dge.txt.gz
│ │ ├── GSM4487943_Mouse_Nasal_IFNa_NS12_dge.txt.gz
│ │ ├── GSM4487944_Mouse_Nasal_IFNa_NS21_dge.txt.gz
│ │ ├── GSM4487945_Mouse_Nasal_IFNa_NS22_dge.txt.gz
│ │ ├── GSM4488219_Mouse_basal_stim_IFNA_0.1_REP1_TPM.txt.gz
│ │ ├── GSM4488220_Mouse_basal_stim_IFNA_0.1_REP2_TPM.txt.gz
│ │ ├── GSM4488221_Mouse_basal_stim_IFNA_0.1_REP3_TPM.txt.gz
│ │ ├── GSM4488222_Mouse_basal_stim_IFNA_0.5_REP1_TPM.txt.gz
│ │ ├── GSM4488223_Mouse_basal_stim_IFNA_0.5_REP2_TPM.txt.gz
│ │ ├── GSM4488224_Mouse_basal_stim_IFNA_0.5_REP3_TPM.txt.gz
│ │ ├── GSM4488225_Mouse_basal_stim_IFNB_0.1_REP1_TPM.txt.gz
│ │ ├── GSM4488226_Mouse_basal_stim_IFNB_0.1_REP2_TPM.txt.gz
│ │ ├── GSM4488227_Mouse_basal_stim_IFNB_0.1_REP3_TPM.txt.gz
│ │ ├── GSM4488228_Mouse_basal_stim_IFNB_0.5_REP1_TPM.txt.gz
│ │ ├── GSM4488229_Mouse_basal_stim_IFNB_0.5_REP2_TPM.txt.gz
│ │ ├── GSM4488230_Mouse_basal_stim_IFNB_0.5_REP3_TPM.txt.gz
│ │ ├── GSM4488231_Mouse_basal_stim_IFNA_1_REP1_TPM.txt.gz
│ │ ├── GSM4488232_Mouse_basal_stim_IFNA_1_REP2_TPM.txt.gz
│ │ ├── GSM4488233_Mouse_basal_stim_IFNA_1_REP3_TPM.txt.gz
│ │ ├── GSM4488234_Mouse_basal_stim_IFNA_2_REP1_TPM.txt.gz
│ │ ├── GSM4488235_Mouse_basal_stim_IFNA_2_REP2_TPM.txt.gz
│ │ ├── GSM4488236_Mouse_basal_stim_IFNA_2_REP3_TPM.txt.gz
│ │ ├── GSM4488237_Mouse_basal_stim_IFNB_1_REP1_TPM.txt.gz
│ │ ├── GSM4488238_Mouse_basal_stim_IFNB_1_REP2_TPM.txt.gz
│ │ ├── GSM4488239_Mouse_basal_stim_IFNB_1_REP3_TPM.txt.gz
│ │ ├── GSM4488240_Mouse_basal_stim_IFNB_2_REP1_TPM.txt.gz
│ │ ├── GSM4488241_Mouse_basal_stim_IFNB_2_REP2_TPM.txt.gz
│ │ ├── GSM4488242_Mouse_basal_stim_IFNB_2_REP3_TPM.txt.gz
│ │ ├── GSM4488243_Mouse_basal_stim_IFNA_5_REP1_TPM.txt.gz
│ │ ├── GSM4488244_Mouse_basal_stim_IFNA_5_REP2_TPM.txt.gz
│ │ ├── GSM4488245_Mouse_basal_stim_IFNA_5_REP3_TPM.txt.gz
│ │ ├── GSM4488246_Mouse_basal_stim_IFNA_10_REP1_TPM.txt.gz
│ │ ├── GSM4488247_Mouse_basal_stim_IFNA_10_REP2_TPM.txt.gz
│ │ ├── GSM4488248_Mouse_basal_stim_IFNA_10_REP3_TPM.txt.gz
│ │ ├── GSM4488249_Mouse_basal_stim_IFNB_5_REP1_TPM.txt.gz
│ │ ├── GSM4488250_Mouse_basal_stim_IFNB_5_REP2_TPM.txt.gz
│ │ ├── GSM4488251_Mouse_basal_stim_IFNB_5_REP3_TPM.txt.gz
│ │ ├── GSM4488252_Mouse_basal_stim_IFNB_10_REP1_TPM.txt.gz
│ │ ├── GSM4488253_Mouse_basal_stim_IFNB_10_REP2_TPM.txt.gz
│ │ ├── GSM4488254_Mouse_basal_stim_IFNB_10_REP3_TPM.txt.gz
│ │ ├── GSM4488255_Mouse_basal_stim_IFNG_0.1_REP1_TPM.txt.gz
│ │ ├── GSM4488256_Mouse_basal_stim_IFNG_0.1_REP2_TPM.txt.gz
│ │ ├── GSM4488257_Mouse_basal_stim_IFNG_0.1_REP3_TPM.txt.gz
│ │ ├── GSM4488258_Mouse_basal_stim_IFNG_0.5_REP1_TPM.txt.gz
│ │ ├── GSM4488259_Mouse_basal_stim_IFNG_0.5_REP2_TPM.txt.gz
│ │ ├── GSM4488260_Mouse_basal_stim_IFNG_0.5_REP3_TPM.txt.gz
│ │ ├── GSM4488261_Mouse_basal_stim_IFNG_1_REP1_TPM.txt.gz
│ │ ├── GSM4488262_Mouse_basal_stim_IFNG_1_REP2_TPM.txt.gz
│ │ ├── GSM4488263_Mouse_basal_stim_IFNG_1_REP3_TPM.txt.gz
│ │ ├── GSM4488264_Mouse_basal_stim_IFNG_2_REP1_TPM.txt.gz
│ │ ├── GSM4488265_Mouse_basal_stim_IFNG_2_REP2_TPM.txt.gz
│ │ ├── GSM4488266_Mouse_basal_stim_IFNG_2_REP3_TPM.txt.gz
│ │ ├── GSM4488267_Mouse_basal_stim_IFNG_5_REP1_TPM.txt.gz
│ │ ├── GSM4488268_Mouse_basal_stim_IFNG_5_REP2_TPM.txt.gz
│ │ ├── GSM4488269_Mouse_basal_stim_IFNG_5_REP3_TPM.txt.gz
│ │ ├── GSM4488270_Mouse_basal_stim_IFNG_10_REP1_TPM.txt.gz
│ │ ├── GSM4488271_Mouse_basal_stim_IFNG_10_REP2_TPM.txt.gz
│ │ ├── GSM4488272_Mouse_basal_stim_IFNG_10_REP3_TPM.txt.gz
│ │ ├── GSM4488273_Mouse_basal_stim_Untreated_0_REP1_TPM.txt.gz
│ │ ├── GSM4488274_Mouse_basal_stim_Untreated_0_REP2_TPM.txt.gz
│ │ ├── GSM4488275_Mouse_basal_stim_Untreated_0_REP3_TPM.txt.gz
│ │ ├── GSM4488276_Mouse_basal_stim_Untreated_0_REP4_TPM.txt.gz
│ │ ├── GSM4488277_Mouse_basal_stim_Untreated_0_REP5_TPM.txt.gz
│ │ ├── GSM4488278_Mouse_basal_stim_Untreated_0_REP6_TPM.txt.gz
│ │ ├── GSM4488279_Mouse_basal_stim_Untreated_0_REP7_TPM.txt.gz
│ │ ├── GSM4488280_Mouse_basal_stim_Untreated_0_REP8_TPM.txt.gz
│ │ ├── GSM4488281_Mouse_basal_stim_Untreated_0_REP9_TPM.txt.gz
│ │ ├── GSM4488282_Mouse_basal_stim_Untreated_0_REP10_TPM.txt.gz
│ │ ├── GSM4488283_Mouse_basal_stim_Untreated_0_REP11_TPM.txt.gz
│ │ └── GSM4488284_Mouse_basal_stim_Untreated_0_REP12_TPM.txt.gz
│ └── GSE148829_RAW.tar
├── GSE75073
│ ├── GSE75073_RAW.tar
│ ├── GSE75073_Signals_450k.txt.gz
│ └── GSE75073_Signals_Epic.txt.gz
├── GSE76073
│ ├── GSE76073_Normalized_counts.txt.gz
│ └── GSE76073_Raw_counts.txt.gz
├── GSE923_series_matrix.txt.gz
├── Images
│ ├── 2020-04-20CORT.png
│ ├── 20200403_citrul.png
│ ├── 20200404_citrul.png
│ ├── 20200404_ifabp.png
│ ├── Bioconductor.png
│ ├── Biostrings.png
│ ├── BrowseSeq
│ │ ├── Dia1.PNG
│ │ └── sequence_alignment_plots
│ │ ├── Dia1.PNG
│ │ ├── Dia2.PNG
│ │ ├── Dia3.PNG
│ │ ├── Dia4.PNG
│ │ ├── Dia5.PNG
│ │ ├── Dia6.PNG
│ │ ├── Dia7.PNG
│ │ └── Dia8.PNG
│ ├── DESeq2.png
│ ├── Dia1.JPG
│ ├── Dia3.JPG
│ ├── GenomicRanges.png
│ ├── Git_Setup_1.JPG
│ ├── Glimma.png
│ ├── IRanges.png
│ ├── MSnbase.png
│ ├── Project_1.jpg
│ ├── Project_2.jpg
│ ├── Rlogo.png
│ ├── Rtools.png
│ ├── bioconductorlogo.jpg
│ ├── cat_sleeping_on_keyboard.jpg
│ ├── citrulline.png
│ ├── connectome-1.png
│ ├── cortline.png
│ ├── covid19_ziekenhuis.png
│ ├── dotdotdot_rotate_axis.png
│ ├── edgeR.png
│ ├── exercise_2_1_1.png
│ ├── exercise_2_1_1_c.png
│ ├── forcats.png
│ ├── git_web.png
│ ├── gitinstallermac.png
│ ├── heatmap.png
│ ├── heatmap1.png
│ ├── heatmap2.png
│ ├── hex-ggplot2.png
│ ├── installpackages.png
│ ├── loadpackages.png
│ ├── logo_R.jpg
│ ├── ma1.png
│ ├── ma2.png
│ ├── mzR_hl.png
│ ├── new_proj_done.png
│ ├── new_proj_git.png
│ ├── one-doesnot-simply.jpg
│ ├── open_486.jpg
│ ├── open_compaq.jpg
│ ├── pca1.png
│ ├── pepper.jpg
│ ├── pipe.png
│ ├── purrr_small.png
│ ├── purrr_sticker.jpg
│ ├── pvalues.png
│ ├── r4ds.png
│ ├── readr_sticker.png
│ ├── readxl_sticker.png
│ ├── rmac_download.png
│ ├── rstudio_download.png
│ ├── rstudio_leeg.png
│ ├── rwin_download.png
│ ├── security.png
│ ├── setup-wizard-rstudio.png
│ ├── setupRtools.png
│ ├── stringr.png
│ ├── summarizedexperiment.png
│ ├── tidyr_sticker.png
│ ├── tidyverse_sticker.png
│ ├── usrlocal.png
│ ├── wizardRmac.png
│ ├── wizardRwindows.png
│ ├── wontinstall.png
│ ├── workflow_course.png
│ ├── xquarkdisk.png
│ ├── xquarkwizard.png
│ ├── xquartz.png
│ └── yoda_the_last_jedi.jpg
├── John.csv
├── Les 6.Rmd
├── Les4.Rmd
├── Les4.nb.html
├── Les5.Rmd
├── Lesson_5_Exercise_a1.Rmd
├── Mike.csv
├── R
│ └── do_tidy_pertussis.R
├── README.md
├── Steve.csv
├── answers
│ ├── daur2_lesson51_demo_multiple_sequence_alignment.Rmd
│ ├── daur2_lesson51_demo_multiple_sequence_alignment.html
│ ├── daur2_lesson51_demo_multiple_sequence_alignment_files
│ │ └── figure-html
│ │ ├── unnamed-chunk-23-1.png
│ │ ├── unnamed-chunk-24-1.png
│ │ ├── unnamed-chunk-29-1.png
│ │ └── unnamed-chunk-30-1.png
│ ├── daur2_lesson_6_exercise_rnaseq.Rmd
│ ├── daur2_lesson_6_exercise_rnaseq.html
│ ├── daur2_lesson_6_exercise_rnaseq_student_version.Rmd
│ └── daur2_lesson_6_exercise_rnaseq_student_version.html
├── caspase_align.aux
├── caspase_align.log
├── caspase_align.tex
├── code
│ └── load_data.R
├── data
│ ├── CISID_pertussis_10082018.csv
│ ├── biomaRt_homology_example.rds
│ ├── biomaRt_homology_example_aa.rds
│ ├── biomaRt_homology_example_dna.rds
│ ├── chircus_blg_aminoacids.fasta
│ ├── course_r_packages.tsv
│ ├── course_r_packages.txt
│ ├── covid_rivm
│ ├── go_bp-hit.csv
│ ├── go_cc-hit.csv
│ ├── go_mf-hit.csv
│ ├── hobbit_chapter.txt
│ ├── hobbit_dwarves.txt
│ ├── hox_aa_sequences.txt
│ ├── hox_dna_sequences.txt
│ ├── human_hoxb1_protein.fasta
│ ├── human_hoxb1_protein.gb
│ ├── human_rhodopsin_protein.fasta
│ ├── human_ttn_genomic_dna.fasta
│ ├── jp_coldata.txt
│ ├── jp_counts.txt
│ ├── jp_features.txt
│ ├── jp_miame.rds
│ ├── jp_miame.txt
│ ├── lesson2
│ │ ├── ENST00000642929.1.fa
│ │ ├── ENST00000643997.1.fa
│ │ ├── ENST00000644264.1.fa
│ │ ├── ENST00000644696.1.fa
│ │ ├── ENST00000644919.1.fa
│ │ ├── ENST00000646017.1.fa
│ │ ├── ENST00000646618.2.fa
│ │ ├── ENST00000646644.1.fa
│ │ ├── ENST00000647220.1.fa
│ │ ├── ENST00000647443.1.fa
│ │ ├── cholesterol.xlsx
│ │ ├── diet
│ │ │ ├── Andy.csv
│ │ │ ├── David.csv
│ │ │ ├── John.csv
│ │ │ ├── Mike.csv
│ │ │ └── Steve.csv
│ │ ├── diet_data.zip
│ │ └── sequence1.txt
│ └── lesson3
│ ├── GO-terms.txt
│ ├── dna_random
│ ├── sequence.fasta
│ ├── sequence1.txt
│ └── sequence2.txt
├── daurII_students
│ ├── R
│ │ └── do_tidy_pertussis.R
│ ├── README.md
│ ├── code
│ │ └── load_data.R
│ ├── data
│ │ ├── CISID_pertussis_10082018.csv
│ │ ├── course_r_packages.tsv
│ │ ├── covid_rivm
│ │ ├── go_bp-hit.csv
│ │ ├── go_cc-hit.csv
│ │ ├── go_mf-hit.csv
│ │ ├── hobbit_chapter.txt
│ │ ├── hobbit_dwarves.txt
│ │ ├── hox_aa_sequences.txt
│ │ ├── hox_dna_sequences.txt
│ │ ├── human_hoxb1_protein.fasta
│ │ ├── human_hoxb1_protein.gb
│ │ ├── human_rhodopsin_protein.fasta
│ │ ├── human_ttn_genomic_dna.fasta
│ │ └── lesson2
│ │ └── diet_data.zip
│ ├── daurII_students.Rproj
│ └── demos
│ └── 01_demo_functions.Rmd
├── daurII_students.Rproj
├── demos
│ ├── 01_demo_functions.Rmd
│ ├── 021_demo_2_agenda.Rmd
│ ├── 021_demo_2_agenda.html
│ ├── 022_lesson2_map_nest.Rmd
│ ├── 022_lesson2_map_nest.html
│ ├── 031_strings_and_regex.Rmd
│ ├── 031_strings_and_regex.html
│ ├── 031_strings_and_regex.pdf
│ ├── 041_biomart.Rmd
│ ├── 042_geoquery.Rmd
│ ├── 043_rentrez.Rmd
│ ├── 051_demo_multiple_sequence_alignment.Rmd
│ ├── 051_demo_multiple_sequence_alignment.html
│ ├── 051_demo_multiple_sequence_alignment.md
│ ├── 051_demo_multiple_sequence_alignment_files
│ │ └── figure-html
│ │ ├── unnamed-chunk-23-1.png
│ │ ├── unnamed-chunk-24-1.png
│ │ ├── unnamed-chunk-29-1.png
│ │ └── unnamed-chunk-30-1.png
│ ├── 052_demo_summarized_experiment.Rmd
│ ├── 052_demo_summarized_experiment.html
│ ├── 052_demo_summarized_experiment.md
│ ├── 61_heatmaps.Rmd
│ ├── 61_heatmaps.html
│ ├── 62_rnseq_workflow.Rmd
│ └── 62_rnseq_workflow.html
├── les1.Rmd
├── les1.nb.html
├── les2.Rmd
├── les2.nb.html
├── les3.Rmd
├── my_transcript.fasta
├── resources.Rmd
└── unzip
├── Andy.csv
├── David.csv
├── John.csv
├── Mike.csv
└── Steve.csv

As you can see, the original file structure used during the earlier data science course is quite the mess. Instead we now make usage of the Guerilla analytics framework to store data and manage files. Using simple structures to store data, code, generated graphs and makes it easier to use and replicate analyses, even years after the analyses are performed.
Making sure all files in a environment pertain only to that single project is important. And to make data storage simple, a central data folder in the root of the enviornment is used with clear naming conventions, dates and versions. Also metadata files are used to allow anyone with access to the project to see what the contents of the project are.

Here is the above file structure, with the guerilla analytics framework retroactively applied:

New folder structure:

├── EXAM
│ ├── Alex_Groot copy.csv
│ ├── capstone_exam.Rmd
│ ├── capstone_exam.html
│ └── capstone_exam.md
├── Images
│ ├── 2020-04-20CORT.png
│ ├── 20200403_citrul.png
│ ├── BrowseSeq
│ │ ├── Dia1.PNG
│ │ └── sequence_alignment_plots
│ │ ├── Dia1.PNG
│ │ ├── Dia2.PNG
│ └── yoda_the_last_jedi.jpg
├── R
│ └── do_tidy_pertussis.R
├── README.md
├── caspase_align.aux
├── caspase_align.log
├── caspase_align.tex
├── code
│ └── load_data.R
├── data_raw
│ ├── lesson2
│ │ ├── ENST00000642929.1.fa
│ │ ├── ENST00000643997.1.fa
│ │ ├── ENST00000644264.1.fa
│ │ ├── cholesterol.xlsx
│ │ ├── diet
│ │ │ ├── Andy.csv
│ │ │ └── Steve.csv
│ │ ├── diet_data.zip
│ │ └── sequence1.txt
│ ├── lesson3
│ │ ├── GO-terms.txt
│ │ ├── dna_random
│ │ ├── sequence.fasta
│ │ ├── sequence1.txt
│ │ └── sequence2.txt
│ └── unsorted
│ ├── Andy.csv
│ ├── CISID_pertussis_10082018.csv
│ ├── David.csv
│ ├── GSE116936
│ │ └── GSE116936_RAW.tar
│ ├── GSE116936_series_matrix.txt.gz
│ ├── GSE116936_supp_files
│ │ ├── GSM3264576_A1-Ctrl.txt.gz
│ │ ├── GSM3264577_A2-Ctrl.txt.gz
│ │ ├── GSM3264578_A3-Ctrl.txt.gz
│ │ ├── GSM3264585_J1-TNF+PGE2.txt.gz
│ │ ├── GSM3264586_J2-TNF+PGE2.txt.gz
│ │ └── GSM3264587_J3-TNF+PGE2.txt.gz
│ ├── GSE148829
│ │ ├── GSE148829_BEAS_Basal_Pops_TPM.txt.gz
│ │ ├── GSE148829_Human1_Basal_Pops_TPM.txt.gz
│ │ ├── GSE148829_Human2_Basal_Pops_TPM.txt.gz
│ │ │ ├── GSM4487938_Mouse_Nasal_IFNa_NH11_dge.txt.gz
│ │ │ ├── GSM4487939_Mouse_Nasal_IFNa_NH12_dge.txt.gz
│ │ │ ├── GSM4487940_Mouse_Nasal_IFNa_NH21_dge.txt.gz
│ │ │ ├── GSM4487941_Mouse_Nasal_IFNa_NH22_dge.txt.gz
│ │ │ ├── GSM4487942_Mouse_Nasal_IFNa_NS11_dge.txt.gz
│ │ │ ├── GSM4487943_Mouse_Nasal_IFNa_NS12_dge.txt.gz
│ │ │ ├── GSM4487944_Mouse_Nasal_IFNa_NS21_dge.txt.gz
│ │ │ ├── GSM4487945_Mouse_Nasal_IFNa_NS22_dge.txt.gz
│ │ │ ├── GSM4488282_Mouse_basal_stim_Untreated_0_REP10_TPM.txt.gz
│ │ │ ├── GSM4488283_Mouse_basal_stim_Untreated_0_REP11_TPM.txt.gz
│ │ │ └── GSM4488284_Mouse_basal_stim_Untreated_0_REP12_TPM.txt.gz
│ │ └── GSE148829_RAW.tar
│ ├── GSE75073
│ │ ├── GSE75073_RAW.tar
│ │ ├── GSE75073_Signals_450k.txt.gz
│ │ └── GSE75073_Signals_Epic.txt.gz
│ ├── GSE76073
│ │ ├── GSE76073_Normalized_counts.txt.gz
│ │ └── GSE76073_Raw_counts.txt.gz
│ ├── GSE923_series_matrix.txt.gz
│ ├── human_hoxb1_protein.fasta
│ ├── human_hoxb1_protein.gb
│ ├── human_rhodopsin_protein.fasta
│ ├── human_ttn_genomic_dna.fasta
│ ├── jp_coldata.txt
│ ├── jp_counts.txt
│ ├── jp_features.txt
│ ├── jp_miame.txt
│ ├── my_transcript.fasta
│ └── unzip
│ ├── Andy.csv
│ ├── David.csv
│ ├── John.csv
│ ├── Mike.csv
│ └── Steve.csv
├── data_rds
│ ├── biomaRt_homology_example.rds
│ ├── biomaRt_homology_example_aa.rds
│ ├── biomaRt_homology_example_dna.rds
│ └── jp_miame.rds
├── daurII_students
│ ├── R
│ │ └── do_tidy_pertussis.R
│ ├── README.md
│ ├── code
│ │ └── load_data.R
│ ├── data
│ │ ├── CISID_pertussis_10082018.csv
│ │ ├── course_r_packages.tsv
│ │ ├── covid_rivm
│ │ ├── human_rhodopsin_protein.fasta
│ │ ├── human_ttn_genomic_dna.fasta
│ │ └── lesson2
│ │ └── diet_data.zip
│ ├── daurII_students.Rproj
│ └── demos
│ └── 01_demo_functions.Rmd
├── daurII_students.Rproj
├── demos
│ ├── 01_demo_functions.Rmd
│ ├── 021_demo_2_agenda.Rmd
│ ├── 021_demo_2_agenda.html
│ ├── 61_heatmaps.Rmd
│ ├── 61_heatmaps.html
│ ├── 62_rnseq_workflow.Rmd
│ └── 62_rnseq_workflow.html
└── markdown
├── Alex_Groot_1730201_Capstone_Exam_ABDS_DAUR2_BIOC.Rmd
├── Alex_Groot_1730201_Capstone_Exam_ABDS_DAUR2_BIOC.html
├── Les 6.Rmd
├── Les4.Rmd
├── Les4.nb.html
├── Les5.Rmd
├── Lesson_5_Exercise_a1.Rmd
├── answers
│ ├── daur2_lesson51_demo_multiple_sequence_alignment.Rmd
│ ├── daur2_lesson51_demo_multiple_sequence_alignment.html
│ ├── daur2_lesson51_demo_multiple_sequence_alignment_files
│ │ └── figure-html
│ │ ├── unnamed-chunk-23-1.png
│ │ ├── unnamed-chunk-24-1.png
│ │ ├── unnamed-chunk-29-1.png
│ │ └── unnamed-chunk-30-1.png
│ ├── daur2_lesson_6_exercise_rnaseq.Rmd
│ ├── daur2_lesson_6_exercise_rnaseq.html
│ ├── daur2_lesson_6_exercise_rnaseq_student_version.Rmd
│ └── daur2_lesson_6_exercise_rnaseq_student_version.html
├── les1.Rmd
├── les1.nb.html
├── les2.Rmd
├── les2.nb.html
├── les3.Rmd
└── resources.Rmd

In order to have the folder structured in the Guerrilla analytics framework method, I retroactively moved the files. This is just used as an example to show what a tidier and better organized structure might look like. The big problem with this is that it will break many, if not all, of the scripts inside due to changing of file paths.

The best way to keep files well-managed is to set up a file structure before starting and sticking to it. Keeping a folder with all the raw data with its names the way they were obtained makes sure confusion is kept to a minimum when the data is looked at. Creating seperate data folders enables to track transformed data while preserving the raw data.

And keeping metadata files in all the folders that explain not just folder structure, but importantly also file naming conventions and potentially used abbreviations for them will mean that even years later someone will be able to look at the project and learn what is inside.

This portfolio was made with the Guerrilla analytics framework in mind from the start:

Old folder structure:

/home/alexgroot/dsfb2_quarto
├── 001_CV.qmd
├── 002_Looking_ahead.qmd
├── 003_Guerrilla_analytics_framework.qmd
├── 004_C_elegans.qmd
├── 005_open_peer_review.qmd
├── 006_relational_databases.qmd
├── 007_R_Package.qmd
├── 008_Zotero.qmd
├── 008_Zotero.quarto_ipynb
├── 009_Parameterized_report.qmd
├── 010_ML_Getting_started.qmd
├── 404.qmd
├── LICENSE
├── _quarto.yml
├── _site
├── about.qmd
├── css
│   └── 008_Zotero.css
├── data_output
│   ├── data_00100
│   └── data_0060
│       ├── dengue.csv
│       ├── dengue.rds
│       ├── flu.csv
│       ├── ...
├── data_raw
│   ├── data_00100
│   │   ├── README
│   │   ├── breast_cancer
│   │   │   ├── Index
│   │   │   ├── breast-cancer.data
│   │   │   └── breast-cancer.names
│   │   └── heart_failure.csv
│   ├── data_0040
│   │   ├── CE.LIQ.FLOW.062_Tidydata.xlsx
│   │   └── README
│   ├── data_0050
│   │   ├── README
│   │   ├── TMNT Combined Data OSF.csv
│   │   ├── TMNT Meta-analysis OSF.csv
│   │   ├── ...
│   │   ├── osfstorage-archive
│   │   │   ├── TMNT Combined Data OSF.csv
│   │   │   ├── TMNT Meta-analysis OSF.csv
│   │   │   ├── ...
│   │   │   └── readme.txt
│   │   ├── osfstorage-archive.zip
│   │   └── readme.txt
│   ├── data_0060
│   │   ├── README
│   │   ├── dengue_data.csv
│   │   └── flu_data.csv
│   └── data_0090
│       └── data.csv
├── dsfb2_quarto.Rproj
├── images
│   ├── data_00100
│   │   ├── baseline_accuracy_model_performance.png
│   │   └── baseline_precision_model_performance.png
│   ├── data_0050
│   │   └── original_plot.png
│   └── projects
│       ├── 003_Guerrilla_analytics_framework.png
│       ├── 006_relational_databases.png
│       ├── ...
├── index.qmd
├── projects.ejs
├── projects.qmd
├── references
│   └── 008_nanopore.bib
├── scripts
│   └── 009_Parameterized_report.R
└── styles.css

All the required main files are in the root folder of the project, as required for the website to render, e.g. the quarto documents, yml, and project files.
Next there are extra folders required by the rendered site; images, css, _site, references.
Then folders used for the R chunks and scripts; data_raw/output, scripts.
Finally Github related files such as .gitignore and LICENSE.
Structured systems as such make it easier to both find old files and place new ones.

--- title: "Guerrilla analytics" format: html: other-links: - text: Guerrilla analytics book icon: book href: https://www.sciencedirect.com/book/9780128002186/guerrilla-analytics --- Guerrilla Analytics is data analytics performed in a very dynamic project environment that presents the team with varied and frequent disruptions and constrains the team in terms of the resources they can bring to bear on their analytics problem. From: <a href="https://www.sciencedirect.com/science/article/abs/pii/B9780128002186000011">Guerrilla Analytics, 2015</a> Data science projects can get big, and keeping all files and folders tidy makes sure that it will stay possible to go through all the files and see how the analyses are performed. Keeping a well managed structure also allows for easier reproduction of the performed analyses when the folders are sent to other data scientist. First we take a look at what a general folder structure might look like after a few weeks of assignments: ```{css, echo=FALSE} .FixedHeightContainer { height: 700px; padding: 3px; background: rgba(0, 0, 0, .05); } .Content { height: 624px; overflow: scroll; margin: 4px; background: rgba(0, 0, 0, .05); } ::-webkit-scrollbar { -webkit-appearance: none; width: 7px; } ::-webkit-scrollbar-thumb { border-radius: 4px; background-color: rgba(0, 0, 0, .5); box-shadow: 0 0 1px rgba(255, 255, 255, .5); } ``` :::{.card} <div class="FixedHeightContainer"> <h3>Old folder structure:</h3> <div class="Content"> fs::dir_tree(path = "/Users/Alex/Documents/daur2/daurII_students_OLD") /Users/Alex/Documents/daur2/daurII_students_OLD ├── Alex_Groot copy.csv ├── Alex_Groot_1730201_Capstone_Exam_ABDS_DAUR2_BIOC.Rmd ├── Alex_Groot_1730201_Capstone_Exam_ABDS_DAUR2_BIOC.html ├── Andy.csv ├── David.csv ├── EXAM │ ├── capstone_exam.Rmd │ ├── capstone_exam.html │ └── capstone_exam.md ├── GDS858.soft.gz ├── GPL11154.soft ├── GPL96.soft ├── GSE116936 │ └── GSE116936_RAW.tar ├── GSE116936_series_matrix.txt.gz ├── GSE116936_supp_files │ ├── GSM3264576_A1-Ctrl.txt.gz │ ├── GSM3264577_A2-Ctrl.txt.gz │ ├── GSM3264578_A3-Ctrl.txt.gz │ ├── GSM3264579_B1-PGE2.txt.gz │ ├── GSM3264580_B2-PGE2.txt.gz │ ├── GSM3264581_B3-PGE2.txt.gz │ ├── GSM3264582_I1-TNF.txt.gz │ ├── GSM3264583_I2-TNF.txt.gz │ ├── GSM3264584_I3-TNF.txt.gz │ ├── GSM3264585_J1-TNF+PGE2.txt.gz │ ├── GSM3264586_J2-TNF+PGE2.txt.gz │ └── GSM3264587_J3-TNF+PGE2.txt.gz ├── GSE148829 │ ├── GSE148829_BEAS_Basal_Pops_TPM.txt.gz │ ├── GSE148829_Human1_Basal_Pops_TPM.txt.gz │ ├── GSE148829_Human2_Basal_Pops_TPM.txt.gz │ ├── GSE148829_Human_Ileum_absorptiveAndCryptentero_dge.csv.gz │ ├── GSE148829_Human_lung_epithelial_cell_raw_counts.txt.gz │ ├── GSE148829_Mouse_Basal_Pops_TPM.txt.gz │ ├── GSE148829_NHP_TB_Grans_and_Lung_dge.txt.gz │ ├── GSE148829_NHP_ileum_epithelial_raw_expression_counts.csv.gz │ ├── GSE148829_NHP_lung_epithelial_raw_expression_counts.csv.gz │ ├── GSE148829_NasalSSS_cleaned_raw_expression_counts.csv.gz │ ├── GSE148829_RAW │ │ ├── GSM4487938_Mouse_Nasal_IFNa_NH11_dge.txt.gz │ │ ├── GSM4487939_Mouse_Nasal_IFNa_NH12_dge.txt.gz │ │ ├── GSM4487940_Mouse_Nasal_IFNa_NH21_dge.txt.gz │ │ ├── GSM4487941_Mouse_Nasal_IFNa_NH22_dge.txt.gz │ │ ├── GSM4487942_Mouse_Nasal_IFNa_NS11_dge.txt.gz │ │ ├── GSM4487943_Mouse_Nasal_IFNa_NS12_dge.txt.gz │ │ ├── GSM4487944_Mouse_Nasal_IFNa_NS21_dge.txt.gz │ │ ├── GSM4487945_Mouse_Nasal_IFNa_NS22_dge.txt.gz │ │ ├── GSM4488219_Mouse_basal_stim_IFNA_0.1_REP1_TPM.txt.gz │ │ ├── GSM4488220_Mouse_basal_stim_IFNA_0.1_REP2_TPM.txt.gz │ │ ├── GSM4488221_Mouse_basal_stim_IFNA_0.1_REP3_TPM.txt.gz │ │ ├── GSM4488222_Mouse_basal_stim_IFNA_0.5_REP1_TPM.txt.gz │ │ ├── GSM4488223_Mouse_basal_stim_IFNA_0.5_REP2_TPM.txt.gz │ │ ├── GSM4488224_Mouse_basal_stim_IFNA_0.5_REP3_TPM.txt.gz │ │ ├── GSM4488225_Mouse_basal_stim_IFNB_0.1_REP1_TPM.txt.gz │ │ ├── GSM4488226_Mouse_basal_stim_IFNB_0.1_REP2_TPM.txt.gz │ │ ├── GSM4488227_Mouse_basal_stim_IFNB_0.1_REP3_TPM.txt.gz │ │ ├── GSM4488228_Mouse_basal_stim_IFNB_0.5_REP1_TPM.txt.gz │ │ ├── GSM4488229_Mouse_basal_stim_IFNB_0.5_REP2_TPM.txt.gz │ │ ├── GSM4488230_Mouse_basal_stim_IFNB_0.5_REP3_TPM.txt.gz │ │ ├── GSM4488231_Mouse_basal_stim_IFNA_1_REP1_TPM.txt.gz │ │ ├── GSM4488232_Mouse_basal_stim_IFNA_1_REP2_TPM.txt.gz │ │ ├── GSM4488233_Mouse_basal_stim_IFNA_1_REP3_TPM.txt.gz │ │ ├── GSM4488234_Mouse_basal_stim_IFNA_2_REP1_TPM.txt.gz │ │ ├── GSM4488235_Mouse_basal_stim_IFNA_2_REP2_TPM.txt.gz │ │ ├── GSM4488236_Mouse_basal_stim_IFNA_2_REP3_TPM.txt.gz │ │ ├── GSM4488237_Mouse_basal_stim_IFNB_1_REP1_TPM.txt.gz │ │ ├── GSM4488238_Mouse_basal_stim_IFNB_1_REP2_TPM.txt.gz │ │ ├── GSM4488239_Mouse_basal_stim_IFNB_1_REP3_TPM.txt.gz │ │ ├── GSM4488240_Mouse_basal_stim_IFNB_2_REP1_TPM.txt.gz │ │ ├── GSM4488241_Mouse_basal_stim_IFNB_2_REP2_TPM.txt.gz │ │ ├── GSM4488242_Mouse_basal_stim_IFNB_2_REP3_TPM.txt.gz │ │ ├── GSM4488243_Mouse_basal_stim_IFNA_5_REP1_TPM.txt.gz │ │ ├── GSM4488244_Mouse_basal_stim_IFNA_5_REP2_TPM.txt.gz │ │ ├── GSM4488245_Mouse_basal_stim_IFNA_5_REP3_TPM.txt.gz │ │ ├── GSM4488246_Mouse_basal_stim_IFNA_10_REP1_TPM.txt.gz │ │ ├── GSM4488247_Mouse_basal_stim_IFNA_10_REP2_TPM.txt.gz │ │ ├── GSM4488248_Mouse_basal_stim_IFNA_10_REP3_TPM.txt.gz │ │ ├── GSM4488249_Mouse_basal_stim_IFNB_5_REP1_TPM.txt.gz │ │ ├── GSM4488250_Mouse_basal_stim_IFNB_5_REP2_TPM.txt.gz │ │ ├── GSM4488251_Mouse_basal_stim_IFNB_5_REP3_TPM.txt.gz │ │ ├── GSM4488252_Mouse_basal_stim_IFNB_10_REP1_TPM.txt.gz │ │ ├── GSM4488253_Mouse_basal_stim_IFNB_10_REP2_TPM.txt.gz │ │ ├── GSM4488254_Mouse_basal_stim_IFNB_10_REP3_TPM.txt.gz │ │ ├── GSM4488255_Mouse_basal_stim_IFNG_0.1_REP1_TPM.txt.gz │ │ ├── GSM4488256_Mouse_basal_stim_IFNG_0.1_REP2_TPM.txt.gz │ │ ├── GSM4488257_Mouse_basal_stim_IFNG_0.1_REP3_TPM.txt.gz │ │ ├── GSM4488258_Mouse_basal_stim_IFNG_0.5_REP1_TPM.txt.gz │ │ ├── GSM4488259_Mouse_basal_stim_IFNG_0.5_REP2_TPM.txt.gz │ │ ├── GSM4488260_Mouse_basal_stim_IFNG_0.5_REP3_TPM.txt.gz │ │ ├── GSM4488261_Mouse_basal_stim_IFNG_1_REP1_TPM.txt.gz │ │ ├── GSM4488262_Mouse_basal_stim_IFNG_1_REP2_TPM.txt.gz │ │ ├── GSM4488263_Mouse_basal_stim_IFNG_1_REP3_TPM.txt.gz │ │ ├── GSM4488264_Mouse_basal_stim_IFNG_2_REP1_TPM.txt.gz │ │ ├── GSM4488265_Mouse_basal_stim_IFNG_2_REP2_TPM.txt.gz │ │ ├── GSM4488266_Mouse_basal_stim_IFNG_2_REP3_TPM.txt.gz │ │ ├── GSM4488267_Mouse_basal_stim_IFNG_5_REP1_TPM.txt.gz │ │ ├── GSM4488268_Mouse_basal_stim_IFNG_5_REP2_TPM.txt.gz │ │ ├── GSM4488269_Mouse_basal_stim_IFNG_5_REP3_TPM.txt.gz │ │ ├── GSM4488270_Mouse_basal_stim_IFNG_10_REP1_TPM.txt.gz │ │ ├── GSM4488271_Mouse_basal_stim_IFNG_10_REP2_TPM.txt.gz │ │ ├── GSM4488272_Mouse_basal_stim_IFNG_10_REP3_TPM.txt.gz │ │ ├── GSM4488273_Mouse_basal_stim_Untreated_0_REP1_TPM.txt.gz │ │ ├── GSM4488274_Mouse_basal_stim_Untreated_0_REP2_TPM.txt.gz │ │ ├── GSM4488275_Mouse_basal_stim_Untreated_0_REP3_TPM.txt.gz │ │ ├── GSM4488276_Mouse_basal_stim_Untreated_0_REP4_TPM.txt.gz │ │ ├── GSM4488277_Mouse_basal_stim_Untreated_0_REP5_TPM.txt.gz │ │ ├── GSM4488278_Mouse_basal_stim_Untreated_0_REP6_TPM.txt.gz │ │ ├── GSM4488279_Mouse_basal_stim_Untreated_0_REP7_TPM.txt.gz │ │ ├── GSM4488280_Mouse_basal_stim_Untreated_0_REP8_TPM.txt.gz │ │ ├── GSM4488281_Mouse_basal_stim_Untreated_0_REP9_TPM.txt.gz │ │ ├── GSM4488282_Mouse_basal_stim_Untreated_0_REP10_TPM.txt.gz │ │ ├── GSM4488283_Mouse_basal_stim_Untreated_0_REP11_TPM.txt.gz │ │ └── GSM4488284_Mouse_basal_stim_Untreated_0_REP12_TPM.txt.gz │ └── GSE148829_RAW.tar ├── GSE75073 │ ├── GSE75073_RAW.tar │ ├── GSE75073_Signals_450k.txt.gz │ └── GSE75073_Signals_Epic.txt.gz ├── GSE76073 │ ├── GSE76073_Normalized_counts.txt.gz │ └── GSE76073_Raw_counts.txt.gz ├── GSE923_series_matrix.txt.gz ├── Images │ ├── 2020-04-20CORT.png │ ├── 20200403_citrul.png │ ├── 20200404_citrul.png │ ├── 20200404_ifabp.png │ ├── Bioconductor.png │ ├── Biostrings.png │ ├── BrowseSeq │ │ ├── Dia1.PNG │ │ └── sequence_alignment_plots │ │ ├── Dia1.PNG │ │ ├── Dia2.PNG │ │ ├── Dia3.PNG │ │ ├── Dia4.PNG │ │ ├── Dia5.PNG │ │ ├── Dia6.PNG │ │ ├── Dia7.PNG │ │ └── Dia8.PNG │ ├── DESeq2.png │ ├── Dia1.JPG │ ├── Dia3.JPG │ ├── GenomicRanges.png │ ├── Git_Setup_1.JPG │ ├── Glimma.png │ ├── IRanges.png │ ├── MSnbase.png │ ├── Project_1.jpg │ ├── Project_2.jpg │ ├── Rlogo.png │ ├── Rtools.png │ ├── bioconductorlogo.jpg │ ├── cat_sleeping_on_keyboard.jpg │ ├── citrulline.png │ ├── connectome-1.png │ ├── cortline.png │ ├── covid19_ziekenhuis.png │ ├── dotdotdot_rotate_axis.png │ ├── edgeR.png │ ├── exercise_2_1_1.png │ ├── exercise_2_1_1_c.png │ ├── forcats.png │ ├── git_web.png │ ├── gitinstallermac.png │ ├── heatmap.png │ ├── heatmap1.png │ ├── heatmap2.png │ ├── hex-ggplot2.png │ ├── installpackages.png │ ├── loadpackages.png │ ├── logo_R.jpg │ ├── ma1.png │ ├── ma2.png │ ├── mzR_hl.png │ ├── new_proj_done.png │ ├── new_proj_git.png │ ├── one-doesnot-simply.jpg │ ├── open_486.jpg │ ├── open_compaq.jpg │ ├── pca1.png │ ├── pepper.jpg │ ├── pipe.png │ ├── purrr_small.png │ ├── purrr_sticker.jpg │ ├── pvalues.png │ ├── r4ds.png │ ├── readr_sticker.png │ ├── readxl_sticker.png │ ├── rmac_download.png │ ├── rstudio_download.png │ ├── rstudio_leeg.png │ ├── rwin_download.png │ ├── security.png │ ├── setup-wizard-rstudio.png │ ├── setupRtools.png │ ├── stringr.png │ ├── summarizedexperiment.png │ ├── tidyr_sticker.png │ ├── tidyverse_sticker.png │ ├── usrlocal.png │ ├── wizardRmac.png │ ├── wizardRwindows.png │ ├── wontinstall.png │ ├── workflow_course.png │ ├── xquarkdisk.png │ ├── xquarkwizard.png │ ├── xquartz.png │ └── yoda_the_last_jedi.jpg ├── John.csv ├── Les 6.Rmd ├── Les4.Rmd ├── Les4.nb.html ├── Les5.Rmd ├── Lesson_5_Exercise_a1.Rmd ├── Mike.csv ├── R │ └── do_tidy_pertussis.R ├── README.md ├── Steve.csv ├── answers │ ├── daur2_lesson51_demo_multiple_sequence_alignment.Rmd │ ├── daur2_lesson51_demo_multiple_sequence_alignment.html │ ├── daur2_lesson51_demo_multiple_sequence_alignment_files │ │ └── figure-html │ │ ├── unnamed-chunk-23-1.png │ │ ├── unnamed-chunk-24-1.png │ │ ├── unnamed-chunk-29-1.png │ │ └── unnamed-chunk-30-1.png │ ├── daur2_lesson_6_exercise_rnaseq.Rmd │ ├── daur2_lesson_6_exercise_rnaseq.html │ ├── daur2_lesson_6_exercise_rnaseq_student_version.Rmd │ └── daur2_lesson_6_exercise_rnaseq_student_version.html ├── caspase_align.aux ├── caspase_align.log ├── caspase_align.tex ├── code │ └── load_data.R ├── data │ ├── CISID_pertussis_10082018.csv │ ├── biomaRt_homology_example.rds │ ├── biomaRt_homology_example_aa.rds │ ├── biomaRt_homology_example_dna.rds │ ├── chircus_blg_aminoacids.fasta │ ├── course_r_packages.tsv │ ├── course_r_packages.txt │ ├── covid_rivm │ ├── go_bp-hit.csv │ ├── go_cc-hit.csv │ ├── go_mf-hit.csv │ ├── hobbit_chapter.txt │ ├── hobbit_dwarves.txt │ ├── hox_aa_sequences.txt │ ├── hox_dna_sequences.txt │ ├── human_hoxb1_protein.fasta │ ├── human_hoxb1_protein.gb │ ├── human_rhodopsin_protein.fasta │ ├── human_ttn_genomic_dna.fasta │ ├── jp_coldata.txt │ ├── jp_counts.txt │ ├── jp_features.txt │ ├── jp_miame.rds │ ├── jp_miame.txt │ ├── lesson2 │ │ ├── ENST00000642929.1.fa │ │ ├── ENST00000643997.1.fa │ │ ├── ENST00000644264.1.fa │ │ ├── ENST00000644696.1.fa │ │ ├── ENST00000644919.1.fa │ │ ├── ENST00000646017.1.fa │ │ ├── ENST00000646618.2.fa │ │ ├── ENST00000646644.1.fa │ │ ├── ENST00000647220.1.fa │ │ ├── ENST00000647443.1.fa │ │ ├── cholesterol.xlsx │ │ ├── diet │ │ │ ├── Andy.csv │ │ │ ├── David.csv │ │ │ ├── John.csv │ │ │ ├── Mike.csv │ │ │ └── Steve.csv │ │ ├── diet_data.zip │ │ └── sequence1.txt │ └── lesson3 │ ├── GO-terms.txt │ ├── dna_random │ ├── sequence.fasta │ ├── sequence1.txt │ └── sequence2.txt ├── daurII_students │ ├── R │ │ └── do_tidy_pertussis.R │ ├── README.md │ ├── code │ │ └── load_data.R │ ├── data │ │ ├── CISID_pertussis_10082018.csv │ │ ├── course_r_packages.tsv │ │ ├── covid_rivm │ │ ├── go_bp-hit.csv │ │ ├── go_cc-hit.csv │ │ ├── go_mf-hit.csv │ │ ├── hobbit_chapter.txt │ │ ├── hobbit_dwarves.txt │ │ ├── hox_aa_sequences.txt │ │ ├── hox_dna_sequences.txt │ │ ├── human_hoxb1_protein.fasta │ │ ├── human_hoxb1_protein.gb │ │ ├── human_rhodopsin_protein.fasta │ │ ├── human_ttn_genomic_dna.fasta │ │ └── lesson2 │ │ └── diet_data.zip │ ├── daurII_students.Rproj │ └── demos │ └── 01_demo_functions.Rmd ├── daurII_students.Rproj ├── demos │ ├── 01_demo_functions.Rmd │ ├── 021_demo_2_agenda.Rmd │ ├── 021_demo_2_agenda.html │ ├── 022_lesson2_map_nest.Rmd │ ├── 022_lesson2_map_nest.html │ ├── 031_strings_and_regex.Rmd │ ├── 031_strings_and_regex.html │ ├── 031_strings_and_regex.pdf │ ├── 041_biomart.Rmd │ ├── 042_geoquery.Rmd │ ├── 043_rentrez.Rmd │ ├── 051_demo_multiple_sequence_alignment.Rmd │ ├── 051_demo_multiple_sequence_alignment.html │ ├── 051_demo_multiple_sequence_alignment.md │ ├── 051_demo_multiple_sequence_alignment_files │ │ └── figure-html │ │ ├── unnamed-chunk-23-1.png │ │ ├── unnamed-chunk-24-1.png │ │ ├── unnamed-chunk-29-1.png │ │ └── unnamed-chunk-30-1.png │ ├── 052_demo_summarized_experiment.Rmd │ ├── 052_demo_summarized_experiment.html │ ├── 052_demo_summarized_experiment.md │ ├── 61_heatmaps.Rmd │ ├── 61_heatmaps.html │ ├── 62_rnseq_workflow.Rmd │ └── 62_rnseq_workflow.html ├── les1.Rmd ├── les1.nb.html ├── les2.Rmd ├── les2.nb.html ├── les3.Rmd ├── my_transcript.fasta ├── resources.Rmd └── unzip ├── Andy.csv ├── David.csv ├── John.csv ├── Mike.csv └── Steve.csv </div> </div> ::: As you can see, the original file structure used during the earlier data science course is quite the mess. Instead we now make usage of the Guerilla analytics framework to store data and manage files. Using simple structures to store data, code, generated graphs and makes it easier to use and replicate analyses, even years after the analyses are performed. Making sure all files in a environment pertain only to that single project is important. And to make data storage simple, a central data folder in the root of the enviornment is used with clear naming conventions, dates and versions. Also metadata files are used to allow anyone with access to the project to see what the contents of the project are. Here is the above file structure, with the guerilla analytics framework retroactively applied: :::{.card} <div class="FixedHeightContainer"> <h3>New folder structure:</h3> <div class="Content"> ├── EXAM │ ├── Alex_Groot copy.csv │ ├── capstone_exam.Rmd │ ├── capstone_exam.html │ └── capstone_exam.md ├── Images │ ├── 2020-04-20CORT.png │ ├── 20200403_citrul.png │ ├── BrowseSeq │ │ ├── Dia1.PNG │ │ └── sequence_alignment_plots │ │ ├── Dia1.PNG │ │ ├── Dia2.PNG │ └── yoda_the_last_jedi.jpg ├── R │ └── do_tidy_pertussis.R ├── README.md ├── caspase_align.aux ├── caspase_align.log ├── caspase_align.tex ├── code │ └── load_data.R ├── data_raw │ ├── lesson2 │ │ ├── ENST00000642929.1.fa │ │ ├── ENST00000643997.1.fa │ │ ├── ENST00000644264.1.fa │ │ ├── cholesterol.xlsx │ │ ├── diet │ │ │ ├── Andy.csv │ │ │ └── Steve.csv │ │ ├── diet_data.zip │ │ └── sequence1.txt │ ├── lesson3 │ │ ├── GO-terms.txt │ │ ├── dna_random │ │ ├── sequence.fasta │ │ ├── sequence1.txt │ │ └── sequence2.txt │ └── unsorted │ ├── Andy.csv │ ├── CISID_pertussis_10082018.csv │ ├── David.csv │ ├── GSE116936 │ │ └── GSE116936_RAW.tar │ ├── GSE116936_series_matrix.txt.gz │ ├── GSE116936_supp_files │ │ ├── GSM3264576_A1-Ctrl.txt.gz │ │ ├── GSM3264577_A2-Ctrl.txt.gz │ │ ├── GSM3264578_A3-Ctrl.txt.gz │ │ ├── GSM3264585_J1-TNF+PGE2.txt.gz │ │ ├── GSM3264586_J2-TNF+PGE2.txt.gz │ │ └── GSM3264587_J3-TNF+PGE2.txt.gz │ ├── GSE148829 │ │ ├── GSE148829_BEAS_Basal_Pops_TPM.txt.gz │ │ ├── GSE148829_Human1_Basal_Pops_TPM.txt.gz │ │ ├── GSE148829_Human2_Basal_Pops_TPM.txt.gz │ │ │ ├── GSM4487938_Mouse_Nasal_IFNa_NH11_dge.txt.gz │ │ │ ├── GSM4487939_Mouse_Nasal_IFNa_NH12_dge.txt.gz │ │ │ ├── GSM4487940_Mouse_Nasal_IFNa_NH21_dge.txt.gz │ │ │ ├── GSM4487941_Mouse_Nasal_IFNa_NH22_dge.txt.gz │ │ │ ├── GSM4487942_Mouse_Nasal_IFNa_NS11_dge.txt.gz │ │ │ ├── GSM4487943_Mouse_Nasal_IFNa_NS12_dge.txt.gz │ │ │ ├── GSM4487944_Mouse_Nasal_IFNa_NS21_dge.txt.gz │ │ │ ├── GSM4487945_Mouse_Nasal_IFNa_NS22_dge.txt.gz │ │ │ ├── GSM4488282_Mouse_basal_stim_Untreated_0_REP10_TPM.txt.gz │ │ │ ├── GSM4488283_Mouse_basal_stim_Untreated_0_REP11_TPM.txt.gz │ │ │ └── GSM4488284_Mouse_basal_stim_Untreated_0_REP12_TPM.txt.gz │ │ └── GSE148829_RAW.tar │ ├── GSE75073 │ │ ├── GSE75073_RAW.tar │ │ ├── GSE75073_Signals_450k.txt.gz │ │ └── GSE75073_Signals_Epic.txt.gz │ ├── GSE76073 │ │ ├── GSE76073_Normalized_counts.txt.gz │ │ └── GSE76073_Raw_counts.txt.gz │ ├── GSE923_series_matrix.txt.gz │ ├── human_hoxb1_protein.fasta │ ├── human_hoxb1_protein.gb │ ├── human_rhodopsin_protein.fasta │ ├── human_ttn_genomic_dna.fasta │ ├── jp_coldata.txt │ ├── jp_counts.txt │ ├── jp_features.txt │ ├── jp_miame.txt │ ├── my_transcript.fasta │ └── unzip │ ├── Andy.csv │ ├── David.csv │ ├── John.csv │ ├── Mike.csv │ └── Steve.csv ├── data_rds │ ├── biomaRt_homology_example.rds │ ├── biomaRt_homology_example_aa.rds │ ├── biomaRt_homology_example_dna.rds │ └── jp_miame.rds ├── daurII_students │ ├── R │ │ └── do_tidy_pertussis.R │ ├── README.md │ ├── code │ │ └── load_data.R │ ├── data │ │ ├── CISID_pertussis_10082018.csv │ │ ├── course_r_packages.tsv │ │ ├── covid_rivm │ │ ├── human_rhodopsin_protein.fasta │ │ ├── human_ttn_genomic_dna.fasta │ │ └── lesson2 │ │ └── diet_data.zip │ ├── daurII_students.Rproj │ └── demos │ └── 01_demo_functions.Rmd ├── daurII_students.Rproj ├── demos │ ├── 01_demo_functions.Rmd │ ├── 021_demo_2_agenda.Rmd │ ├── 021_demo_2_agenda.html │ ├── 61_heatmaps.Rmd │ ├── 61_heatmaps.html │ ├── 62_rnseq_workflow.Rmd │ └── 62_rnseq_workflow.html └── markdown ├── Alex_Groot_1730201_Capstone_Exam_ABDS_DAUR2_BIOC.Rmd ├── Alex_Groot_1730201_Capstone_Exam_ABDS_DAUR2_BIOC.html ├── Les 6.Rmd ├── Les4.Rmd ├── Les4.nb.html ├── Les5.Rmd ├── Lesson_5_Exercise_a1.Rmd ├── answers │ ├── daur2_lesson51_demo_multiple_sequence_alignment.Rmd │ ├── daur2_lesson51_demo_multiple_sequence_alignment.html │ ├── daur2_lesson51_demo_multiple_sequence_alignment_files │ │ └── figure-html │ │ ├── unnamed-chunk-23-1.png │ │ ├── unnamed-chunk-24-1.png │ │ ├── unnamed-chunk-29-1.png │ │ └── unnamed-chunk-30-1.png │ ├── daur2_lesson_6_exercise_rnaseq.Rmd │ ├── daur2_lesson_6_exercise_rnaseq.html │ ├── daur2_lesson_6_exercise_rnaseq_student_version.Rmd │ └── daur2_lesson_6_exercise_rnaseq_student_version.html ├── les1.Rmd ├── les1.nb.html ├── les2.Rmd ├── les2.nb.html ├── les3.Rmd └── resources.Rmd </div> </div> ::: In order to have the folder structured in the Guerrilla analytics framework method, I retroactively moved the files. This is just used as an example to show what a tidier and better organized structure might look like. The big problem with this is that it will break many, if not all, of the scripts inside due to changing of file paths. The best way to keep files well-managed is to set up a file structure before starting and sticking to it. Keeping a folder with all the raw data with its names the way they were obtained makes sure confusion is kept to a minimum when the data is looked at. Creating seperate data folders enables to track transformed data while preserving the raw data. And keeping metadata files in all the folders that explain not just folder structure, but importantly also file naming conventions and potentially used abbreviations for them will mean that even years later someone will be able to look at the project and learn what is inside. This portfolio was made with the Guerrilla analytics framework in mind from the start: ```{r, echo=FALSE} # fs::dir_tree(here::here()) ``` ::: {.card} <div class="FixedHeightContainer"> <h3>Old folder structure:</h3> <div class="Content"> <pre> /home/alexgroot/dsfb2_quarto ├── 001_CV.qmd ├── 002_Looking_ahead.qmd ├── 003_Guerrilla_analytics_framework.qmd ├── 004_C_elegans.qmd ├── 005_open_peer_review.qmd ├── 006_relational_databases.qmd ├── 007_R_Package.qmd ├── 008_Zotero.qmd ├── 008_Zotero.quarto_ipynb ├── 009_Parameterized_report.qmd ├── 010_ML_Getting_started.qmd ├── 404.qmd ├── LICENSE ├── _quarto.yml ├── _site ├── about.qmd ├── css │ └── 008_Zotero.css ├── data_output │ ├── data_00100 │ └── data_0060 │ ├── dengue.csv │ ├── dengue.rds │ ├── flu.csv │ ├── ... ├── data_raw │ ├── data_00100 │ │ ├── README │ │ ├── breast_cancer │ │ │ ├── Index │ │ │ ├── breast-cancer.data │ │ │ └── breast-cancer.names │ │ └── heart_failure.csv │ ├── data_0040 │ │ ├── CE.LIQ.FLOW.062_Tidydata.xlsx │ │ └── README │ ├── data_0050 │ │ ├── README │ │ ├── TMNT Combined Data OSF.csv │ │ ├── TMNT Meta-analysis OSF.csv │ │ ├── ... │ │ ├── osfstorage-archive │ │ │ ├── TMNT Combined Data OSF.csv │ │ │ ├── TMNT Meta-analysis OSF.csv │ │ │ ├── ... │ │ │ └── readme.txt │ │ ├── osfstorage-archive.zip │ │ └── readme.txt │ ├── data_0060 │ │ ├── README │ │ ├── dengue_data.csv │ │ └── flu_data.csv │ └── data_0090 │ └── data.csv ├── dsfb2_quarto.Rproj ├── images │ ├── data_00100 │ │ ├── baseline_accuracy_model_performance.png │ │ └── baseline_precision_model_performance.png │ ├── data_0050 │ │ └── original_plot.png │ └── projects │ ├── 003_Guerrilla_analytics_framework.png │ ├── 006_relational_databases.png │ ├── ... ├── index.qmd ├── projects.ejs ├── projects.qmd ├── references │ └── 008_nanopore.bib ├── scripts │ └── 009_Parameterized_report.R └── styles.css </pre> </div> </div> ::: :::{.card} <ul> <li> All the required main files are in the root folder of the project, as required for the website to render, e.g. the quarto documents, yml, and project files. </li> <li> Next there are extra folders required by the rendered site; images, css, _site, references.</li> <li> Then folders used for the R chunks and scripts; data_raw/output, scripts.</li> <li> Finally Github related files such as .gitignore and LICENSE.</li> <li> Structured systems as such make it easier to both find old files and place new ones. </li> </ul> :::

Other Links

Code Links

Guerrilla analytics

Old folder structure:

New folder structure:

Old folder structure: