# Base1

Scripts to prepare BASE-I data for transfer to ZPID

# Criteria for checking variables

- Value range (min-max)
- NA-labelling
- Outliers (distribution)
- data type
- language value-labels
- direction of coding (reversed items)
- consistency with documentation
- at which time points available

# Notes 2021-09-02

- generate tentative plan on September 23rd


# Notes 2021-08-26

- how are we dealing with missing values? 
- work on an automated way of referring the variables with their translations

# Notes 2021-09-15 Meeting with Katarina Blask

- **How to handle missing values?**

  Use NA for missing values

- **Dealing with outliers?**
  
  Use Tukey's method to define outliers

- **Keep aggregate values?**

  Discard if you don't know about them. However, query Martin Becker to explain how the variables were constructed.

- **Data description in separate file?**

  Use JSON file with the following definition:
  > name, label, instruction (e.g. item wording), valid values, missing values, value label for valid values, value labels for
  missing values and it should also be apparent which measure was used to collect the data presented within the variable either
  by making use of a corresponding naming scheme or adding a metadate “measure/instrument” to each variable description

# Notes 2021-09-17 Meeting with Martin Becker

- **1. What is the significance of the .POR files?**

  The .POR files contain information in addition to the .SAR files. This will also have to be included in the data.

- **2. Is there a standard for coding missing values that is used across all datasets?**
  
  There is no general standard followed across all the datasets.

- **3. Is there a list of participants per time point?**
  
  Yes, Martin will be providing this list.

- **4. Can materials (e.g. a specific questionnaire) be related to variable names?**
  
  Possible in some cases but difficult in others. Martin will provide a lab file that might be helpful.

- **5. Regarding variable constructs: Can we reconstruct the way that constructs were build?**

  Not clear in all cases. But Martin recommends that even in the case where there is no clear description of the constructs, they should not be  
  omitted from the data.

Next meeting on 7th October.

# Notes 2021-09-27: Coming up with ideas

- "participants_at_timepoints.sav" and .LAB files have been added to Seafile
  (as discussed in the meeting with Martin)

- Current sources of information available to us :
  
  - Documentation (HTML files)
  - Lifebrain Excel
  - data files themselves
  - Questionnaires
  - LAB files 
  - Participants per timepoint
  - Martins emails
  <br>
- The final objective is to have a script that generates :
  1. Cleaned dataset (actual data with all the quality checks - most probably in .tsv or .csv format)
  2. Data description (Metadata about all the variables - .json format)
  <br>
- The script should ideally work for both .POR and .SAV files.

- For now, we focus on making a single file containing all the variables along with all the extracted information or metadata. 
  In this file, we can then look at the blank spaces (wherever the information could not be extracted) and try to figure out a way forward.
  
- Rohil will add a complete list of information to be extracted for each variable. This list will be cross-checked and agreed upon by all others.
 
# Notes 2022-03-03 Meeting with Martin Becker

- Martin mentions as additional source if information the manual for the sociological data

- Martin encourage to go from documentation to data when trying to match the two sources

- For T5-T7 there are only psychological data

- For T8 there are psychological data as well as the intake assessment

- There are also genetical data with Lars bertram in Lübeck

[- we can try to install regular meetings if needed]

# Notes 2022-05-31 Meeting with Katharina Blask

## To do:

Rohil & Raafay:

- ~~split data accoding to documentation tables & adapt json files accordingly~~

- insert columns: Research Unit, Domain, Topic, Instrument :white_check_mark: 

- Declare missing labels as such: Insert "missing value: " in description

Julia & Maike:

- Translate value labels

- complete english translations

- link measurement instruments :white_check_mark: 

# Notes 2022-12-13 Meeting with Katharina Blask

- Präsentation des Mockup (https://9514682e-29ee-450e-ae5f-8d2c9e1eef31-base.surge.sh/)

  Rückfragen dazu:

  - Brauchen wir Kennwerte (z.B. mean oder range) in der Preview? 

  - Brauchen wir eine Mehrfachauswahl bei den Domains? Dies führt dazu, dass sehr viele Variable gelistet werden und ist unter UX-Design Aspekten daher eher nicht so schön. Das gleiche gilt für Research Units. _Ich frage Martin Becker, wie die Zusammensetzung bei bisherigen Datenanträgen war._

- Weitere Punkte zu den Daten:

  - Die ID taucht nicht in den Daten auf?_ @Rohil: Woran liegt das?_

  - Haben wir gecheckt, dass Werte nur innerhalb der validen Range sind? Es scheint Diskrepanzen zwischen Daten und Doku zu geben (z.B. s1mu02 hat in der Doku keine -4 in den Daten schon). Und die _Codes sind bisher nicht miteinbezogen worden_. 

  - Es gibt sehr viele leere Zellen in den Daten, woran liegt das? An der Transformation der Daten. _KB sucht dafür eine Lösung._

  - Haben wir für Typos gecheckt, insbesondere bei den Filtervariablen, weil die sonst nicht funktionieren. Nein. ABER, wir fügen diese ja ein!!!

  - Obwohl wir mal entschieden hatten, die deutsche Version wegzulassen, denkt Frau Blask, dass es (für Deutschsprachige) Sinn macht, sie doch drin zu lassen

  - Der "Index" in Spalte "Unnamed: 0" soll in den Metadaten enthalten sein, damit Fragen danach gruppiert werden können.

  - Sämtliche Codes müssen in den relevanten Value labels untergrbracht werden.

  - Sicherstellen, dass wirklich nur die demografischen Variablen keine Inhalte bei Filtervariablen (domain, Topic etc.) haben.

- Material:

   - Sind die Fragebögen alle Eigenkreationen. Wenn nicht, müssen wir wegen des Urheberrechts vorsichtig sein, ob wir die Items überhaupt so abbilden dürfen. Ggf. müssen die Texte herausgenommen und eine Referenz gesetzt werden. _Maike Klärt dies mit Julia Delius._

   - Die Eigenkreationen würden auf PsychArchives veröffentlicht und dann ebenfalls in den Study Materials verlinkt.

- Sonstiges:

   - Wie werden Datenbestellungen bisher zusammengefügt? Gibt es dafür ein Skript? Maie klärt das mit _Martin Becker_.

# Backlog

## Data

- Additional Sozio tables that have more than 2 ID columns

## Documentation

- replace and complete variable names that are indicated by slashes (z6e34x1 /x2 --> z6e34x1 / z6e34x2) 

- Psychologie: die 3 großen Bereiche (von Doku-Startseite als Domain übernehmen)

- Correct measurement instruments in the **final** meta data file (when variables are split by time points)

- Include additional Code files somewhere (e.g. Sozio/c_*.htm) --> after talking to Katharina Blask, these need to be inlcuded in the value labels

- chech that measurement instruments are actually correct?!

## Filters

- group by `Unnamed: 0` to keep variables belonging to the same "Super-Question" together




