Submission scientific data

2025-12-15 13:47:28 +01:00
commit b53e63f57e
107 changed files with 8464 additions and 0 deletions
@@ -0,0 +1,114 @@
+# Preproccessing HMC longitudinal study
+
+[![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0)
+
+This is a Python application designed to preprocess data from the HMC longitudinal study. 
+It focuses on transforming survey data into a structured format suitable for analysis by employing yaml configuration files that document the different questionnaires employed over the waves.
+
+## Features
+- Flexible data preprocessing using YAML configuration files
+- Automatic generation of database documentation (Markdown and PDF)
+- Support for multiple output formats (CSV, SQLite)
+- Processing and validation of scales and composite scores across multiple survey waves
+- Modular architecture for easy extensibility
+
+## Installation
+
+Clone the repository and install using pip:
+
+```bash
+git clone https://gitea.iwm-tuebingen.de/HMC/preprocessing.git
+cd preprocessing
+pip install .
+```
+
+This uses the `pyproject.toml` file for all dependencies and build instructions.
+Note that the project requires Python 3.10 or higher and use of [virtual environments](https://docs.python.org/3/library/venv.html) is recommended.
+
+## Usage
+
+### 1. Global Settings
+
+In order to use the project to process the data you need to first create the global settings file `settings.yaml` in the root directory of the project.
+You can follow the example provided in `settings_example.yaml` to create your own settings file.
+The main settings are used to define the location of the configuration and data files.
+
+### 2. Configuration Files
+
+The project uses YAML configuration files to define the structure of the questionnaires and the processing steps.
+Check the `config` directory to make sure that all required questionnaires are present.
+
+### 3. Running the Preprocessing
+
+To run the preprocessing, you can use the command line interface:
+
+```bash
+python HMC_preprocessing.py
+```
+
+### Output
+
+The preprocessing will generate several output files:
+- sqlite database file `hmc_data.db` containing the processed data in a denormalized format.
+- markdown documentation `database_api_reference.md` containing the documentation of the database schema.
+- pdf documentation `database_api_reference.pdf` containing the documentation of the database schema.
+
+Furthermore each wave will be exported as a separate csv or Excel file in the `results` directory.
+
+
+## Code Structure
+
+Our approach centers on the flexible processing of a wide range of psychological scales and composite measures across multiple survey waves. 
+The project leverages YAML configuration files to describe the structure, scoring, and validation rules for each questionnaire, 
+allowing new scales or response formats to be integrated with minimal code changes.
+
+For each wave, the system reads the relevant configuration, imports the raw data, and applies the specified processing logic, 
+such as item inversion, custom scoring, and subgroup filtering—entirely based on the configuration. 
+This enables researchers to adapt the pipeline to evolving study designs or new measurement instruments without modifying the core codebase.
+
+Processed data from all waves is consolidated into a unified database, and the schema is automatically documented. 
+The modular design ensures that each step—from data import to scale computation and documentation is 
+transparent, reproducible, and easily extensible for future requirements.
+
+In order to achieve this the following modules are used:
+- `settings_loader.py`: Loads and validates global settings
+- `data_loader.py`: Imports raw survey data
+- `scale_processor.py`: Processes individual scales
+- `composite_processor.py`: Computes composite scores (e.g. for the user and non-user groups)
+- `process_all_waves.py`: Orchestrates processing across all waves
+- `database_populator.py`: Exports processed data to the database
+- `database_documentation_generator.py`: Generates database documentation
+- `logging_config.py`: Configures logging for the entire process
+
+## Additional Information
+
+- if a combined scales is used (combining user and no_user items), the group-specific scales are not included in the dataset
+- boolean columns are saved as 0/1 in csv and excel files for better interoperability
+- if single items are retained, they are named as {scale_name}-item_{item_number}
+- in impact_of_delegation_on_skills 6 is coded as NA as it does not fit the answer
+- hope and concern is coded to have higher values for higher hope
+- in wave 1 delegation_comfort item 3 was always NA. > no cronbachs alpha
+
+## Contributing
+
+In order to contribute please follow these steps:
+Before making a pull request:
+- Please review the CONTRIBUTING.md guidelines.
+- Add corresponding unit or integration tests for new code.
+- Ensure your code passes linting and typing checks (black, ruff, mypy).
+
+For development, you can install the package and all dev dependencies in editable mode:
+
+```bash
+pip install -e .[dev]
+```
+
+Note that the project uses [pre-commit](https://pre-commit.com/) to ensure code quality and consistency,
+and that the main branch is protected to ensure that all tests pass before merging.
+
+## License
+
+This code in this project is licensed under the GNU General Public License v3.0. See the [LICENSE](LICENSE) file for details.
+
+## Contact
+For questions, feedback, or to report issues, please contact Gerrit Anders at g.anders@iwm-tuebingen.de.