project_whispertrans_code/README.md

Project_whisperTrans01_Code

Content

Transcriber code

Setup

Before running this script, make sure to install the necessary python packages using the following command:

pip install -r requirements.txt

Running the script

To generate transcripts create a new job with your specifications (see jobs section below).
If you need to download the transcriber model (not saved in data/models yet) set your huggingface token:

On Linux/Mac:

export HF_TOKEN="your-token-here"

On Windows (Command prompt):

set HF_TOKEN="your-token-here"

Then run the following command to generate your transcript:

main.py --job {name}

where name is the name of your job.

Jobs

Jobs specify tasks for transcript creation. A job contains the following information.

• name: string: The name of the job
• source: List[string]: The sources of the videos (paths, YouTube links, TibAV links)
• modelpath: Optional[string]: The path to the model if already saved locally (will be downloaded temporarily if not)
• language: Optional[string]: The language of the video (If not specified employs auto-detection)

to-do

• seperate data production form code
• remember to accept the conditions on diarization if you use huggingface
• do a config json
• do option to have models already stored and point to them (is this necessary? > if you do not want to interact with huggingface yes)
  • also for single language models
• check warnings
• downloader for tib-av
• more optional setting
• add progress bar

Structure

project_root/
├── src/
│   ├── file_manager.py   # Contains FileManager class
│   ├── transcriber.py    # Contains Transcriber class
│   ├── transcript_generator.py    # Contains TranscriptGenerator class
│   ├── data_handling.py  # Contains download_audio and convert_to_wav methods
│   └── main.py    # Main script that runs the process
├── tests/    
│   ├── test_file_manager.py
│   ├── test_transcriber.py
│   └── test_transcript_generator.py
├── data/
│   ├── audio/   # downloaded mp4 and output wav files
│   ├── models/   # models used for transcripts (are ignored by git and not necessarily)
│   └── transcripts/  # JSON and text transcript files
└── README.md

• src/: This directory will contain all your source code. Inside the src folder, I added:

  • main.py: This will be the main entry point for the program. It will use the TranscriptGenerator class which orchestrates the use of FileManager and Transcriber.
  • file_manager.py: Contains the FileManager class implementation.
  • transcriber.py: Contains the Transcriber class implementation.
  • transcript_generator.py: Contains the TranscriptGenerator class implementation.

• tests/: This directory contains all your test scripts corresponding to your source files.

• data/: This directory contains all the data needed for your code. For simplicity, I categorize the files into separate directories for audio, model and transcript files.