project_whispertrans_code/README.md

Project_whisperTrans_Code

Overview

whisperTrans is a video2transcript tool built upon Whisper and WhisperX. For automated speech recognition (ASR). The project provides an efficient implementation for running transcriptions locally on open source models from local files and YouTube videos. It is structured to handle large-scale audio transcription tasks with accuracy and ease using diarization and alignment techniques.

Setup

The project is tested under python 3.10.12. In order to edit video material ffmpeg is required. Furthermore, machine learning models are employed in this project to convert audio content to text. Which models are used depends on the language of the video content. The models are downloaded on demand from the model hosting platform huggingface. To do so you need to create an account and set the environment variable HF_TOKEN to your token. This is necessary to accept the conditions of the models used.

Before running this script, make sure to install the necessary python packages. We recommend using a virtual environment to manage dependencies. You can install the required packages with the following command:

pip install -r requirements.txt  --index-url https://download.pytorch.org/whl/cu118

If your system does not have ffmpeg installed you can install it via:

sudo apt install ffmpeg

For GPU usage you need to install the CUDA toolkit. You can install it via:

sudo apt install nvidia-cuda-toolkit-11-8

This might require to add the package the following way:

wget https://developer.download.nvidia.com/compute/cuda/11.8.0/local_installers/cuda-repo-ubuntu2204-11-8-local_11.8.0-520.61.05-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu2204-11-8-local_11.8.0-520.61.05-1_amd64.deb
sudo cp /var/cuda-repo-ubuntu2204-11-8-local/cuda-*-keyring.gpg /usr/share/keyrings/

Afterward, set the environmental variables:

echo 'export PATH=/usr/local/cuda-11.8/bin${PATH:+:${PATH}}' >> ~/.bashrc
echo 'export LD_LIBRARY_PATH=/usr/local/cuda-11.8/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}' >> ~/.bashrc

and reload your ~/.bashrc file to apply these changes:

source ~/.bashrc

Prerequisites

• Python 3.10
• ffmpeg
• Access to Hugging Face for model downloads
• CUDA 11.8, and cuBLAS 11.x and cuDNN 8.x
• CUDA-compatible GPU with <8GB VRAM (recommended for faster processing)

Usage

To generate transcripts create a new job with your specifications (see Jobs secion below).
If you need to download the transcriber model set your huggingface token:

On Linux/Mac:

export HF_TOKEN="your-token-here"

On Windows (Command prompt):

set HF_TOKEN="your-token-here"

After setting up the token, execute the script with:

python main.py --job {name}

Here, name is the identifier for your transcription job. Note that you might need to go to huggingface to accept the conditions of the models used. Especially when running the transcriber for the first time on your huggingface account.

Jobs

Jobs specify tasks for transcript creation. A job contains the following information.

• name: string: The name of the job
• source: List[string]: The sources of the videos (paths, YouTube links)
• language: Optional[string]: The language of the video (If not specified employs auto-detection) as a language code (e.g. 'en', 'de', 'fr')
• output_format: dict: The requested output formats
  • get_json_version: bool: Whether to get the full data json version of the transcript (including word level time stamps)
  • get_txt_version: bool: Whether to get a plain text version of the transcript
  • get_sentence_level_json_version: bool: Whether to get a sentence level json version of the transcript

Contributing

Contributions are welcome! Please fork the repository and make your changes in a separate branch. Submit a pull request with your proposed modifications. For major changes, please open an issue first to discuss what you would like to change.

License

This project is licensed under the MIT License. See the LICENSE file for details.

Contact

This project is maintained by Gerrit Anders. For any questions or support, please open an issue on this repository or reach out via email at g.anders@iwm-tuebingen.de.

Future plans

• Tib-AV Downloader: Add a downloader capability for TibAV video links.
• Progress Bar: Implement a progress bar for better user feedback during processing.
• Local Model Directory: Allow models to be stored locally for offline use and faster processing.
• Extend Mock Tests: Add actual references for tests.

Structure

project_root/
├── src/
│   ├── video_manager.py   # Contains VideoManager class
│   ├── transcriber.py    # Contains Transcriber class
│   ├── transcript_generator.py    # Contains TranscriptGenerator class
│   └── data_handling.py  # Contains download_audio and convert_to_wav methods
├── tests/    
│   ├── test_file_manager.py
│   ├── test_transcriber.py
│   └── test_transcript_generator.py
├── data/
│   ├── logs/  # logs of previous jobs
│   ├── jobs/   # place to store job files
│   └── results/   # place to store results
├── main.py    # Main script that runs the process
├── requirements.txt
├── LICENSE
└── README.md