150 lines
5.2 KiB
Markdown
150 lines
5.2 KiB
Markdown
# Project_whisperTrans_Code
|
|
|
|
## Overview
|
|
|
|
whisperTrans is a video2transcript tool built upon [Whisper](https://github.com/openai/whisper)
|
|
and [WhisperX](https://github.com/m-bain/whisperX). For automated speech recognition (ASR).
|
|
The project provides an efficient implementation for running transcriptions locally on open source models from local
|
|
files and YouTube videos. It is structured to handle large-scale audio transcription tasks with accuracy and ease using
|
|
diarization and alignment techniques.
|
|
|
|
## Setup
|
|
|
|
The project is tested under python 3.10.12.
|
|
In order to edit video material ffmpeg is required. Furthermore, machine learning models are employed in this project to
|
|
convert audio content to text. Which models are used depends on the language of the video content. The models are
|
|
downloaded on demand from the model hosting platform [huggingface](https://huggingface.co/). To do so you need to create
|
|
an account and set the environment variable HF_TOKEN to your token. This is necessary to accept the conditions of the
|
|
models used.
|
|
|
|
Before running this script, make sure to install the necessary python packages. We recommend using a virtual environment
|
|
to manage dependencies. You can install the required packages with the following command:
|
|
|
|
```bash
|
|
pip install -r requirements.txt --index-url https://download.pytorch.org/whl/cu118
|
|
```
|
|
|
|
If your system does not have ffmpeg installed you can install it via:
|
|
|
|
```bash
|
|
sudo apt install ffmpeg
|
|
```
|
|
|
|
For GPU usage you need to install the CUDA toolkit. You can install it via:
|
|
|
|
```bash
|
|
sudo apt install nvidia-cuda-toolkit-11-8
|
|
```
|
|
|
|
This might require to add the package the following way:
|
|
|
|
```bash
|
|
wget https://developer.download.nvidia.com/compute/cuda/11.8.0/local_installers/cuda-repo-ubuntu2204-11-8-local_11.8.0-520.61.05-1_amd64.deb
|
|
sudo dpkg -i cuda-repo-ubuntu2204-11-8-local_11.8.0-520.61.05-1_amd64.deb
|
|
sudo cp /var/cuda-repo-ubuntu2204-11-8-local/cuda-*-keyring.gpg /usr/share/keyrings/
|
|
```
|
|
|
|
Afterward, set the environmental variables:
|
|
|
|
```bash
|
|
echo 'export PATH=/usr/local/cuda-11.8/bin${PATH:+:${PATH}}' >> ~/.bashrc
|
|
echo 'export LD_LIBRARY_PATH=/usr/local/cuda-11.8/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}' >> ~/.bashrc
|
|
```
|
|
|
|
and reload your `~/.bashrc` file to apply these changes:
|
|
|
|
```bash
|
|
source ~/.bashrc
|
|
```
|
|
|
|
## Prerequisites
|
|
|
|
- Python 3.10
|
|
- ffmpeg
|
|
- Access to Hugging Face for model downloads
|
|
- CUDA 11.8, and cuBLAS 11.x and cuDNN 8.x
|
|
- CUDA-compatible GPU with <8GB VRAM (recommended for faster processing)
|
|
|
|
## Usage
|
|
|
|
To generate transcripts create a new job with your specifications (see [Jobs secion](#jobs) below).
|
|
<br/>
|
|
If you need to download the transcriber model set your huggingface token:
|
|
|
|
On Linux/Mac:
|
|
|
|
```bash
|
|
export HF_TOKEN="your-token-here"
|
|
```
|
|
|
|
On Windows (Command prompt):
|
|
|
|
```cmd
|
|
set HF_TOKEN="your-token-here"
|
|
```
|
|
|
|
After setting up the token, execute the script with:
|
|
|
|
```bash
|
|
python main.py --job {name}
|
|
```
|
|
|
|
Here, `name` is the identifier for your transcription job.
|
|
Note that you might need to go to huggingface to accept the conditions of the models used. Especially when running the
|
|
transcriber for the first time on your huggingface account.
|
|
|
|
## Jobs
|
|
|
|
Jobs specify tasks for transcript creation. A job contains the following information.
|
|
|
|
- `name`: string: The name of the job
|
|
- `source`: List[string]: The sources of the videos (paths, YouTube links)
|
|
- `language`: Optional[string]: The language of the video (If not specified employs auto-detection) as a language code
|
|
(e.g. 'en', 'de', 'fr')
|
|
- `output_format`:
|
|
|
|
## Contributing
|
|
|
|
Contributions are welcome! Please fork the repository and make your changes in a separate branch. Submit a pull request
|
|
with your proposed modifications. For major changes, please open an issue first to discuss what you would like to
|
|
change.
|
|
|
|
## License
|
|
|
|
This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.
|
|
|
|
## Contact
|
|
|
|
This project is maintained by [Gerrit Anders](https://www.iwm-tuebingen.de/www/de/personen/ma.html?uid=ganders).
|
|
For any questions or support, please open an issue on this repository or reach out via email
|
|
at [g.anders@iwm-tuebingen.de](mailto:g.anders@iwm-tuebingen.de).
|
|
|
|
## Future plans
|
|
|
|
- **Tib-AV Downloader**: Add a downloader capability for [TibAV](https://av.tib.eu/) video links.
|
|
- **Progress Bar**: Implement a progress bar for better user feedback during processing.
|
|
- **Local Model Directory**: Allow models to be stored locally for offline use and faster processing.
|
|
- **Extend Mock Tests**: Add actual references for tests.
|
|
|
|
## Structure
|
|
|
|
```
|
|
project_root/
|
|
├── src/
|
|
│ ├── video_manager.py # Contains VideoManager class
|
|
│ ├── transcriber.py # Contains Transcriber class
|
|
│ ├── transcript_generator.py # Contains TranscriptGenerator class
|
|
│ └── data_handling.py # Contains download_audio and convert_to_wav methods
|
|
├── tests/
|
|
│ ├── test_file_manager.py
|
|
│ ├── test_transcriber.py
|
|
│ └── test_transcript_generator.py
|
|
├── data/
|
|
│ ├── logs/ # logs of previous jobs
|
|
│ ├── jobs/ # place to store job files
|
|
│ └── results/ # place to store results
|
|
├── main.py # Main script that runs the process
|
|
├── requirements.txt
|
|
├── LICENSE
|
|
└── README.md
|
|
``` |