AG4/project_whispertrans_code
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Refactoring: adapted for changes in cuda, updated README.md, improved logger and added sentence level transcript generation.

Browse Source 
Signed-off-by: Orihara Izaya <xloki.stormbringery@gmail.com>
This commit is contained in:
Orihara Izaya
2024-11-27 16:06:36 +01:00
parent df1a3c075b
commit bff7ceb81c
15 changed files with 576 additions and 237 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+93 -43
View File
@@ -1,15 +1,27 @@
# Project_whisperTrans01_Code
# Project_whisperTrans_Code
## Content
## Overview
Transcriber code
whisperTrans is a video2transcript tool built upon [Whisper](https://github.com/openai/whisper)
and [WhisperX](https://github.com/m-bain/whisperX). For automated speech recognition (ASR).
The project provides an efficient implementation for running transcriptions locally on open source models from local
files and YouTube videos. It is structured to handle large-scale audio transcription tasks with accuracy and ease using
diarization and alignment techniques.
## Setup
Before running this script, make sure to install the necessary python packages using the following command:
The project is tested under python 3.10.12.
In order to edit video material ffmpeg is required. Furthermore, machine learning models are employed in this project to
convert audio content to text. Which models are used depends on the language of the video content. The models are
downloaded on demand from the model hosting platform [huggingface](https://huggingface.co/). To do so you need to create
an account and set the environment variable HF_TOKEN to your token. This is necessary to accept the conditions of the
models used.
Before running this script, make sure to install the necessary python packages. We recommend using a virtual environment
to manage dependencies. You can install the required packages with the following command:
```bash
pip install -r requirements.txt
pip install -r requirements.txt --index-url https://download.pytorch.org/whl/cu118
```
If your system does not have ffmpeg installed you can install it via:
@@ -18,9 +30,44 @@ If your system does not have ffmpeg installed you can install it via:
sudo apt install ffmpeg
```
## Running the script
For GPU usage you need to install the CUDA toolkit. You can install it via:
To generate transcripts create a new job with your specifications (see jobs section below).
```bash
sudo apt install nvidia-cuda-toolkit-11-8
```
This might require to add the package the following way:
```bash
wget https://developer.download.nvidia.com/compute/cuda/11.8.0/local_installers/cuda-repo-ubuntu2204-11-8-local_11.8.0-520.61.05-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu2204-11-8-local_11.8.0-520.61.05-1_amd64.deb
sudo cp /var/cuda-repo-ubuntu2204-11-8-local/cuda-*-keyring.gpg /usr/share/keyrings/
```
Afterward, set the environmental variables:
```bash
echo 'export PATH=/usr/local/cuda-11.8/bin${PATH:+:${PATH}}' >> ~/.bashrc
echo 'export LD_LIBRARY_PATH=/usr/local/cuda-11.8/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}' >> ~/.bashrc
```
and reload your `~/.bashrc` file to apply these changes:
```bash
source ~/.bashrc
```
## Prerequisites
- Python 3.10
- ffmpeg
- Access to Hugging Face for model downloads
- CUDA 11.8, and cuBLAS 11.x and cuDNN 8.x
- CUDA-compatible GPU with <8GB VRAM (recommended for faster processing)
## Usage
To generate transcripts create a new job with your specifications (see [Jobs secion](#jobs) below).
<br/>
If you need to download the transcriber model set your huggingface token:
@@ -36,65 +83,68 @@ On Windows (Command prompt):
set HF_TOKEN="your-token-here"
```
Then run the following command to generate your transcript:
After setting up the token, execute the script with:
```bash
main.py --job {name}
python main.py --job {name}
```
where name is the name of your job.
Here, `name` is the identifier for your transcription job.
Note that you might need to go to huggingface to accept the conditions of the models used. Especially when running the
transcriber for the first time on your huggingface account.
## Jobs
Jobs specify tasks for transcript creation. A job contains the following information.
- `name`: string: The name of the job
- `source`: List[string]: The sources of the videos (paths, YouTube links, TibAV links)
- `modelpath`: Optional[string]: The path to the model if already saved locally (will be downloaded temporarily if not)
- `language`: Optional[string]: The language of the video (If not specified employs auto-detection)
- `source`: List[string]: The sources of the videos (paths, YouTube links)
- `language`: Optional[string]: The language of the video (If not specified employs auto-detection) as a language code
(e.g. 'en', 'de', 'fr')
- `output_format`:
## to-do
## Contributing
- seperate data production form code
- remember to accept the conditions on diarization if you use huggingface
- do a config json
- do option to have models already stored and point to them (is this necessary? > if you do not want to interact with
huggingface yes)
- also for single language models
- check warnings
- downloader for tib-av
- more optional setting
- add progress bar
Contributions are welcome! Please fork the repository and make your changes in a separate branch. Submit a pull request
with your proposed modifications. For major changes, please open an issue first to discuss what you would like to
change.
## License
This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.
## Contact
This project is maintained by [Gerrit Anders](https://www.iwm-tuebingen.de/www/de/personen/ma.html?uid=ganders).
For any questions or support, please open an issue on this repository or reach out via email
at [g.anders@iwm-tuebingen.de](mailto:g.anders@iwm-tuebingen.de).
## Future plans
- **Tib-AV Downloader**: Add a downloader capability for [TibAV](https://av.tib.eu/) video links.
- **Progress Bar**: Implement a progress bar for better user feedback during processing.
- **Local Model Directory**: Allow models to be stored locally for offline use and faster processing.
- **Extend Mock Tests**: Add actual references for tests.
## Structure
```
project_root/
├── src/
│ ├── file_manager.py # Contains FileManager class
│ ├── video_manager.py # Contains VideoManager class
│ ├── transcriber.py # Contains Transcriber class
│ ├── transcript_generator.py # Contains TranscriptGenerator class
│ ├── data_handling.py # Contains download_audio and convert_to_wav methods
│ └── main.py # Main script that runs the process
│ └── data_handling.py # Contains download_audio and convert_to_wav methods
├── tests/
│ ├── test_file_manager.py
│ ├── test_transcriber.py
│ └── test_transcript_generator.py
├── data/
│ ├── audio/ # downloaded mp4 and output wav files
│ ├── models/ # models used for transcripts (are ignored by git and not necessarily)
│ └── transcripts/ # JSON and text transcript files
│ ├── logs/ # logs of previous jobs
│ ├── jobs/ # place to store job files
│ └── results/ # place to store results
├── main.py # Main script that runs the process
├── requirements.txt
├── LICENSE
└── README.md
```
- **src/**: This directory will contain all your source code. Inside the src folder, I added:
- **main.py**: This will be the main entry point for the program. It will use the `TranscriptGenerator` class which
orchestrates the use of `FileManager` and `Transcriber`.
- **file_manager.py**: Contains the `FileManager` class implementation.
- **transcriber.py**: Contains the `Transcriber` class implementation.
- **transcript_generator.py**: Contains the `TranscriptGenerator` class implementation.
- **tests/**: This directory contains all your test scripts corresponding to your source files.
- **data/**: This directory contains all the data needed for your code. For simplicity, I categorize the files into
separate directories for audio, model and transcript files.
```