update

4a689e82 · matteospanio · 00acb567 · 4a689e82 · 4a689e82 · 4a689e82
Commit 4a689e82 authored May 16, 2023 by matteospanio
--- a/.env
+++ b/.env
 PROXY_PORT=8888
 DATA_VOLUME=./data
 LOGS_VOLUME=./logs
 AUDIO_ANALYSER_PORT=50051
 VIDEO_ANALYSER_PORT=50052
 TAPE_IRREGULARITY_PORT=50053
 TAPE_AUDIO_RESTORATION_PORT=50054
 PACKAGER_PORT=50055
\ No newline at end of file
--- a/.gitignore
+++ b/.gitignore
 /data
 /logs
 /opencv_source
 /eq_detection
\ No newline at end of file
+**/__pycache__
\ No newline at end of file
--- a/.gitmodules
+++ b/.gitmodules
 [submodule "packager"]
 	path = packager
 	url = https://gitlab.dei.unipd.it/mpai/packager.git
 [submodule "tape-audio-restoration"]
 	path = tape-audio-restoration
 	url = https://gitlab.dei.unipd.it/mpai/tape-audio-restoration.git
 [submodule "tape-irregularity-classifier"]
 	path = tape-irregularity-classifier
 	url = https://gitlab.dei.unipd.it/mpai/tape-irregularity-classifier.git
 [submodule "video-analyzer"]
 	path = video-analyzer
 	url = https://gitlab.dei.unipd.it/mpai/video-analyzer.git
 [submodule "audio-analyzer"]
 	path = audio-analyzer
 	url = https://gitlab.dei.unipd.it/mpai/audio-analyzer.git
--- a/README.md
+++ b/README.md
 # MPAI-CAE ARP Workflow
 [![MPAI-CAE ARP](https://img.shields.io/badge/MPAI%20CAE--ARP-gray?style=for-the-badge&logo=AppleMusic&logoColor=cyan&link=https://mpai.community/standards/mpai-cae/about-mpai-cae/)](https://mpai.community/standards/mpai-cae/about-mpai-cae/)
 This is the repository containing the MPAI-CAE ARP workflow implementation. The workflow is composed of the following steps:
 1. Audio and video analysis;
 2. Video irregularity classification;
 3. Tape audio restoration;
 4. Packaging.
 The workflow is implemented as a set of docker containers, each one performing a specific task. The containers are orchestrated by docker-compose. Once the composition is started, the containers will communicate with each other through a shared volume, which will be used to store the intermediate results of the workflow. The starting point of the workflow is a client application, which will specify the input path. The client and the workflow containers communicate via gRPC.
 ## Requirements
 The following software is required to run the workflow:
 * Docker;
 * Docker-compose;
 * Python 3.10 or higher.
 * poetry
 * git
 ## Installation
 To install the workflow, clone the repository with (you may need to install git first, and request access to the repository):
 ```bash
 git clone https://gitlab.dei.unipd.it/mpai/workflow.git
 cd workflow
 ```
 Then, since the workflow has been developed in many repositories (one for each step), you need to clone the submodules with:
 ```bash
 git submodule update --init --recursive
 ```
 Once the repository is cloned, you need to build the docker images. To do so start docker and run:
 ```bash
 docker compose up -d
 ```
 On the first run, docker will download the base images and build the custom ones. This may take a while.
 If everithing went fine, you should be to see the containers running with:
 ```bash
 docker ps
 ```
 Create and copy your digital tape files in the following directories:
 ```bash
 mkdir data
 mkdir data/PreservationAudioFile
 mkdir data/PreservationAudioVisualFile
 ```
 To install the client application, you need to install poetry with:
 ```bash
 pip install poetry
 ```
 Then, you can install the client application by fisrt moving to the client directory:
 ```bash
 cd client
 ```
 And then running:
 ```bash
 poetry install
 ```
 ## Usage
 If the containers are not running, you can start them with:
 ```bash
 docker compose up -d
 ```
 To run the client application, you need to move to the client directory:
 ```bash
 cd client
 ```
 And then run:
 ```bash
 poetry run mpai-cae-arp -h
 ```
 It will show the help message with some instructions on how to use the client application:
 ```
 usage: mpai-cae-arp [-h] -f FILES_NAME [-s HOST]
 The MPAI-CAE ARP client.
 options:
  -h, --help            show this help message and exit
  -f FILES_NAME, --files-name FILES_NAME
                        Specify the name of the Preservation files (without extension)
  -s HOST, --host HOST  Specify the host of the server (default is localhost)
 ```
 The application expects to find the input files in the `data` directory, that is shared with the containers and should have the following structure:
 ```
 data
 ├── PreservationAudioFile
 │   ├── BERIO052.wav
 │   └── BERIO100.wav
 └── PreservationAudioVisualFile
     ├── BERIO052.mov
     └── BERIO100.mov
 ```
 To execute this program for file `BERIO100` prompt:
 ```
 poetry run mpai-cae-arp -f BERIO100
 ```
 It will take a while. When you see the message "Success", you can open the data folder where you will find the restoration files.
 ## Support
 If you require additional information or have any problem running the software, please contact us at:
 * Matteo Spanio (dev2@audioinnova.com)
 ## Authors and acknowledgment
 This project was developed by:
 * Nadir Dalla Pozza (University of Padova);
 * Niccolò Pretto (University of Padova);
 * Sergio Canazza (University of Padova).
 ## License
 This project is licensed with [GNU GPL v3.0](https://www.gnu.org/licenses/gpl-3.0.html).
--- a/client/poetry.lock
+++ b/client/poetry.lock
--- a/client/pyproject.toml
+++ b/client/pyproject.toml
 [tool.poetry]
 name = "client"
 version = "0.1.0"
 description = ""
 authors = ["Matteo Spanio <dev2@audioinnova.com>"]
 readme = "README.md"
 packages = [
    { include = "src" },
 ]
 [tool.poetry.dependencies]
 python = "^3.10"
 rich = "^13.3.3"
 grpcio-tools = "^1.53.0"
 mpai-cae-arp = "^0.4.1"
 [build-system]
 requires = ["poetry-core"]
 build-backend = "poetry.core.masonry.api"
 [tool.poetry.scripts]
 mpai-cae-arp = 'src.app:run'
--- a/client/src/__init__.py
+++ b/client/src/__init__.py
--- a/client/src/__pycache__/app.cpython-310.pyc
+++ b/client/src/__pycache__/app.cpython-310.pyc
--- a/client/src/app.py
+++ b/client/src/app.py
 from rich.console import Console
 import os
 from argparse import ArgumentParser
 from time import sleep
 import grpc
 from mpai_cae_arp.network import arp_pb2
 from mpai_cae_arp.network import arp_pb2_grpc
 channels = {
    "AudioAnalyser": grpc.insecure_channel("localhost:50051"),
    "VideoAnalyser": grpc.insecure_channel("localhost:50052"),
    "TapeIrregularityClassifier": grpc.insecure_channel("localhost:50053"),
    "TapeAudioRestoration": grpc.insecure_channel("localhost:50054"),
    "Packager": grpc.insecure_channel("localhost:50055"),
 }
 def get_args() -> tuple[str, str]:
    parser = ArgumentParser(
        prog="mpai-cae-arp",
        description="The MPAI-CAE ARP client.",
    )
    parser.add_argument("-f", "--files-name", help="Specify the name of the Preservation files (without extension)", required=True)
    parser.add_argument("-s", "--host", help="Specify the host of the server (default is localhost)", default="localhost")
    args = parser.parse_args()
    return args.files_name, args.host
 def run():
    console = Console()
    files_name, host = get_args()
    if host != "localhost":
        console.print("[red bold]Warning![/] :warning:")
        console.print("[italic red]You are not using the default host. Make sure you are using the same host as the server.[/]")
        exit(os.EX_USAGE)
    audio_analyser = arp_pb2_grpc.AIMStub(channels["AudioAnalyser"])
    video_analyser = arp_pb2_grpc.AIMStub(channels["VideoAnalyser"])
    tape_irreg_classifier = arp_pb2_grpc.AIMStub(channels["TapeIrregularityClassifier"])
    tape_audio_restoration = arp_pb2_grpc.AIMStub(channels["TapeAudioRestoration"])
    packager = arp_pb2_grpc.AIMStub(channels["Packager"])
    request = arp_pb2.InfoRequest()
    try:
        for aim in [audio_analyser, video_analyser, tape_irreg_classifier, tape_audio_restoration, packager]:
            response = aim.getInfo(request)
            console.print("[bold]{}[/], v{}".format(response.title, response.version))
            sleep(0.3)
    except grpc.RpcError as e:
        console.print("[bold red]Error![/] :boom:")
        console.print(f"[italic red]{e.details()}[/]")
        exit(69)
    request = arp_pb2.JobRequest(
        working_dir="/data",
        files_name=files_name,
        index=1,
    )
    with console.status("[bold]Computing AudioAnalyser IrregularityFile 1...", spinner="bouncingBall"):
        for result in audio_analyser.work(request):
            if result.status == "error":
                console.print("[bold red]Error![/] :boom:")
                console.print(f"[italic red]{result.message}")
                for channel in channels.values():
                    channel.close()
                exit(os.EX_SOFTWARE)
            console.print(result.message)
    request.files_name = f"{files_name}.mov"
    with console.status("[bold]Computing VideoAnalyser IrregularityFiles...", spinner="bouncingBall"):
        for result in video_analyser.work(request):
            if result.status == "error":
                console.print("[bold red]Error![/] :boom:")
                console.print(f"[italic red]{result.message}")
                for channel in channels.values():
                    channel.close()
                exit(os.EX_SOFTWARE)
            console.print(result.message)
    request.index = 2
    request.files_name = files_name
    with console.status("[bold]Computing AudioAnalyser IrregularityFile 2...", spinner="bouncingBall"):
        for result in audio_analyser.work(request):
            if result.status == "error":
                console.print("[bold red]Error![/] :boom:")
                console.print(f"[italic red]{result.message}")
                for channel in channels.values():
                    channel.close()
                exit(os.EX_SOFTWARE)
            console.print(result.message)
    with console.status("[bold]Computing TapeIrregularityClassifier...", spinner="bouncingBall"):
        for result in tape_irreg_classifier.work(request):
            if result.status == "error":
                console.print("[bold red]Error![/] :boom:")
                console.print(f"[italic red]{result.message}")
                for channel in channels.values():
                    channel.close()
                exit(os.EX_SOFTWARE)
            console.print(result.message)
    with console.status("[bold]Computing TapeAudioRestoration...", spinner="bouncingBall"):
        for result in tape_audio_restoration.work(request):
            if result.status == "error":
                console.print("[bold red]Error![/] :boom:")
                console.print(f"[italic red]{result.message}")
                for channel in channels.values():
                    channel.close()
                exit(os.EX_SOFTWARE)
            console.print(result.message)
    with console.status("[bold]Packaging...", spinner="bouncingBall"):
        for result in packager.work(request):
            if result.status == "error":
                console.print("[bold red]Error![/] :boom:")
                console.print(f"[italic red]{result.message}")
                for channel in channels.values():
                    channel.close()
                exit(os.EX_SOFTWARE)
            console.print(result.message)
    channels["AudioAnalyser"].close()
    channels["VideoAnalyser"].close()
    channels["TapeIrregularityClassifier"].close()
    channels["TapeAudioRestoration"].close()
    channels["Packager"].close()
    console.print("[bold green]Success![/] :tada:")
 if __name__ == '__main__':
    run()
--- a/docker-compose.yml
+++ b/docker-compose.yml
 version: '3.9'
 services:
  audio-analyzer:
    image: mpai-audio-analyzer
    build: ./audio-analyzer
    ports:
      - "${AUDIO_ANALYSER_PORT}:50051"
    volumes:
      - ${DATA_VOLUME}:/data
  video-analyzer:
    image: mpai-video-analyzer
    build: ./video-analyzer
    ports:
      - "${VIDEO_ANALYSER_PORT}:50051"
    volumes:
      - ${DATA_VOLUME}:/data
  tape-irregularity-classifier:
    image: mpai-tape-irregularity-classifier
    build: ./tape-irregularity-classifier
    ports:
      - "${TAPE_IRREGULARITY_PORT}:50051"
    volumes:
      - ${DATA_VOLUME}:/data
  tape-audio-restoration:
    image: mpai-tape-audio-restoration
    build: ./tape-audio-restoration
    ports:
      - "${TAPE_AUDIO_RESTORATION_PORT}:50051"
    volumes:
      - ${DATA_VOLUME}:/data
  packager:
    image: mpai-packager
    build: ./packager
    ports:
      - "${PACKAGER_PORT}:50051"
    volumes:
      - ${DATA_VOLUME}:/data
 volumes:
  data: