Commit df88fb26 authored by Matteo's avatar Matteo
Browse files

refactor

parent 7eb5bc11
...@@ -90,4 +90,43 @@ In addition to the Google C++ Style Guide, the following rules are applied: ...@@ -90,4 +90,43 @@ In addition to the Google C++ Style Guide, the following rules are applied:
- when returning multiple values, use `std::tuple` or `std::pair`, instead of passing by reference; - when returning multiple values, use `std::tuple` or `std::pair`, instead of passing by reference;
- when dealing with nullable values, use `std::optional`; - when dealing with nullable values, use `std::optional`;
- avoid to manipulate global variables, if you need to share data between different parts of the code, use dependency injection and pass the data as a parameter; - avoid to manipulate global variables, if you need to share data between different parts of the code, use dependency injection and pass the data as a parameter;
\ No newline at end of file
## Naming conventions
In C++, there are several naming conventions that are widely followed to improve code readability and maintainability. Although there is no strict standard enforced by the language, the following conventions are commonly used:
1. Variable Names:
- Use descriptive and meaningful names that reflect the purpose of the variable.
- Prefer lowercase letters for variable names.
- Use underscores (_) to separate words in multi-word variable names.
- Avoid using single uppercase letters as variable names, especially as standalone variables.
Example: `int num_items;`
2. Function Names:
- Use verbs or verb phrases to describe actions or operations performed by the function.
- Prefer lowercase letters for function names.
- Use underscores (_) to separate words in multi-word function names.
- Use parentheses () for function parameters, even if they are empty.
Example: `void calculate_average();`
3. Class/Struct Names:
- Use noun phrases or nouns to describe the purpose or nature of the class/struct.
- Use uppercase letters for each word (known as "PascalCase" or "camel case").
- Avoid abbreviations unless they are widely recognized.
Example: `class CustomerData;`
4. Constant Names:
- Use uppercase letters for constants.
- Use underscores (_) to separate words in multi-word constant names.
- Prefer meaningful and self-explanatory names for constants.
Example: `const int MAX_SIZE = 100;`
5. Global Variable Names:
- Avoid using global variables whenever possible. However, if necessary, prefix them with `g_` or use a namespace to indicate their global nature.
Example: `int g_global_variable;` or `namespace globals { int global_variable; }`
# General Requirements
The main goal of the MPAI-CAE ARP software is to take audio an video input of a open reel tape, analyse them and produce as output some kind of classifications and restorations (if needed).
# The input
As already said, the input consists of a video and an audio file of an open reel tape. The video file contains the video of the tape reproduced on the recorder, pointing the camera to the capstan, the pinch roller and the reading head. The audio file contains the audio of the tape reproduced on the recorder and captured from the headphone exit.
Focusing on the video analysis, the software should be able to detect tape irregularities, such as:
- Splices
- Brands on tape
- Start of tape
- End of tape
- Damaged tape
- Dirt
- Marks
- Shadows
- Wow and flutter
Most of the brands consist of the full name of the tape manufacturer, logo, or tape model codes. The brand changes in size, shape, and colour, depending on the tape used.
# Software Requirements # Software Requirements
The software should be able to, given as input the video of an open reel tape, produce as output two irregularity files where are listed the irregularities found in the video and the irregularities found in the audio. The software should be able to, given as input the video of an open reel tape, produce as output two irregularity files where are listed the irregularities found in the video and the irregularities found in the audio.
......
...@@ -23,7 +23,7 @@ ...@@ -23,7 +23,7 @@
* @copyright 2023, Audio Innova S.r.l. * @copyright 2023, Audio Innova S.r.l.
* @credits Niccolò Pretto, Nadir Dalla Pozza, Sergio Canazza * @credits Niccolò Pretto, Nadir Dalla Pozza, Sergio Canazza
* @license GPL v3.0 * @license GPL v3.0
* @version 1.1.2 * @version 1.1.3
* @status Production * @status Production
*/ */
#include <stdlib.h> #include <stdlib.h>
...@@ -66,33 +66,25 @@ namespace fs = std::filesystem; ...@@ -66,33 +66,25 @@ namespace fs = std::filesystem;
namespace po = boost::program_options; namespace po = boost::program_options;
/** /**
* @const bool useSURF * @const bool g_use_surf
* @brief If true, SURF is used for capstan detection, otherwise GHT is used. * @brief If true, SURF is used for capstan detection, otherwise GHT is used.
* *
* For capstan detection, there are two alternative approaches: * For capstan detection, there are two alternative approaches:
* 1. Generalized Hough Transform * 1. Generalized Hough Transform
* 2. SURF. * 2. SURF.
*/ */
bool useSURF = true; bool g_use_surf = true;
bool g_end_tape_saved = false;
bool endTapeSaved = false; bool g_first_brand = true; // The first frame containing brands on tape must be saved
float mediaPrevFrame = 0; float g_first_instant = 0;
float g_mean_prev_frame_color = 0; // Average frame color
/**
* @var bool firstBrand static fs::path g_output_path{};
* @brief The first frame containing brands on tape must be saved static fs::path g_irregularity_images_path{};
*/ static json g_irregularity_file_1{};
bool firstBrand = true; static json g_irregularity_file_2{};
float firstInstant = 0;
// Path variables
static fs::path outputPath{};
static fs::path irregularityImagesPath{};
// JSON files
static json irregularityFileOutput1{};
static json irregularityFileOutput2{};
// RotatedRect identifying the processing area // RotatedRect identifying the processing area
RotatedRect rect, rectTape, rectCapstan; RotatedRect rect, g_rect_tape, g_rect_capstan;
/** /**
* @fn void pprint(string text, string color) * @fn void pprint(string text, string color)
...@@ -202,16 +194,19 @@ Frame get_next_frame(VideoCapture& cap, float speed, bool skip = false) { ...@@ -202,16 +194,19 @@ Frame get_next_frame(VideoCapture& cap, float speed, bool skip = false) {
return frame; return frame;
} }
double rotatedRectArea(RotatedRect rect) { return rect.size.width * rect.size.height; } float rotated_rect_area(RotatedRect rect) { return rect.size.width * rect.size.height; }
/** /**
* @fn std::tuple<int, int, double, double, vector<Vec4f>, vector<Vec4f>> * @fn std::tuple<int, int, double, double, vector<Vec4f>, vector<Vec4f>>
* findObject(Mat model, SceneObject object) * find_object(Mat model, SceneObject object)
* @brief Find the model in the scene using the Generalized Hough Transform. * @brief Find the model in the scene using the Generalized Hough Transform.
* It returns the best matches. Find the best matches for positive and negative * It returns the best matches. Find the best matches for positive and negative
* angles. If there are more than one shape, then choose the one with the * angles. If there are more than one shape, then choose the one with the
* highest score. If there are more than one with the same highest score, then * highest score. If there are more than one with the same highest score, then
* arbitrarily choose the latest * arbitrarily choose the latest.
*
* For informations about the Generalized Hough Guild usage see the tutorial
* at https://docs.opencv.org/4.7.0/da/ddc/tutorial_generalized_hough_ballard_guil.html
* *
* @param model the template image to be searched with the Generalized Hough * @param model the template image to be searched with the Generalized Hough
* Transform * Transform
...@@ -220,15 +215,11 @@ double rotatedRectArea(RotatedRect rect) { return rect.size.width * rect.size.he ...@@ -220,15 +215,11 @@ double rotatedRectArea(RotatedRect rect) { return rect.size.width * rect.size.he
* @return std::tuple<int, int, double, double, vector<Vec4f>, vector<Vec4f>> a * @return std::tuple<int, int, double, double, vector<Vec4f>, vector<Vec4f>> a
* tuple containing the best matches for positive and negative angles * tuple containing the best matches for positive and negative angles
*/ */
std::tuple<int, int, double, double, vector<Vec4f>, vector<Vec4f>> findObject(Mat model, SceneObject object, std::tuple<int, int, double, double, vector<Vec4f>, vector<Vec4f>> find_object(Mat model, SceneObject object,
Mat processing_area) { Mat processing_area) {
// Algorithm and parameters
// for informations about the Generalized Hough Guild usage see the tutorial
// at
// https://docs.opencv.org/4.7.0/da/ddc/tutorial_generalized_hough_ballard_guil.html
Ptr<GeneralizedHoughGuil> alg = createGeneralizedHoughGuil(); Ptr<GeneralizedHoughGuil> alg = createGeneralizedHoughGuil();
vector<Vec4f> positionsPos, positionsNeg; vector<Vec4f> positive_positions, negative_positions;
Mat votesPos, votesNeg; Mat votesPos, votesNeg;
double maxValPos = 0, maxValNeg = 0; double maxValPos = 0, maxValNeg = 0;
...@@ -254,7 +245,7 @@ std::tuple<int, int, double, double, vector<Vec4f>, vector<Vec4f>> findObject(Ma ...@@ -254,7 +245,7 @@ std::tuple<int, int, double, double, vector<Vec4f>, vector<Vec4f>> findObject(Ma
alg->setTemplate(model); alg->setTemplate(model);
utility::detectShape(alg, model, object.threshold.pos, positionsPos, votesPos, positionsNeg, votesNeg, utility::detectShape(alg, model, object.threshold.pos, positive_positions, votesPos, negative_positions, votesNeg,
processing_area); processing_area);
for (int i = 0; i < votesPos.size().width; i++) { for (int i = 0; i < votesPos.size().width; i++) {
...@@ -271,27 +262,27 @@ std::tuple<int, int, double, double, vector<Vec4f>, vector<Vec4f>> findObject(Ma ...@@ -271,27 +262,27 @@ std::tuple<int, int, double, double, vector<Vec4f>, vector<Vec4f>> findObject(Ma
} }
} }
return {indexPos, indexNeg, maxValPos, maxValNeg, positionsPos, positionsNeg}; return {indexPos, indexNeg, maxValPos, maxValNeg, positive_positions, negative_positions};
} }
/** /**
* @fn bool findProcessingAreas(Mat myFrame) * @fn bool find_processing_areas(Mat my_frame)
* @brief Identifies the Regions Of Interest (ROIs) on the video, * @brief Identifies the Regions Of Interest (ROIs) on the video,
* which are: * which are:
* - The reading head; * - The reading head;
* - The tape area under the tape head (computed on the basis of the detected * - The tape area under the tape head (computed on the basis of the detected
* reading head); * reading head);
* - The capstan. * - The capstan.
* @param myFrame The current frame of the video. * @param my_frame The current frame of the video.
* @return true if some areas have been detected; * @return true if some areas have been detected;
* @return false otherwise. * @return false otherwise.
*/ */
bool findProcessingAreas(Mat myFrame, SceneObject tape, SceneObject capstan) { bool find_processing_areas(Mat my_frame, SceneObject tape, SceneObject capstan) {
/************************** READING HEAD DETECTION ***********************/ /************************** READING HEAD DETECTION ***********************/
// Save a grayscale version of myFrame in myFrameGrayscale and downsample it // Save a grayscale version of my_frame in myFrameGrayscale and downsample it
// in half pixels for performance reasons // in half pixels for performance reasons
Frame gray_current_frame = Frame(myFrame).convertColor(COLOR_BGR2GRAY); Frame gray_current_frame = Frame(my_frame).convertColor(COLOR_BGR2GRAY);
Frame halved_gray_current_frame = gray_current_frame.clone().downsample(2); Frame halved_gray_current_frame = gray_current_frame.clone().downsample(2);
...@@ -306,16 +297,16 @@ bool findProcessingAreas(Mat myFrame, SceneObject tape, SceneObject capstan) { ...@@ -306,16 +297,16 @@ bool findProcessingAreas(Mat myFrame, SceneObject tape, SceneObject capstan) {
RotatedRect rectPos, rectNeg; RotatedRect rectPos, rectNeg;
auto [indexPos, indexNeg, maxValPos, maxValNeg, positionsPos, positionsNeg] = auto [indexPos, indexNeg, maxValPos, maxValNeg, positionsPos, positionsNeg] =
findObject(reading_head_template, tape, processingImage); find_object(reading_head_template, tape, processingImage);
// The color is progressively darkened to emphasize that the algorithm found // The color is progressively darkened to emphasize that the algorithm found
// more than one shape // more than one shape
if (positionsPos.size() > 0) if (positionsPos.size() > 0)
rectPos = utility::drawShapes(myFrame, positionsPos[indexPos], Scalar(0, 0, 255 - indexPos * 64), rectPos = utility::drawShapes(my_frame, positionsPos[indexPos], Scalar(0, 0, 255 - indexPos * 64),
reading_head_template.cols, reading_head_template.rows, reading_head_template.cols, reading_head_template.rows,
halved_gray_current_frame.cols / 4, halved_gray_current_frame.rows / 2, 2); halved_gray_current_frame.cols / 4, halved_gray_current_frame.rows / 2, 2);
if (positionsNeg.size() > 0) if (positionsNeg.size() > 0)
rectNeg = utility::drawShapes(myFrame, positionsNeg[indexNeg], Scalar(128, 128, 255 - indexNeg * 64), rectNeg = utility::drawShapes(my_frame, positionsNeg[indexNeg], Scalar(128, 128, 255 - indexNeg * 64),
reading_head_template.cols, reading_head_template.rows, reading_head_template.cols, reading_head_template.rows,
halved_gray_current_frame.cols / 4, halved_gray_current_frame.rows / 2, 2); halved_gray_current_frame.cols / 4, halved_gray_current_frame.rows / 2, 2);
...@@ -338,10 +329,10 @@ bool findProcessingAreas(Mat myFrame, SceneObject tape, SceneObject capstan) { ...@@ -338,10 +329,10 @@ bool findProcessingAreas(Mat myFrame, SceneObject tape, SceneObject capstan) {
/************************************ TAPE AREA DETECTION ****************/ /************************************ TAPE AREA DETECTION ****************/
// Compute area basing on reading head detection // Compute area basing on reading head detection
Vec4f positionTape(rect.center.x, rect.center.y + rect.size.height / 2 + 20 * (rect.size.width / 200), 1, Vec4f tape_position(rect.center.x, rect.center.y + rect.size.height / 2 + 20 * (rect.size.width / 200), 1,
rect.angle); rect.angle);
rectTape = utility::drawShapes(myFrame, positionTape, Scalar(0, 255 - indexPos * 64, 0), rect.size.width, g_rect_tape = utility::drawShapes(my_frame, tape_position, Scalar(0, 255 - indexPos * 64, 0), rect.size.width,
50 * (rect.size.width / 200), 0, 0, 1); 50 * (rect.size.width / 200), 0, 0, 1);
/************************************* CAPSTAN DETECTION ******************/ /************************************* CAPSTAN DETECTION ******************/
...@@ -349,11 +340,11 @@ bool findProcessingAreas(Mat myFrame, SceneObject tape, SceneObject capstan) { ...@@ -349,11 +340,11 @@ bool findProcessingAreas(Mat myFrame, SceneObject tape, SceneObject capstan) {
// need to downsample // need to downsample
Mat capstan_template = cv::imread(CAPSTAN_TEMPLATE_IMG, IMREAD_GRAYSCALE); Mat capstan_template = cv::imread(CAPSTAN_TEMPLATE_IMG, IMREAD_GRAYSCALE);
if (useSURF) { if (g_use_surf) {
// Step 1: Detect the keypoints using SURF Detector, compute the // Step 1: Detect the keypoints using SURF Detector, compute the
// descriptors // descriptors
int minHessian = 100; int min_hessian = 100;
Ptr<xfeatures2d::SURF> detector = xfeatures2d::SURF::create(minHessian); Ptr<xfeatures2d::SURF> detector = xfeatures2d::SURF::create(min_hessian);
vector<KeyPoint> keypoints_object, keypoints_scene; vector<KeyPoint> keypoints_object, keypoints_scene;
Mat descriptors_object, descriptors_scene; Mat descriptors_object, descriptors_scene;
...@@ -366,10 +357,10 @@ bool findProcessingAreas(Mat myFrame, SceneObject tape, SceneObject capstan) { ...@@ -366,10 +357,10 @@ bool findProcessingAreas(Mat myFrame, SceneObject tape, SceneObject capstan) {
vector<vector<DMatch>> knn_matches; vector<vector<DMatch>> knn_matches;
matcher->knnMatch(descriptors_object, descriptors_scene, knn_matches, 2); matcher->knnMatch(descriptors_object, descriptors_scene, knn_matches, 2);
//-- Filter matches using the Lowe's ratio test //-- Filter matches using the Lowe's ratio test
const float ratio_thresh = 0.75f; const float RATIO_THRESH = 0.75f;
vector<DMatch> good_matches; vector<DMatch> good_matches;
for (size_t i = 0; i < knn_matches.size(); i++) { for (size_t i = 0; i < knn_matches.size(); i++) {
if (knn_matches[i][0].distance < ratio_thresh * knn_matches[i][1].distance) { if (knn_matches[i][0].distance < RATIO_THRESH * knn_matches[i][1].distance) {
good_matches.push_back(knn_matches[i][0]); good_matches.push_back(knn_matches[i][0]);
} }
} }
...@@ -405,53 +396,53 @@ bool findProcessingAreas(Mat myFrame, SceneObject tape, SceneObject capstan) { ...@@ -405,53 +396,53 @@ bool findProcessingAreas(Mat myFrame, SceneObject tape, SceneObject capstan) {
// rectangle: +10 in X for centering and -20 in width +45 in Y for // rectangle: +10 in X for centering and -20 in width +45 in Y for
// centering and -90 in height // centering and -90 in height
Vec4f positionCapstan(capstanX + 10, capstanY + 45, 1, 0); Vec4f positionCapstan(capstanX + 10, capstanY + 45, 1, 0);
rectCapstan = utility::drawShapes(myFrame, positionCapstan, Scalar(255 - indexPos * 64, 0, 0), g_rect_capstan = utility::drawShapes(my_frame, positionCapstan, Scalar(255 - indexPos * 64, 0, 0),
capstan_template.cols - 20, capstan_template.rows - 90, 0, 0, 1); capstan_template.cols - 20, capstan_template.rows - 90, 0, 0, 1);
} else { } else {
// Process only right portion of the image, where the capstain always // Process only right portion of the image, where the capstain always
// appears // appears
int capstanProcessingAreaRectX = myFrame.cols * 3 / 4; int capstanProcessingAreaRectX = my_frame.cols * 3 / 4;
int capstanProcessingAreaRectY = myFrame.rows / 2; int capstanProcessingAreaRectY = my_frame.rows / 2;
int capstanProcessingAreaRectWidth = myFrame.cols / 4; int capstanProcessingAreaRectWidth = my_frame.cols / 4;
int capstanProcessingAreaRectHeight = myFrame.rows / 2; int capstanProcessingAreaRectHeight = my_frame.rows / 2;
Rect capstanProcessingAreaRect(capstanProcessingAreaRectX, capstanProcessingAreaRectY, Rect capstanProcessingAreaRect(capstanProcessingAreaRectX, capstanProcessingAreaRectY,
capstanProcessingAreaRectWidth, capstanProcessingAreaRectHeight); capstanProcessingAreaRectWidth, capstanProcessingAreaRectHeight);
Mat capstanProcessingAreaGrayscale = gray_current_frame(capstanProcessingAreaRect); Mat capstanProcessingAreaGrayscale = gray_current_frame(capstanProcessingAreaRect);
// Reset algorithm and set parameters // Reset algorithm and set parameters
auto [indexPos, indexNeg, maxValPos, maxValNeg, positionsC1Pos, positionsC1Neg] = auto [indexPos, indexNeg, maxValPos, maxValNeg, positionsC1Pos, positionsC1Neg] =
findObject(capstan_template, capstan, capstanProcessingAreaGrayscale); find_object(capstan_template, capstan, capstanProcessingAreaGrayscale);
RotatedRect rectCapstanPos, rectCapstanNeg; RotatedRect rectCapstanPos, rectCapstanNeg;
if (positionsC1Pos.size() > 0) if (positionsC1Pos.size() > 0)
rectCapstanPos = utility::drawShapes(myFrame, positionsC1Pos[indexPos], Scalar(255 - indexPos * 64, 0, 0), rectCapstanPos = utility::drawShapes(my_frame, positionsC1Pos[indexPos], Scalar(255 - indexPos * 64, 0, 0),
capstan_template.cols - 22, capstan_template.rows - 92, capstan_template.cols - 22, capstan_template.rows - 92,
capstanProcessingAreaRectX + 11, capstanProcessingAreaRectY + 46, 1); capstanProcessingAreaRectX + 11, capstanProcessingAreaRectY + 46, 1);
if (positionsC1Neg.size() > 0) if (positionsC1Neg.size() > 0)
rectCapstanNeg = utility::drawShapes(myFrame, positionsC1Neg[indexNeg], Scalar(255 - indexNeg * 64, 128, 0), rectCapstanNeg = utility::drawShapes(
capstan_template.cols - 22, capstan_template.rows - 92, my_frame, positionsC1Neg[indexNeg], Scalar(255 - indexNeg * 64, 128, 0), capstan_template.cols - 22,
capstanProcessingAreaRectX + 11, capstanProcessingAreaRectY + 46, 1); capstan_template.rows - 92, capstanProcessingAreaRectX + 11, capstanProcessingAreaRectY + 46, 1);
if (maxValPos > 0) if (maxValPos > 0)
if (maxValNeg > 0) if (maxValNeg > 0)
if (maxValPos > maxValNeg) { if (maxValPos > maxValNeg) {
rectCapstan = rectCapstanPos; g_rect_capstan = rectCapstanPos;
} else { } else {
rectCapstan = rectCapstanNeg; g_rect_capstan = rectCapstanNeg;
} }
else { else {
rectCapstan = rectCapstanPos; g_rect_capstan = rectCapstanPos;
} }
else if (maxValNeg > 0) { else if (maxValNeg > 0) {
rectCapstan = rectCapstanNeg; g_rect_capstan = rectCapstanNeg;
} else { } else {
return false; return false;
} }
} }
// Save the image containing the ROIs // Save the image containing the ROIs
cv::imwrite(outputPath.string() + "/tapeAreas.jpg", myFrame); cv::imwrite(g_output_path.string() + "/tape_areas.jpg", my_frame);
return true; return true;
} }
...@@ -495,88 +486,85 @@ Frame get_difference_for_roi(Frame previous, Frame current, RotatedRect roi) { ...@@ -495,88 +486,85 @@ Frame get_difference_for_roi(Frame previous, Frame current, RotatedRect roi) {
} }
/** /**
* @fn bool is_frame_different(cv::Mat prevFrame, cv::Mat currentFrame, int * @fn bool is_frame_different(cv::Mat prev_frame, cv::Mat current_frame, int
* msToEnd) * ms_to_end)
* @brief Compares two consecutive video frames and establish if there * @brief Compares two consecutive video frames and establish if there
* potentially is an Irregularity. The comparison is pixel-wise and based on * potentially is an Irregularity. The comparison is pixel-wise and based on
* threshold values set on config.json file. * threshold values set on config.json file.
* *
* @param prevFrame the frame before the current one; * @param prev_frame the frame before the current one;
* @param currentFrame the current frame; * @param current_frame the current frame;
* @param msToEnd the number of milliseconds left before the end of the video. * @param ms_to_end the number of milliseconds left before the end of the video.
* Useful for capstan analysis. * Useful for capstan analysis.
* @return true if a potential Irregularity has been found; * @return true if a potential Irregularity has been found;
* @return false otherwise. * @return false otherwise.
*/ */
bool is_frame_different(cv::Mat prevFrame, cv::Mat currentFrame, int msToEnd, SceneObject capstan, SceneObject tape, bool is_frame_different(cv::Mat prev_frame, cv::Mat current_frame, int ms_to_end, SceneObject capstan, SceneObject tape,
Args args) { Args args) {
bool result = false; bool result = false;
int num_different_pixels = 0;
/*********************** Capstan analysis ************************/ /*********************** Capstan analysis ************************/
// In the last minute of the video, check for pinchRoller position for // In the last minute of the video, check for pinchRoller position for
// endTape event // endTape event
if (!endTapeSaved && msToEnd < 60000) { if (!g_end_tape_saved && ms_to_end < 60000) {
float capstanDifferentPixelsThreshold = RotatedRect corrected_capstan_roi = check_skew(g_rect_capstan);
(rectCapstan.size.width * rectCapstan.size.height) * capstan.threshold.percentual / 100; Frame difference_frame = get_difference_for_roi(Frame(prev_frame), Frame(current_frame), corrected_capstan_roi);
RotatedRect corrected_capstan_roi = check_skew(rectCapstan);
Frame difference_frame = get_difference_for_roi(Frame(prevFrame), Frame(currentFrame), corrected_capstan_roi);
int blackPixelsCapstan = 0;
for (int i = 0; i < difference_frame.rows; i++) { for (int i = 0; i < difference_frame.rows; i++) {
for (int j = 0; j < difference_frame.cols; j++) { for (int j = 0; j < difference_frame.cols; j++) {
if (difference_frame.at<cv::Vec3b>(i, j)[0] == 0) { if (difference_frame.at<cv::Vec3b>(i, j)[0] == 0) {
// There is a black pixel, then there is a difference // There is a black pixel, then there is a difference
// between previous and current frames // between previous and current frames
blackPixelsCapstan++; num_different_pixels++;
} }
} }
} }
if (blackPixelsCapstan > capstanDifferentPixelsThreshold) { float capstan_pixel_threshold = rotated_rect_area(g_rect_capstan) * capstan.threshold.percentual / 100;
endTapeSaved = true; // Never check again for end tape instant if (num_different_pixels > capstan_pixel_threshold) {
g_end_tape_saved = true; // Never check again for end tape instant
return true; return true;
} }
} }
/********************* Tape analysis *********************/ /********************* Tape analysis *********************/
// Tape area RotatedRect corrected_tape_roi = check_skew(g_rect_tape);
int tapeAreaPixels = rotatedRectArea(rectTape); Frame cropped_current_frame =
float tapeDifferentPixelsThreshold = tapeAreaPixels * tape.threshold.percentual / 100; Frame(current_frame)
.warp(getRotationMatrix2D(corrected_tape_roi.center, corrected_tape_roi.angle, 1.0))
RotatedRect corrected_tape_roi = check_skew(rectTape); .crop(corrected_tape_roi.size, corrected_tape_roi.center);
Frame croppedCurrentFrame = Frame(currentFrame) Frame difference_frame = get_difference_for_roi(Frame(prev_frame), Frame(current_frame), corrected_tape_roi);
.warp(getRotationMatrix2D(corrected_tape_roi.center, corrected_tape_roi.angle, 1.0))
.crop(corrected_tape_roi.size, corrected_tape_roi.center);
Frame difference_frame = get_difference_for_roi(Frame(prevFrame), Frame(currentFrame), corrected_tape_roi);
/********************** Segment analysis ************************/ /********************** Segment analysis ************************/
int blackPixels = 0; num_different_pixels = 0;
float mediaCurrFrame; float mean_current_frame_color;
int totColoreCF = 0; int current_frame_color_sum = 0;
for (int i = 0; i < croppedCurrentFrame.rows; i++) { for (int i = 0; i < cropped_current_frame.rows; i++) {
for (int j = 0; j < croppedCurrentFrame.cols; j++) { for (int j = 0; j < cropped_current_frame.cols; j++) {
totColoreCF += croppedCurrentFrame.at<cv::Vec3b>(i, j)[0] + croppedCurrentFrame.at<cv::Vec3b>(i, j)[1] + current_frame_color_sum += cropped_current_frame.at<cv::Vec3b>(i, j)[0] +
croppedCurrentFrame.at<cv::Vec3b>(i, j)[2]; cropped_current_frame.at<cv::Vec3b>(i, j)[1] +
cropped_current_frame.at<cv::Vec3b>(i, j)[2];
if (difference_frame.at<cv::Vec3b>(i, j)[0] == 0) { if (difference_frame.at<cv::Vec3b>(i, j)[0] == 0) {
blackPixels++; num_different_pixels++;
} }
} }
} }
mediaCurrFrame = totColoreCF / tapeAreaPixels; float tape_area_pixels_sq = rotated_rect_area(g_rect_tape);
mean_current_frame_color = current_frame_color_sum / tape_area_pixels_sq;
/*********************** Decision stage ************************/ /*********************** Decision stage ************************/
if (blackPixels > tapeDifferentPixelsThreshold) { // The threshold must be passed float tape_pixel_threshold = tape_area_pixels_sq * tape.threshold.percentual / 100;
if (num_different_pixels > tape_pixel_threshold) { // The threshold must be passed
/***** AVERAGE_COLOR-BASED DECISION *****/ /***** AVERAGE_COLOR-BASED DECISION *****/
if (mediaPrevFrame > (mediaCurrFrame + 7) || if (g_mean_prev_frame_color > (mean_current_frame_color + 7) ||
mediaPrevFrame < (mediaCurrFrame - 7)) { // They are not similar for color average g_mean_prev_frame_color < (mean_current_frame_color - 7)) { // They are not similar for color average
result = true; result = true;
} }
...@@ -585,20 +573,19 @@ bool is_frame_different(cv::Mat prevFrame, cv::Mat currentFrame, int msToEnd, Sc ...@@ -585,20 +573,19 @@ bool is_frame_different(cv::Mat prevFrame, cv::Mat currentFrame, int msToEnd, Sc
// next Irregularity to consider it as a brand. It is not guaranteed // next Irregularity to consider it as a brand. It is not guaranteed
// that it will be the first brand, but it is generally a safe // that it will be the first brand, but it is generally a safe
// approach to have a correct image // approach to have a correct image
if (args.brands && firstBrand && firstInstant - msToEnd > 5000) { if (args.brands && g_first_brand && g_first_instant - ms_to_end > 5000) {
firstBrand = false; g_first_brand = false;
result = true; result = true;
} }
} }
// Update mediaPrevFrame g_mean_prev_frame_color = mean_current_frame_color;
mediaPrevFrame = mediaCurrFrame;
return result; return result;
} }
/** /**
* @fn void processing(cv::VideoCapture videoCapture, SceneObject capstan, * @fn void processing(cv::VideoCapture video_capture, SceneObject capstan,
* SceneObject tape, Args args) * SceneObject tape, Args args)
* @brief video processing phase, where each frame is analysed. * @brief video processing phase, where each frame is analysed.
* It saves the IrregularityImages and updates the IrregularityFiles if an * It saves the IrregularityImages and updates the IrregularityFiles if an
...@@ -612,64 +599,64 @@ bool is_frame_different(cv::Mat prevFrame, cv::Mat currentFrame, int msToEnd, Sc ...@@ -612,64 +599,64 @@ bool is_frame_different(cv::Mat prevFrame, cv::Mat currentFrame, int msToEnd, Sc
* 224x224 as in the past. If instead you decide to use the new neural network, * 224x224 as in the past. If instead you decide to use the new neural network,
* no changes are needed. * no changes are needed.
* *
* @param videoCapture the input Preservation Audio-Visual File; * @param video_capture the input Preservation Audio-Visual File;
* @param capstan the capstan SceneObject; * @param capstan the capstan SceneObject;
* @param tape the tape SceneObject; * @param tape the tape SceneObject;
* @param args the command line arguments. * @param args the command line arguments.
*/ */
void processing(cv::VideoCapture videoCapture, SceneObject capstan, SceneObject tape, Args args) { void processing(cv::VideoCapture video_capture, SceneObject capstan, SceneObject tape, Args args) {
const int video_length_ms = ((float)videoCapture.get(CAP_PROP_FRAME_COUNT) / videoCapture.get(CAP_PROP_FPS)) * 1000; const int video_length_ms =
int video_current_ms = videoCapture.get(CAP_PROP_POS_MSEC); ((float)video_capture.get(CAP_PROP_FRAME_COUNT) / video_capture.get(CAP_PROP_FPS)) * 1000;
// counters int video_current_ms = video_capture.get(CAP_PROP_POS_MSEC);
int savedFrames = 0; int num_saved_frames = 0;
bool irregularity_found = false; bool irregularity_found = false;
// The first frame of the video won't be processed // The first frame of the video won't be processed
cv::Mat prevFrame = get_next_frame(videoCapture, args.speed, irregularity_found); cv::Mat prev_frame = get_next_frame(video_capture, args.speed, irregularity_found);
firstInstant = video_length_ms - video_current_ms; g_first_instant = video_length_ms - video_current_ms;
while (videoCapture.isOpened()) { while (video_capture.isOpened()) {
Frame frame = get_next_frame(videoCapture, args.speed, irregularity_found); Frame frame = get_next_frame(video_capture, args.speed, irregularity_found);
video_current_ms = videoCapture.get(CAP_PROP_POS_MSEC); video_current_ms = video_capture.get(CAP_PROP_POS_MSEC);
if (frame.empty()) { if (frame.empty()) {
std::cout << endl << "Empty frame!" << endl; std::cout << endl << "Empty frame!" << endl;
videoCapture.release(); video_capture.release();
return; return;
} }
int msToEnd = video_length_ms - video_current_ms; int ms_to_end = video_length_ms - video_current_ms;
if (video_current_ms == 0) // With OpenCV library, this happens at the last few frames of if (video_current_ms == 0) // With OpenCV library, this happens at the last few frames of
// the video before realising that "frame" is empty. // the video before realising that "frame" is empty.
return; return;
// Display program status // Display program status
int secToEnd = msToEnd / 1000; int sec_to_end = ms_to_end / 1000;
int minToEnd = (secToEnd / 60) % 60; int min_to_end = (sec_to_end / 60) % 60;
secToEnd = secToEnd % 60; sec_to_end = sec_to_end % 60;
string secStrToEnd = (secToEnd < 10 ? "0" : "") + to_string(secToEnd); string secStrToEnd = (sec_to_end < 10 ? "0" : "") + to_string(sec_to_end);
string minStrToEnd = (minToEnd < 10 ? "0" : "") + to_string(minToEnd); string minStrToEnd = (min_to_end < 10 ? "0" : "") + to_string(min_to_end);
std::cout << "\rIrregularities: " << savedFrames << ". "; std::cout << "\rIrregularities: " << num_saved_frames << ". ";
std::cout << "Remaining video time [mm:ss]: " << minStrToEnd << ":" << secStrToEnd << flush; std::cout << "Remaining video time [mm:ss]: " << minStrToEnd << ":" << secStrToEnd << flush;
irregularity_found = is_frame_different(prevFrame, frame, msToEnd, capstan, tape, args); irregularity_found = is_frame_different(prev_frame, frame, ms_to_end, capstan, tape, args);
if (irregularity_found) { if (irregularity_found) {
auto [odd_frame, _] = frame.deinterlace(); auto [odd_frame, _] = frame.deinterlace();
string irregularityImageFilename = string irregularityImageFilename =
to_string(savedFrames) + "_" + getTimeLabel(video_current_ms, "-") + ".jpg"; to_string(num_saved_frames) + "_" + getTimeLabel(video_current_ms, "-") + ".jpg";
cv::imwrite(irregularityImagesPath / irregularityImageFilename, odd_frame); cv::imwrite(g_irregularity_images_path / irregularityImageFilename, odd_frame);
// Append Irregularity information to JSON // Append Irregularity information to JSON
Irregularity irreg = Irregularity(Source::Video, getTimeLabel(video_current_ms, ":")); Irregularity irreg = Irregularity(Source::Video, getTimeLabel(video_current_ms, ":"));
irregularityFileOutput1["Irregularities"] += irreg.to_JSON(); g_irregularity_file_1["Irregularities"] += irreg.to_JSON();
irregularityFileOutput2["Irregularities"] += g_irregularity_file_2["Irregularities"] +=
irreg.set_image_URI(irregularityImagesPath.string() + "/" + irregularityImageFilename).to_JSON(); irreg.set_image_URI(g_irregularity_images_path.string() + "/" + irregularityImageFilename).to_JSON();
savedFrames++; num_saved_frames++;
} }
prevFrame = frame; prev_frame = frame;
} }
} }
...@@ -695,24 +682,24 @@ void processing(cv::VideoCapture videoCapture, SceneObject capstan, SceneObject ...@@ -695,24 +682,24 @@ void processing(cv::VideoCapture videoCapture, SceneObject capstan, SceneObject
int main(int argc, char** argv) { int main(int argc, char** argv) {
SceneObject capstan = SceneObject::from_file(CONFIG_FILE, ROI::CAPSTAN); SceneObject capstan = SceneObject::from_file(CONFIG_FILE, ROI::CAPSTAN);
SceneObject tape = SceneObject::from_file(CONFIG_FILE, ROI::TAPE); SceneObject tape = SceneObject::from_file(CONFIG_FILE, ROI::TAPE);
const Args args = argc > 1 ? Args::from_cli(argc, argv) : Args::from_file(CONFIG_FILE); Args args = argc > 1 ? Args::from_cli(argc, argv) : Args::from_file(CONFIG_FILE);
const fs::path VIDEO_PATH = args.workingPath / "PreservationAudioVisualFile" / args.filesName; const fs::path VIDEO_PATH = args.workingPath / "PreservationAudioVisualFile" / args.filesName;
const auto [fileName, extension] = files::get_filename_and_extension(VIDEO_PATH); const auto [FILE_NAME, FILE_FORMAT] = files::get_filename_and_extension(VIDEO_PATH);
const fs::path irregularityFileInputPath = args.workingPath / "temp" / fileName / A_IRREG_FILE_1; const fs::path AUDIO_IRR_FILE_PATH = args.workingPath / "temp" / FILE_NAME / A_IRREG_FILE_1;
std::cout << "Video to be analysed: " << endl; std::cout << "Video to be analysed: " << endl;
std::cout << "\tFile name: " << fileName << endl; std::cout << "\tFile name: " << FILE_NAME << endl;
std::cout << "\tExtension: " << extension << endl; std::cout << "\tExtension: " << FILE_FORMAT << endl;
if (extension.compare("avi") != 0 && extension.compare("mp4") != 0 && extension.compare("mov") != 0) if (FILE_FORMAT.compare("avi") != 0 && FILE_FORMAT.compare("mp4") != 0 && FILE_FORMAT.compare("mov") != 0)
print_error_and_exit("Input error", "The input file must be an AVI, MP4 or MOV file."); print_error_and_exit("Input error", "The input file must be an AVI, MP4 or MOV file.");
ifstream iJSON(irregularityFileInputPath); ifstream iJSON(AUDIO_IRR_FILE_PATH);
if (iJSON.fail()) if (iJSON.fail())
print_error_and_exit("config.json error", irregularityFileInputPath.string() + " cannot be found or opened."); print_error_and_exit("config.json error", AUDIO_IRR_FILE_PATH.string() + " cannot be found or opened.");
// Read input JSON json audio_irr_file;
json irregularityFileInput; iJSON >> audio_irr_file;
iJSON >> irregularityFileInput;
// Adjust input paramenters (considering given ones as pertinent to a speed reference = 7.5) // Adjust input paramenters (considering given ones as pertinent to a speed reference = 7.5)
if (args.speed == 15) { if (args.speed == 15) {
...@@ -720,34 +707,33 @@ int main(int argc, char** argv) { ...@@ -720,34 +707,33 @@ int main(int argc, char** argv) {
} else if (!args.brands) } else if (!args.brands)
tape.threshold.percentual += 21; tape.threshold.percentual += 21;
// Make directory with fileName name g_output_path = args.workingPath / "temp" / FILE_NAME;
outputPath = args.workingPath / "temp" / fileName; fs::create_directory(g_output_path);
fs::create_directory(outputPath);
irregularityImagesPath = outputPath / "IrregularityImages"; g_irregularity_images_path = g_output_path / "IrregularityImages";
fs::create_directory(irregularityImagesPath); fs::create_directory(g_irregularity_images_path);
cv::VideoCapture videoCapture(VIDEO_PATH); // Open video file cv::VideoCapture video_capture(VIDEO_PATH); // Open video file
if (!videoCapture.isOpened()) print_error_and_exit("Video error", "Video file cannot be opened."); if (!video_capture.isOpened()) print_error_and_exit("Video error", "Video file cannot be opened.");
videoCapture.set(CAP_PROP_POS_FRAMES, video_capture.set(CAP_PROP_POS_FRAMES,
videoCapture.get(CAP_PROP_FRAME_COUNT) / 2); // Set frame position to half video length video_capture.get(CAP_PROP_FRAME_COUNT) / 2); // Set frame position to half video length
cv::Mat middle_frame = get_next_frame(videoCapture, args.speed); cv::Mat middle_frame = get_next_frame(video_capture, args.speed);
videoCapture.set(CAP_PROP_POS_FRAMES, 0); // Reset frame position video_capture.set(CAP_PROP_POS_FRAMES, 0); // Reset frame position
std::cout << "\tResolution: " << middle_frame.cols << "x" << middle_frame.rows << "\n\n"; std::cout << "\tResolution: " << middle_frame.cols << "x" << middle_frame.rows << "\n\n";
bool found = findProcessingAreas(middle_frame, tape, capstan); bool found = find_processing_areas(middle_frame, tape, capstan);
if (!found) print_error_and_exit("Processing area not found", "Try changing JSON parameters."); if (!found) print_error_and_exit("Processing area not found", "Try changing JSON parameters.");
pprint("Processing...", CYAN); pprint("Processing...", CYAN);
processing(videoCapture, capstan, tape, args); processing(video_capture, capstan, tape, args);
files::save_file(outputPath / V_IRREG_FILE_1, irregularityFileOutput1.dump(4)); files::save_file(g_output_path / V_IRREG_FILE_1, g_irregularity_file_1.dump(4));
// Irregularities to extract for the AudioAnalyser and to the TapeIrregularityClassifier // Irregularities to extract for the AudioAnalyser and to the TapeIrregularityClassifier
extractIrregularityImagesForAudio(outputPath, VIDEO_PATH, irregularityFileInput, irregularityFileOutput2); extractIrregularityImagesForAudio(g_output_path, VIDEO_PATH, audio_irr_file, g_irregularity_file_2);
files::save_file(outputPath / V_IRREG_FILE_2, irregularityFileOutput2.dump(4)); files::save_file(g_output_path / V_IRREG_FILE_2, g_irregularity_file_2.dump(4));
return EXIT_SUCCESS; return EXIT_SUCCESS;
} }
...@@ -141,7 +141,7 @@ void utility::detectShape(Ptr<GeneralizedHoughGuil> alg, Mat templateShape, int ...@@ -141,7 +141,7 @@ void utility::detectShape(Ptr<GeneralizedHoughGuil> alg, Mat templateShape, int
} }
} }
RotatedRect utility::drawShapes(Mat frame, Vec4f& positions, Scalar color, int width, int height, int offsetX, RotatedRect utility::drawShapes(Mat frame, const Vec4f& positions, Scalar color, int width, int height, int offsetX,
int offsetY, float processingScale) { int offsetY, float processingScale) {
RotatedRect rr; RotatedRect rr;
Point2f rrpts[4]; Point2f rrpts[4];
......
...@@ -110,7 +110,7 @@ void detectShape(Ptr<GeneralizedHoughGuil> alg, Mat templateShape, int posThresh ...@@ -110,7 +110,7 @@ void detectShape(Ptr<GeneralizedHoughGuil> alg, Mat templateShape, int posThresh
* @param processingScale Scaling factor, useful for downsizing. * @param processingScale Scaling factor, useful for downsizing.
* @return RotatedRect Object representing the drawn rectangle. * @return RotatedRect Object representing the drawn rectangle.
*/ */
RotatedRect drawShapes(Mat frame, Vec4f& positions, Scalar color, int width, int height, int offsetX, int offsetY, RotatedRect drawShapes(Mat frame, const Vec4f& positions, Scalar color, int width, int height, int offsetX, int offsetY,
float processingScale); float processingScale);
/** /**
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment