DATA Research Task (Video Ads Trial)

Jul 12, 2024

Dataset

Videos: 150 video advertisements of varying duration from various companies

Textual Data:

Ad campaign descriptions from companies
Transcriptions of the videos
On-screen text extracted from the videos

Ground Truth:

Consists of answers from mostly 3 human coders (some have 2, some have 4) to the 15 yes/no questions. The majority vote is taken as the ground truth, with ties resolved in favor of positive responses.

Questionnaire

Answer the following 21 yes/no questions for each of the 150 videos:

Is there a call to go online (e.g., shop online, visit the Web)?
Is online contact information provided (e.g., URL, website)?
Is there a visual or verbal call to purchase (e.g., buy now, order now)?
Does the ad portray a sense of urgency to act (e.g., buy before sales end, order before it ends)?
Is there an incentive to buy (e.g., a discount, a coupon, a sale, or "limited time offer")?
Is offline contact information provided (e.g., phone, mail, store location)?
Is there mention of something free?
Does the ad mention at least one specific product or service (e.g., model, type, item)?
Is there any verbal or visual mention of the price?
Does the ad show the brand (logo, brand name) or trademark (something that most people know is the brand) multiple times?
Does the ad show the brand or trademark exactly once at the end of the ad?
Is the ad intended to affect the viewer emotionally, either with positive emotion (fun, joy), negative emotion (sad, anxious) or another type of emotion? (Note: You may not personally agree, but assess if that was the intention.)
Does the ad give you a positive feeling about the brand?
Does the ad have a story arc, with a beginning and an end?
Does the ad have a reversal of fortune, where something changes for the better, or changes for the worse?
Does the ad have relatable characters?
Is the ad creative/clever?
Is the ad intended to be funny? (Note: You may not personally agree, but assess if that was the intention.)
Does this ad provide sensory stimulation (e.g., cool visuals, arousing music, mouth-watering)?
Is the ad visually pleasing?
Does the ad have cute elements like animals, babies, animated, characters, etc?

Methodology

Data Preparation:

Loaded text and ground truth data from CSV files.
Normalized textual data through stemming.
Converted categorical answers to binary values.
Merged datasets on a common identifier.
Extracted video frames as tensors.

Model Development:

Implemented a multimodal dataset class.
Architected a neural network combining ResNet18 and BERT models for video and text data, respectively.
Concatenated outputs from both models and passed them through a fully connected layer for final prediction.

Training:

Trained the model using Binary Cross Entropy with Logits Loss and the Adam optimizer.
Training lasted three days with checkpointing to capture the best model states.

Results

Overall Metrics
- Total Agreement: 3.333 (All or nothing approach - This is true if all questions are correctly predicted)
- Average Agreement: 81.43%
- Macro F1 Score: 0.74
- Macro Precision: 0.81
- Macro Recall: 0.72
Performance Metrics Per Question:

Insights

High Efficacy: The classifier demonstrated high efficacy in detecting specific features such as branding elements (Q10, F1=0.8649; Q11, F1=0.8859) and emotional intentions (Q12, F1=0.8085).
Strong Performance: Questions related to specific product mentions (Q8, F1=0.9078), price (Q9, F1=0.8929), and positive feelings towards the brand (Q13, F1=0.9007) showed strong performance.
Mixed Results: Questions involving a call to action (Q1, F1=0.5000), offline contact information (Q6, F1=0.5000), and narrative elements (Q14, F1=0.5581) showed moderate to low performance.
Areas for Improvement: Narrative elements and abstract features such as story arc (Q14, F1=0.5581) and relatable characters (Q16, F1=0.5263) demonstrated lower predictive accuracy, indicating areas needing improvement.

Bonus Questions

Analysis of Inconsistent Video Performance:
- Complex Narratives: Videos with complex story arcs and multiple characters may not align well with the classifier’s capabilities, leading to lower performance on questions like Q14 (story arc) and Q16 (relatable characters).
- Abstract Features: Questions involving abstract elements such as emotional intent or creative aspects (Q12, Q17) can be subjective, making it harder for the classifier to consistently interpret these features.
- Sparse Data: Certain features might be underrepresented in the training data, causing the classifier to struggle with questions like Q7 (mention of something free), which had an F1 score of 0.4000 despite high precision.
Analysis of Human Coders' Responses vs. Classifier Performance:
- Human Judgment Variability: Human coders might have varied interpretations of abstract or subjective features, introducing inconsistencies in the ground truth data.
- Alignment on Clear Features: The classifier aligns well with human coders on clear, unambiguous features such as specific product mentions (Q8) and brand displays (Q10, Q11), evidenced by high F1 scores.
- Subjective Evaluations: Discrepancies in questions involving subjective judgment (e.g., Q12, Q17) suggest a need for more nuanced training data or model adjustments to better capture human-like interpretations.
Observed Patterns and Anomalies:
- High Precision, Low Recall: Questions like Q7 (mention of something free) and Q16 (relatable characters) showed high precision but low recall, indicating the classifier identifies true positives accurately but misses many instances.
- High Recall, Lower Precision: Conversely, questions such as Q5 (incentive to buy) showed high recall, capturing most relevant instances but sometimes misclassifying irrelevant ones.
- Data Distribution: Anomalies in classifier performance can often be traced back to the uneven distribution of certain features in the training dataset, suggesting a need for more balanced and comprehensive training data.

Future Work

Future efforts will focus on enhancing model capabilities through:

Attention Mechanisms: Incorporating attention mechanisms to better capture contextual information.
Audio Model Integration: Adding audio analysis to enrich multimodal understanding.
Computational Resources: Leveraging more powerful computational resources for improved training and inference.