Computer Vision Input

Here you will find all the necessary resources for the computer vision input.

Timeline

Introduction - What is Computer Vision? (9:00 - 10:00)

This is a general presentation about computer vision. The main questions that the video should answer are, what computer vision, where it is used and how it is related to interaction design in general?

Watch the presentation and write down questions and thoughts that come to your mind! We can discuss them later in the Zoom conference.

📼IAD-CV-Introduction Video

• Microsoft - Training deep control policies for the real world

• Deep Vision - Understanding machine perception

• Kollision - Urban Future

• Moovelab - Who want’s to be a self driving car?

• Philipp Schmitt - Camera Restricta

• ClearView AI - Brüssel - Facial Recognition NZZ

Zoom Conference (10:00)

Let’s talk together about computer vision in general and discuss questions and annotations.

https://zhdk.zoom.us/j/4710337215

Topcis

• What do you expect of today?

• How was it to watch the presentation?

• Specific ideas for your group project?

• How is the input structured?

• Use Slack / Zoom for help

Processing & Media (10:30 - 12:00)

Do your own first steps with processing and computer vision by creating a paint tool with your own interaction method.

📼IAD-CV-ProcessingMedia Video

• Video Library 2.0

• PImage (Processing)

• Filter (Processing)

• Computer Vision Wiki IAD

• How do common virtual reality tracking systems work?

Task 1

Create a new processing sketch which loads this image and displays it. How can we find the position of the moon?

int w = img.width;
int h = img.height;

for(int y = 0; y < h; y++) {
  for(int x = 0; x < w; x++) {
    
    // do something
    img.set(x, y, color(0)):
  }
}

Task 2

Install the Video Library 2.0 and display the webcam image!

Task 3 a)

Combine brightest point tracking with the webcam capture to track the brightest point in your webcam video.

Task 3 b)

Create an own drawing software that is controlled by the light of your smartphone flashlight.

• Think about additional features you could implement!

Zoom Conference (13:00)

Discuss the morning & share what everyone has created.

https://zhdk.zoom.us/j/4710337215

• Technical Input

• Was it possible to follow?

OpenCV (15:00 - 16:30)

Getting deeper into computer vision by using OpenCV, a framework to analyse images with traditional computer vision algorithms.

Task 4

Use opencv to track the brightest spot.

• opencv.blur(10);

• PVector location = opencv.max();

opencv.loadImage(cam);

// process
opencv.blur(10);
PVector location = opencv.max();

image(opencv.getSnapshot(), 0, 0);

noFill();
stroke(100, 255, 80);
circle(location.x, location.y, 10);

import processing.video.*;
import gab.opencv.*;
import org.opencv.core.*;

Capture cam;
OpenCV opencv;

void setup() {
  size(500, 500, FX2D);

  String[] inputDevices = Capture.list();
  cam = new Capture(this, 640, 480, inputDevices[0]);
  cam.start();

  opencv = new OpenCV(this, 640, 480);
}

void draw() {
  background(0);

  if (cam.available())
    cam.read();

  opencv.loadImage(cam);

  // do something

  image(opencv.getSnapshot(), 0, 0);
}

Task 5

Write your own sketch that is able to detect motion! Use either background subtraction, image difference or your own algorithm.

How will you notify yourself?

// import the following at the top of your sketch:
import org.opencv.core.*;

// run this after you have loaded the image to opencv
int count = Core.countNonZero(opencv.getGray());

import processing.video.*;
import gab.opencv.*;
import org.opencv.core.*;

Capture cam;
OpenCV opencv;

PImage last = null;

void setup() {
  size(640, 480);

  String[] inputDevices = Capture.list();
  cam = new Capture(this, 640, 480, inputDevices[0]);
  opencv = new OpenCV(this, 640, 480);
  
  cam.start();
}

void draw() {
  background(0);
  
  if(cam.available())
    cam.read();
  
  opencv.loadImage(cam);
  
  if(last != null) {
    opencv.diff(last);
    opencv.threshold(40);
  }
  
  last = cam.copy();
  
  float count = Core.countNonZero(opencv.getGray());
  float difference = count / (cam.width * cam.height) * 100;
  
  if(difference > 10.0) {
    println(millis() + "WARNING!");
  }

  image(opencv.getSnapshot(), 0, 0);
  
  fill(0, 255, 255);
  text("Difference: " + difference, 20, 20);
}

Task 6

Use Contour Detection to implement a multi-user drawing system.

import processing.video.*;
import gab.opencv.*;

Capture cam;
OpenCV opencv;

PImage last = null;

void setup() {
  size(640, 480);

  String[] inputDevices = Capture.list();
  cam = new Capture(this, 640, 480, inputDevices[0]);
  cam.start();

  opencv = new OpenCV(this, 640, 480);
}

void draw() {
  background(0);

  if (cam.available())
    cam.read();

  opencv.loadImage(cam);
  opencv.threshold(240);

  blendMode(BLEND);
  image(cam, 0, 0);

  noFill();
  stroke(255, 0, 0);
  strokeWeight(3);
  for (Contour contour : opencv.findContours()) {
    contour.draw();
  }
}

Zoom Conference (15:30)

Discuss the current state and mood.

https://zhdk.zoom.us/j/4710337215

Tuesday 30.03.2020

Depth Sensing (Tuesday 9:00 - 10:00)

Watch the video about depth sensing and on how machines are able to perceive our world.

📼IAD-CV-DepthSensing Video

• DenseDepth - High Quality Monocular Depth Estimation via Transfer Learning [paper][code]

Zoom Conference (Tuesday 10:00 - 10:30)

Questions, notes and comments.

Deep Vision Input (Tuesday 10:30 - 11:45)

This input is about computer vision combined with modern machine learning techniques. It is also an introduction about the topic of machine learning in general. How does a machine learn? How does it see?

Task 7

Download the Deep Vision Library (direct download link) for Processing and install it into your library folder. Try out the examples and see how the library works.

Task 8

Use the FaceDetector Webcam Example and try to combine it with the emotion classifier. To create an emotion detector, use the following code sample:

FERPlusEmotionNetwork emotionNetwork = vision.createFERPlusEmotionClassifier();
emotionNetwork.setup();

// run on a single face
emotionNetwork.run(face)

// run on multiple detected images
emotions = emotionNetwork.runByDetections(testImage, detections);

A Java example on how to combine these two methods you can find here, it is Processing as well, but you may have to adapt it slightly.

Furthermore: Have a look at the Gender & Age prediction example and try to add it to the sketch as well.

Data Analysis and Connection (Friday 13:00-17:00)

• Spatial Interaction Examples

• MQTT / shiftr.io

• Open Sound Control (OSC)

• Wekinator

Task 9

Try to analyze your own set of images and think about what you could use this information for. Use the example provided here.

Task 10

Try out the mouse example to train your own classifier.

• What else could you classify?

• Where do you get the data from?

• How can you include this into your main project?

Content

Please do not share the slides and the content from this page. The images and videos in the presentations are sourced but not licensed.