Some Approaches
Towards Grasping Intelligence

   Institute for Cognition and Robotics (CoR-Lab) &
   Excellence Cluster Cognitive Interaction Technology (CITEC)
   Bielefeld University

DRIVSCO Workshop Juodkrante 2009 #2    Helge Ritter 

The Way of Divine Inspiration

DRIVSCO Workshop Juodkrante 2009 #3    Helge Ritter 

Cognitive Interaction Technology

Intelligent Motion:

How to move around,

manipulate objects, use tools?

Attentiveness:

What mechanisms enable

to share attention with humans, and

to ignore irrelevant detail?

Situated Communication:

How can we coordinate

language,perception and action

to facilitate cooperation with humans?

Memory and Learning:

How can a system

acquire, store and retrieve knowledge

and improve its capabilities by learning?

DRIVSCO Workshop Juodkrante 2009 #4    Helge Ritter 

Interdisciplinarity

Cognitive Interaction at the crossroads of several disciplines: Biology coordination of walking motion, neural control strategies, "intelligent" reflexes already in insects, ... Sports science experimental-analytical methods for elucidating mental representation structures; training methods for optimizing action sequences Linguistics Joint-Action perspective on language, speech generation, language as part of interaction and cooperation, Psychology memory systems, attention control, cognitive decision models

DRIVSCO Workshop Juodkrante 2009 #5    Helge Ritter 

Organization of Action Memory

Sport science:

Organization of Memory for Actions
Thomas Schack + COALA Group

Language and actions Analysis of similarity judgements Optimization of training procedures

DRIVSCO Workshop Juodkrante 2009 #6    Helge Ritter 

Large-Scale System Prototypes: ASIMO Robots

Bielefeld-based CoR-Lab (Cognition & Robotics Lab) closely cooperating with CITEC Cluster platform enabling higher autonomous functions verification of theoretical ideas significant "critical mass" of functionalities realistic interaction with humans important aspect common daily tasks define non-trivial benchmarks

DRIVSCO Workshop Juodkrante 2009 #7    Helge Ritter 

Five Reasons to study manual intelligence

hands are our most important tools hand control is likely to have shaped large portions of our brain hands challenge us beyond perception: they are for interaction! reasonable robot hands and sensors are becoming available manual intelligence is naturally grounded in physical interaction

DRIVSCO Workshop Juodkrante 2009 #8    Helge Ritter 

The Richness of Manual Intelligence

DRIVSCO Workshop Juodkrante 2009 #9    Helge Ritter 

A Short History of Manipulation

DRIVSCO Workshop Juodkrante 2009 #10    Helge Ritter 

Some Robot Hands

DRIVSCO Workshop Juodkrante 2009 #11    Helge Ritter 

The human yardstick

DRIVSCO Workshop Juodkrante 2009 #12    Helge Ritter 

Understanding Physical Contact

Physical contact key mediator of most interactions!

Cognitive agents need to understand many aspects of contact:

DRIVSCO Workshop Juodkrante 2009 #13    Helge Ritter 

The Structure of Contact Patterns

DRIVSCO Workshop Juodkrante 2009 #14    Helge Ritter 

Bielefeld Grasping Lab

7 DOF PA-10 robot arm

dextrous robot hand 20 DOFs

pneumatic muscles pressure pulse-driven inherent compliance

Grasping Lab: hand-arm setup

DRIVSCO Workshop Juodkrante 2009 #15    Helge Ritter 

The Complexity of Manual Interaction

DRIVSCO Workshop Juodkrante 2009 #16    Helge Ritter 

Physics-based simulation

delivers significant portion of "contact cognition" several parts missing: parse visual image into physics simulation (but some examples for line drawings) simulation delivers "low-level reasoning" only in many ways too detailed and brittle ("spaghetti") no directedness, no planning, no choices, ...

DRIVSCO Workshop Juodkrante 2009 #17    Helge Ritter 

Haptic Shape Recognition

Somatosensory area: similarities to visual cortex simplest approach: apply pattern recognition methods to sensor images needs to be complemented by active exploratoin and spatio-temporal integration

DRIVSCO Workshop Juodkrante 2009 #18    Helge Ritter 

Initiating contact patterns:

DRIVSCO Workshop Juodkrante 2009 #19    Helge Ritter 

Shaping Contact Patterns for Stability

what finger positions achieve "force closure"? what finger positions maximize resistance to external forces and torques?

find contact positions with overlapping friction cones! maximize "stability polytope" in wrench space!

DRIVSCO Workshop Juodkrante 2009 #20    Helge Ritter 

The "Clockwork fallacy"

Example: A grasping "clockwork" acquire precise situation data: geometric shape of object friction properties dito for manipulator synthesise optimal target grasp plan trajectory from current state augment with feedback action from sensor events

DRIVSCO Workshop Juodkrante 2009 #21    Helge Ritter 

Cage based strategy

DRIVSCO Workshop Juodkrante 2009 #22    Helge Ritter 

Pre-Grasps as Basis Elements

DRIVSCO Workshop Juodkrante 2009 #23    Helge Ritter 

Some "Live" Examples

DRIVSCO Workshop Juodkrante 2009 #24    Helge Ritter 

Benchmark Results

Benchmark for Grasping: 21 common objects, 4 grasp types:

"force grasp" (Force) "precision grasp" (Pre)

"3-finger grasp" (III) "2-finger grasp" (II)

10 trials per object & grasp (840 trials total) Obj Succ Grasp 1 10 Force (+III) 2 10 (special) 3 10 (+III,-Force) 4 10 Force 5 10 Force (+III) 6 10 Force (+III) 7 9 Force (-III) 8 8 III (+Force,-Pre,-II) 9 8 Force 10 9 Force 11 7 Pre 12 6 III 13 7 Force 14 7 III (+Force,-Pre,-II) 15 6 Force (-III) 16 5 III 17 4 Pre 18 3 Pre 19 4 Pre 20 0 III 21 0 Pre

DRIVSCO Workshop Juodkrante 2009 #25    Helge Ritter 

What can we learn for achieving sensori-motor intelligence?

DRIVSCO Workshop Juodkrante 2009 #26    Helge Ritter 

Grasping needs Knowledge

Many influence factors depend on prior knowledge

firmness friction properties weight

function purpose dangers

DRIVSCO Workshop Juodkrante 2009 #27    Helge Ritter 

Learning from Human Grasps

capture human grasps how do human grasps differ? generate "grasp maps" use "grasp maps" to guide interaction

DRIVSCO Workshop Juodkrante 2009 #28    Helge Ritter 

Extracting human grasping knowledge from data

DRIVSCO Workshop Juodkrante 2009 #29    Helge Ritter 

UKR Manifold Construction

P(x,y) =   ∑ i K(y−yi)K(x−xi)     ⇒     y(x) =   ∑i   yi K(x−xi)   ∑i   K(x−xi)

DRIVSCO Workshop Juodkrante 2009 #30    Helge Ritter 

Navigation within learned "Screw-Manifold"

DRIVSCO Workshop Juodkrante 2009 #31    Helge Ritter 

Using contact patterns for identification

Idea: create "haptic image database" for studying object identification based on (robot) touch sensation

robot moves touch sensor around object simple (passive) exploration algorithm acquisition of 16x16x100 haptic space-time-pattern database for 16 household or toy objects neural network classifier up to 96% recognition rate

DRIVSCO Workshop Juodkrante 2009 #32    Helge Ritter 

Which Features carry relevant information?

Feature No   Feature Description 0-5   Center of gravity (FS,Q,C) 6-11   Distance CG - center of sensor (FS,Q,C) 12   Value maximum texel (FS) 13   Position maximum texel (FS) 14   distance maximum texel from center of sensor (FS) 15   Value minimum texel (FS) 16   Position minimum texel (FS) 17   Distance minimum texel from center of sensor (FS) 18-23   mean texel (FS,Q,C) 24   Value median texel (FS) 25-30   Std deviation (FS, Q, C) 31-36   Third moment/skewness (FS,Q,C) 37-43   Distance max/min texel from center (FS,Q,C) 44   Number of texels > mean + 0.5 std dev (FS) 45   Power spectrum (FS) 46-51   3x3 window around max texel (FS,Q,C)

C 4.5 Decision Tree feature selection according to max Shannon Entropy H = − ∑i N pi log(pi) user specified parameter: nr N of feature clusters to use

DRIVSCO Workshop Juodkrante 2009 #33    Helge Ritter 

Results

DRIVSCO Workshop Juodkrante 2009 #34    Helge Ritter 

Information Contents vs. Cluster Granularity

DRIVSCO Workshop Juodkrante 2009 #35    Helge Ritter 

Towards Integration of Capabilities

DRIVSCO Workshop Juodkrante 2009 #36    Helge Ritter 

Some Challenges Ahead

Dealing with deformable objects I: Shaping clay from haptic feedback

Dealing with deformable objects II: Manipulating paper

Creating a "Grasp Vocabulary"

Interaction models from data

DRIVSCO Workshop Juodkrante 2009 #37    Helge Ritter 

Grasping and Language

Sprache handelt vom Erfassen von Begriffen und Bedeutung

Our hands embody a large

Vocabulary of capabilities:

DRIVSCO Workshop Juodkrante 2009 #38    Helge Ritter 

Principle vs. Complexity: or
How is our essential knowledge organized?

combustion engine: basis principle is simple! BUT: becomes only viable through numerous auxiliary details efficiency and robustness of a commercially successful engine are highly distributed over very many components Large parts of our essential know-how may be of this nature! encouraging fit with the neural style of information representation!

DRIVSCO Workshop Juodkrante 2009 #39    Helge Ritter 

Some Key Questions for MI

DRIVSCO Workshop Juodkrante 2009 #40    Helge Ritter 

Thank you
for your attention!

Coworkers: Robert Haschke Benjamin Inden Hendrik Kösling Jonathan Maycock Michael Pardowitz Alexandra Barchunova Matthias Behnisch Daniel Dornbusch Andrea Finke Christoph Elbrechter Risto Koiva Alexander Lenhard Frank Röthling Florian Schmidt Matthias Schöpfer Carsten Schürmann Jan Steffen Katharina Tluk von Toschanowitz