Fundamentals of people surveillance Prof. Rita Cucchiara Prof Rita Cucchiara Facoltà di Ingegneria “Enzo Ferrari” Università di Modena e Reggio Emilia (Italy) http://Imagelab.ing.unimore.it Abstract People surveillance is one of the hottest topics of the last decade in computer vision and pattern People surveillance is one of the hottest topics of the last decade in computer vision and pattern recognition research; it covers all aspects of computer engineering and computer science, models and algorithms, ,software and hardware architecture, real‐time data processing and management, machine learning and knowledge‐based reasoning, to detect in the space‐time dimensions people living in the real world starting from tsunami of visual data, acquired by networks of static and moving cameras, recognizing their presence also in cluttered and crowded environment, i ii h i l i l d d d d i extracting information about their aspect, motion, action and interaction and eventually behavior. Although similar approaches are proposed for many different targets ( people, vehicles, aircrafts, Alth h i il h df diff tt t ( l hi l i ft animals etc), this course addresses mainly people, representing the principal focus of interest in surveillance for security and safety and in multimedia analytics for forensics, and more complex than other objects due to their non‐rigid structure, unpredictable motion and activity. This short course is a survey of fundamentals of people surveillance: ‐ People detection with motion analysis ( background suppression and motion vector processing) – People detection with appearance cues ( pedestrian detection) , People tracking with single and multiple heterogeneous sensors – People action analysis according with their appearance and motion ( trajectory, body motion..) i ) Many references to the main international research results will be proposed to assess the state‐of‐the‐ art of people surveillance. Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Agenda • • • • • • • Introduction Video Surveillance and Forensics Video Surveillance and Forensics Design Aspects for surveillance CV & PR for people surveillance People shape detection People shape detection People behavior by trajectory analysis Conclusion Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Agenda • • • • • • • Introduction Video Surveillance and Forensics Video Surveillance and Forensics Design Aspects for surveillance CV & PR for people surveillance People shape detection People shape detection People behavior by trajectory analysis Conclusion Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Video surveillance Video surveillance Video‐surveillance concerns models, techniques and systems for • acquiring videos about the 3D external world, • detecting targets along the time and the space, • recognizing interesting or dangerous situations, interesting or dangerous situations, • generating real‐time alarms • recording meaningful data about the controlled scene. Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Multidisciplnary C Context Imaging, g g Image processing, Pattern recognition, Computer vision Machine learning Physics Computer architecture Digital Electronics Ubiquitous computing, networking, Wi d i l Wired‐wireless Communication Multimedia Data managing, knowledge representation AI… Video surveillance Video surveillance Research world • Commercial world ‘60‐70 Hardware • ‘60 ‐70 Analogue cameras ‘80 Military research • ‘80 Digital CCTV systems ‘90 Traffic monitoring • ’90 Digital surveillance systems ‘00 People surveillance • ‘90 Network surveillance systems ‘10 ? Life surveillance • ’10? Ubiquitous, WAN systems Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Vid Video surveillance ill systems** Universal Multimedia Access Acquisition Wired/Wirless Network Storage (Remote) Display *R. Cucchiara,"Video sorveglianza per l'individuazione di persone e l'analisi comportamentale“ in Safety&Security, 2010 Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Video surveillance Prof. Rita Cucchiara – Università di Modena e Reggio Emilia Commercial Video surveillance …hardware Gs3 Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Commercial Video surveillance …software Gs3 Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Videosurveillance market IMS Research (2010) China: 10 million security cameras will be shipped for domestic consumption in China in 2010. Network security cameras are forecast to account for only 3% of the market in terms of unit shipments. Russia: over 20% revenue growth year‐on‐year for the CCTV and video surveillance equipment market in Russia in 2010 and 2011 At EMEA (Europe, Medium East, Africa) growth 33% /year in hardware for video surveillance . Software di video analytics video surveillance Software di video analytics software in surveillance software in surveillance : growth 10%/year (215 M£ in 2009 ) Frost & Sullivan Frost & Sullivan (Dic. 2008) (Dic 2008) 2009 British Government UK : 80 M£ 2009 Department of Homeland Security (DHS) USA 239 M$ Research funds? DARPA FP7 ,JLS, SMEs Companies, ( Miur ) Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Commercial projects Commercial projects in 2010: Chicago: Virtual in 2010: Chicago: Virtual Shield Chicago (2006‐) (2006 ) Partner : IBM Whatson Research 3000 cameras & video analytics (215M$) Chicago Operation Virtual Shield R. Cucchiara,"La visione artificiale per la videosorveglianza“ in Mondo Digitale, vol. 8, n. 3, pp. 39‐47, 2008 Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it IBM S3 Hybrid Surveillance Solution IBM S3 Hybrid Modules: SSE: processes SSE: processes data from data from sensors and generates XML meta‐data. MILS: provides the infrastructure for indexing, g, retrieving and managing event meta‐data [Tian08]Tian, Y.L, [Tian08]Tian, Y.L, Brown, L.M., Brown, L.M., Hampapur, A., Hampapur, A., Lu, M., Lu, M., Senior, A., Senior, A., Shu, C., Shu, C., IBM smart IBM smart surveillance system (S3): system (S3): event based video surveillance system with an open and extensibleframework, MVA(19), 2008, Commercial projects Commercial projects in 2010: Golden shield in 2010: Golden shield program Beijing Cina: Cina: “china china golden shield program golden shield program” Partner : IBM, Honeywell and General Electric 200.000 cameras(2008‐) China's All‐Seeing Eye: Shenzhen 2 million cameras(2009‐) Biometry and commercial datai y ((VISA..) ) Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Research projects in in ‘90‐’00: 90 00: Traffic Traffic surveillance ‘90 t ffi surveillance ‘90.. traffic ill D. Koller, K. Daniilidis, H.‐H. Nagel Model‐Based Object Tracking in Monocular Image Sequences of Road Traffic Scenes (1993) IJCV QUEUE Detection. Dr. Porkili et al Mi bi hi El Mitsubishi Electric Research Labs, USA 2004 i R h L b USA 2004 Detecting stopped vheicles in highways. ImageLab, Modena (I) ‐ Traficon (B) WACV 2004 Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Research projects in Human in Human Surveillance ‘00: 00: Carneige Mellon University VSAM Prof. T. Kanade VSAM Carneige Mellon University USA Carneige Mellon University, USA Darpa Program 1997‐2000 http://www.cs.cmu.edu/~vsam/ ObjectVIDEO.. Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Research projects at Univ. of San Diego: DIVA & ATON Projects Prof. M. Trivedi UCSD San Diego DIVA, DIVA, People and stopped vehicle automatic detection in dangerous zones Andrea Prati, Ivana Mikic, Mohan M. Trivedi, Rita Cucchiara: Detecting Moving Shadows: Algorithms and Evaluation. IEEE Trans. Pattern Anal. Mach. Intell. 25(7): 918‐923 (2003) Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Research projects at Univ. of Central Florida M. Shah et Al. 2007‐2009 The KNIGHT Project detection and tracking The KNIGHT Project: detection and tracking multiple people, •the Florida Department of Transportation, • Orlando Police Department, DARPA Orlando Police Department DARPA •University of Central Florida COCOA Project : tracking from UAV COCOA Project : tracking from UAV WHERE I AM project: auto‐tracking Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Research projects at i‐LIDS‐ UK i‐LIDS Imagery library for intelligent detection systems the government's benchmark for video‐based detection systems The government is committed to promoting the development of VBDS to help in policing and counter‐terrorist operations. Home Office Scientific Development Branch (HOSDB) Dr. PAUL HOSMER – 5 cameras – 1.35 Million frames 1 35 Milli f – Single and multiple target – 1000+ target events IEE International Symposium on Imaging for Crime Detection and y p g g Prevention Dec.2009 London Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Research projects at Univ at Univ of Modena and Sidney : Modena and Sidney : Automatic abandoned pack detection Project Automatic real‐time detection of infiltrated objects for security of airports j j y p and train stations (2006‐2008) • Imagelab University of Modena • University of Technology Sidney (Australian Research Council) University of Technology Sidney (Australian Research Council) Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Research projects at Univ. Of at Univ Of Modena: Modena: Surveillance and privacy 2006‐2008 LAICA Laboratorio di Ambient Intelligence per una Città Amica Regione Emilia Romagna In 1 1 P Telematico Regione Emilia Romagna In 1.1 P. Telematico ImageLab Comune Reggio Emilia WTI (Bridge 129) Univ. Bologna, Modena, Parma… FREE SURF Free Surveillance in a privacy respectful way MIUR PRIN 2006‐2008 Prof. Rita Cucchiara – Università di Modena e Reggio Emilia Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Research projects in Behavioral Analysis … Many……. Dr K.Sudo et al. NTT Cyber Space Lab, Japan Detecting anomalies at ATM (a), (b) normal (c ) (d) abnormal (c ),(d) abnormal Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Agenda • • • • • • • Introduction Video Surveillance and Forensics Video Surveillance and Forensics Design Aspects for surveillance CV & PR for people surveillance People shape detection People shape detection People behavior by trajectory analysis Conclusion Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Video Surveillance & Video Analytics Intelligent, smart, Intelligent, smart, Video Surveillance To Video Analytics To Video and Vid d Multimedia Forensics H Human‐in‐the‐loop i h l Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it (When) multimedia meets surveillance and forensics in people security and forensics in people security Multimedia The use of different media contents (text audio images, video, animation interactivity…) to convey information for users Di i l f Digital forensics i A form of digital investigation that can be entered in a legal court entered in a legal court Forensic video analysis The scientific examination, f comparison and/or evaluation of video in legal matters for investigation People security The degree of protection of The degree of protection of people against damage, danger, criminal or terroristic actions Surveillance The manual/automatic monitoring of situation, behaviour, activity to generate alarm and record meaningful situation alarm and record meaningful situation Video Surveillance ll The acquisition and processing of visual data about the external word to detect target along the time and the space, g g p to recognize interesting and dangerous situations, to generate alarms and record data on the scene R.Cucchiara When multimedia meets surveillance and forensics in people security Keynote Workshop MIFOR2010 at ACM Multimedia 2010 Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Multimedia surveillance Multimedia surveillance • Th The integration of multimedia technologies and sensor i i f l i di h l i d networks constitutes the fundamental infrastructure of new g generation of multimedia surveillance systems, y , • where many different media streams (audio, video, text, 3D graphics, sensor data..) concur to provide an automatic analysis of the controlled environment and a support for l i f h ll d i d f human interpretation of the scene [Cuc05]. • Multimedia surveillance systems to – Enlarge the view – Enhance the view – Explore new views for human security employers [Cuc05]R. Cucchiara “Multimedia surveillance systems” Proc of VSSN’05 at ACM Multimedia Singapore 2005 Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Digital Forensic and Computer Forensics and Computer Forensics DIGITAL FORENSICS DIGITAL FORENSICS • “The digital revolution introduces digital trace of trace of our activity in Real Life” [Franke09] – Computer activities – Digital Social Interactions – Digital Evidence of analog Processes COMPUTER FORENSCIS COMPUTER FORENSCIS • “When computer are involved criminal activities” involved criminal activities [Kruse01]: – Tools for committing crimes – Substrate where crimes are committed are committed [Francke09] Franke ] K., Shriari , S. Computational p Forensics an Overview 2009 [Kruse01] Kruse, W., Heiser, J.: Computer Forensics: Incident Response Essentials. Addison Wesley, Reading (2001) Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Multimedia Forensic Multimedia Forensic MULTIMEDIA FORENSICS [[Bohme09]] • Many data are collected through sensors ONTOLOGY ON FORENSICS • Create a digital counterpart of reality • Digital data can be probative elements in many in many investigation (e.g. video, audio, photos..) “Data must be authentic and reliable” [Boheme09] Böhme, R., Freiling, F. C., Gloe, T., and Kirchner, M. 2009. Multimedia Forensics Is Not Computer Forensics. In Proc. of the 3rd international Workshop on Computational Forensics Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Computational Forensic is not Computer Forensic Computational Forensic refers to applying computer aided techniques to digital data understanding: – Assist in basic and applied research and data mining A i ti b i d li d h dd t i i – Establish or prove the scientific basis of a particular investigative procedure investigative procedure – Support the forensic examiner in their daily case work. “Modern crime investigation shall profit from the hybrid‐ intelligence of humans and machines” intelligence of humans and machines [Francke09] Franke ] K., Shriari , S. Computational p Forensics an Overview 2009 Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Computational Forensics Techniques • • • • • • • Signal / Image / Image Processing : one‐dimensional Processing : one dimensional signals and 2‐dimensional and 2 dimensional images are transformed for the purpose of better human or machine processing, Computer Vision : images are automatically recognized to identify Computer Vision : images are automatically recognized to identify objects, Computer Graphics / Data Visualization : two‐dimensional images or y three‐dimensional scenes are synthesized from multidimensional data for better human understanding, Statistical Pattern Recognition : abstract measurements are classified as belonging to one or more classes, e.g., whether a sample belongs to a k known class and with what probability, l d ih h b bili Data Mining : large volumes of data are processed to discover nuggets of information, e.g., presence of associations, number of clusters, outliers in a cluster a cluster, Robotics : human movements are replicated by a machine, and Machine Learning : a mathematical model is learnt from examples. [Francke09] Franke K., Shriari S. Computational Forensics an Overview 2009 Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Revisited Ontology of Forensic Computational Forensics •Surveillance •Biometrics •Bioinformatics •Data Mining •3D 3D Reconstruction Reconstruction •Document analysis… Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Examples Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Examples Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Multimedia (Video) surveillance Multimedia (Video) forenscis On‐line analysis On‐line analysis Off line analysis on logged Off‐line analysis on logged data Real time response Fast processing in large sets of data Many fixed conditions and pre‐defined d d fi d constraints i Often undefined camera settings and constraints and constraints Correlation between cameras Consistent labeling Recognition Correlation between objects j Re‐identification Mining…. Noise and uncerainity on (movement) data High variability of visual data Need of multimedia data management multimedia data management and analysis tools for people security Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Agenda • • • • • • • Introduction Video Surveillance and Forensics Video Surveillance and Forensics Design Aspects for surveillance CV & PR for people surveillance People shape detection People shape detection People behavior by trajectory analysis Conclusion Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Design Aspects: from hardware….. Design Aspects: from hardware Universal Multimedia Access Acquisition Wired/Wirless Network Storage (Remote) Display Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Design aspects: to software Design aspects: …..to Universal Multimedia Access Acquisition Wired/Wirless Network (Remote) Display (Remote) Display Storage Software for Analysis, Control, Prediction…. Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Architecture: Parallelism and multimodality Architecture: Parallelism and multimodality Parallesim aerial,… Modality Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Design aspects Design aspects • For multicamera, distributed surveillance and sensor networks 1 sensor topology 1. t l 2. 3. 4 4. 5. architecture topology p gy communication aspects d t ffusion data i data processing. Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Design aspects g p 1.1) Sensor topology: placement best best placement placement definition using linear programming and heuristics Using simulating annealing d definition f off a 3D Visibility b l model d l off a tag( for people) and optimization via binary integer programming [E. Hoester, R. Linheart Optimal Placement of Multiple sensors in M . Academic Press 2009, proc of ACM VSSN 2006 Mittal, L. Davis A general method for sensor planning in multi-sensory systems: extension to random occlusion Journal of Computer occlusion, Vision 2008 [J Zhao S. Cheung, TNnguyen ,MULTICAMERA SURVEILLANCE WITH VISUAL TAGGING AND GENERIC CAMERA PLACEMENT M . Academic Press 2009, and Proc id ICSDC 2007 Vm(x y θ r) Vm(x, y, θ, r), Design aspects g p 1.2) Sensor topology: learning detection detection the topology the topology of an existig camera camera network inferring the topology of a camera network by measuring statistical dependence between entrances and exits (GPS‐based performance analysis) p y ) [T.Ellis, D. Makris, J.Black Learning a multicamera topology Proc of VS& PETS2003 T.Ellis, D. Makris, J.Black Bridging the gap across cameras CVPR2004 [ieu, Dalley, Grimson G Inference f off non overlapping camera network topology by measurng statistical dependence Proc of ICCV 2005 [Besma R. Abidi, Nash R. Aragam, Yi Yao, Mongi A. A Abidi: Survey and analysis of multimodal sensor planning and integration for wide area surveillance ACM Computing Reviews 2009 From Ellis et al. Pets 2003 Design aspects g p 2) Architecture topology Centrali Centralized ed ( PRISMATICA project: ( PRISMATICA project central servers with dedicated nodes for cameras, audio, smart cameras and so on)) Semi‐distributed ( ADVISOR project: manyy independent p nodes each one connected with more cameras) Distributed network with embedded systems Distributed network with agent based communication [P. Lai Lo, J. Sun, s. Velastin Fusing visual and audio information in a distributed surveillance system ACTA Automatica SINCA 2003 [M. Valera Espina and S.A. Velastin, "Intelligent Distributed Surveillance Systems: A Review," IEE Proc. Vision, Image and Signal Processing, Apr. 2005, pp. 192—204 [M. Christensen, R. Alblas V2 design issues in distributed video surveillance systems Denemark 2000 [M. A. Patricio, J. Carbó, O. Pérez, J. García, and J. M. Molina Multi-Agent Framework in Visual Sensor Networks EURASIP Journal on Advances in Signal Processing Volume 2007 (2007), Design aspects g p 3) Communication aspects: Shared memory and synchonization and synchoni ation Bandwidth allocation Wireless and mobile protocols p Sensor Networks Security: In distributed I di t ib t d systems t Privacy & Authentication – A book covering all design aspects [C Regazzoni, V.Ramesh, G. Foresti Special Issue on video communication processing and understanding for third generation surveillance systems P O IEE 2001 [ Akyildiz, [. Akyildiz W W. Su, , Y. Y Sankarasubramaniam and E. Cayirci Wireless sensor networks: a survey Computer Networks Volume 38, 38 Issue 44, 15 March 2002 [ Tubaishat, S Madria Sensor networks: an overview - IEEE potentials, 2003 [Adrian Perrig , J.Stankovic , D. Wagner Security in wireless sensor networks Communications of the ACM Volume 47 , Issue 6 (June 2004) M Aghajan, Cavallaro eds. Academic Press 2009 Design aspects g p 4) Data fusion: Calibration of the multicamera the multicamera system, for object association across multiple camera sharing of information obtained by different type of sensors E.g. color and thermal cameras E.g. mobile, wireless, fire, sound… E.g. visual E i l and PIR sensors d PIR 5) Data processing: Computer Vision & pattern recognition [J.Han, B.Bhanu Fusion of color and infrared video for moving human detection Pattern recognition 2007 Y Tseng, Y Wang, K Cheng, Y Hsieh Imouse: an integrated mobile surveillance and wireless sensor system IEEE computer 2007 [. Cucchiara, A. Prati, R. Vezzani, L. Benini, E. Farella, P. Zappi, "Using a Wireless Sensor Network to Enhance Video Surveillance" in Journal of Ubiquitous Computing and Intelligence (JUCI), vol. 1, pp. 1-11, 2006 Agenda • • • • • • • Introduction Video Surveillance and Forensics Video Surveillance and Forensics Design Aspects for surveillance CV & PR for people surveillance People shape detection People shape detection People behavior by trajectory analysis Conclusion Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Algorithms for data processing data processing Multiple Stationary cameras Ptz Cameras Moving cameras Sensor Networks Airborne Ai b cameras Heterogeneous cameras Distributed cameras Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it From detecting to reasoning on people on people People detection p People segmentation Localization People tracking People Re‐identification Re identification Search Soft‐biometryy Face detection & recognition People Identification Motion analysis Activity analysis Identity assessment Biometry Single and and multiple objects & Posture, Gesture, Gate, Trajectories.. Single Single and multiple sensors Action ( & interaction) With environment analysis With moving objects With people Bheavior analysis Understanding Modelingg Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it System Architecture People segmentation and tracking PEOPLE DETECTION AND TRACKINGI WTH STATIC CAMERAS 2000 C. Wren, A. Azarbayejani, T. Darrell, and A.P. Pentland “Pfinder: Pentland, Pfinder: real-time real time tracking of the human body,” IEEE Trans. PAMI, ( 19) 7, 1997. C. Stauffer and W.E.L. Grimson, “Learning patterns of activity using real-time tracking,” IEEE Trans. PAMI, ( 22) 8, 2000. I. Haritaoglu, D. Harwood, and L.S. Davis, “W4: realtime surveillance of people and their activities,” IEEE Trans. PAMI, ( 22) 8, 2000. Tao Sawneir, Sawneir Kumar. Kumar Object tracking with bayesian estimation of dynamic layer representation IEEE Trans on PAMI 24,1 2002 PEOPLE TRACKING WITH MULTIPLE STATIC CAMERAS 2008 A. C. Sankaranarayanan, A.Veeraraghavan, and R.Chellappa, Object Detection, Tracking and Recognition for Multiple Smart Camera Proceedings of the IEEE | Vol. 96, No. 10, October 2008 S. Calderara, S Calderara A. A Prati, Prati R. R Cucchiara, Cucchiara “Bayesian Bayesian-competitive competitive Consistent Labeling for People Surveillance“ on IEEE Trans on PAMI, feb. 2008 Saad M. Khan and Mubarak Shah; Tracking Multiple Occluding People p by y Localizing g on Multiple p Scene Planes;; IEEE TRANS. ON PAMI, VOL. 31, NO. 3, MARCH 2009 R. Cucchiara, C. Grana, M. Piccardi, A. Prati “Detecting Detecting Moving Objects, Ghosts and Shadows in Video Streams“, IEEE Trans on PAMI, 2003 OCCLUSION DETECTION 2004 Nguyen, H.T. N H T Smeulders, A. S ld A Fast occluded object F t l d d bj t tracking by a robust appearance filter IEEE Trans on PAMI, 2004 Tao Zhao Nevatia, R. Tracking multiple humans in complex situations IEEE Trans. PAMI 2004 With forest of cameras With moving and mobile cameras and mobile cameras .. In crowd… ( Shah’ss Talk ACM MM2010) ( Shah Talk ACM MM2010) 2012 Background suppression: milestones et al. Background suppression: Gray level vs. Color Analysis Mixture of Gaussians Background and Shadow detection Background and layered representation Background with Background with kernel density …….. And other hundreds of papers… S.J. McKenna, S. Jabri, Z. Duric, A. Rosenfeld, and H.Wechsler, “Tracking groups of people,” Computer Vision and Image Underst.,( 80)1, 2000. C. Stauffer and W.E.L. Grimson, “Learning patterns of activity using real-time tracking,” IEEE Trans. PAMI, ( 22) 8, 2000. (1400 citation on google !) A SHORT SURVEY A. Prati, I. Mikic, M.M. Trivedi, R. Cucchiara, "Detecting Moving Shadows: Algorithms and Evaluation," IEEE Trans. on PAMI, July 2003 Tao Sawneir, Kumar. Object tracking with bayesian estimation of dynamic layer representation IEEE Trans on PAMI 24,1 2002 A.Elgammal, R.Duraiswami, D.Harwood, L.S. Davis Background and Foreground Modeling Using Nonparametric Kernel Density for Visual Surveillance Proceedings of the IEEE 2003 Detection and tracking: g At each frame Segmentation (into blobs) Tracking (observation model (observation‐model correspondence) At each frame Region of interest Region of interest selection Tracking ( d l b (model‐observation ti correspondence) Initial steps: Initial steps: DETECTION & TRACKING People tracking: milestones Many SURVEYs Most used techniques: Moeslund , Hilton, Kruger A survey of advanced in vision-based human motion capture and analysis CVIU vol 104 2006 Yilmaz Javed Shah Object tracking a survey ACM COMPUTING Survey vol 33 n 4 2006 y , A;; Chellappa, pp , Wu,, H;; Sankaranarayanan, R;Online Empirical Evaluation of Tracking Algorithms IEEE Trans. PAMI 2009 MeanSHIFT Particle filtering Appearance pp based trackingg K Kernel l based b d tracking ki Dorin Comaniciu, Visvanathan Ramesh: Mean Shift and Optimal Prediction for Efficient Object Tracking. ICIP 2000 M. Isard and A. Blake. A smoothing filter for condensation. In Proc. ECCV 1998 ECCV, 1998. ......... Tao Zhao Nevatia, Nevatia R. R Tracking multiple humans in complex situations IEEE Trans. PAMI 2004 …. ..Han, Comancio, Davis sequential kernel density approximation for real time visual tracking IEEE Trans. PAMI 2009 Single camera surveillance at Imagelab Single camera surveillance at Imagelab Detection with SAKBOT Statistical and knowledge and knowledge based Object deTection [TPAMI2003] Smoke analysis [MVA J 2009] y [ ] Abnormal bheavior [CVPR2008] Tracking with AD HOC Apperance based Discriminative Apperance based Discriminative Handling[ with Occlusion [ICPR2004] People and Posture classification p [[TSMC2005]] Recognition stopped vheicles Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it To parallel and distributed surveillance Surveillance systems: A review of commercial systems A review of hardware and software requirements A survey of multicamera and distributed surveillance Valera, M. Velastin, S.A. Digital Imaging Res. Centre, Kingston Univ., UK; Intelligent distributed surveillance ill systems: t a review i Vision, Image and Signal Processing, IEE Proceedings -2005 Hu, Tan, Wang, Maybank: A Survey on visual surveillance of object motion and bheaviors IEEE Trans. On system y man and cybernetics vol34 n 3 2004 A. C. Sankaranarayanan, A.Veeraraghavan, and R.Chellappa, Object Detection, Tracking and Recognition for Multiple Smart Camera Proceedings of the IEEE | Vol. V l 96, 96 No. N 10, 10 October O t b 2008 To parallel and distributed surveillance • Multicamera (multiview) surveillance • • • • ffully synchronized acquisition ; 1 frame f grabber with 1‐20 fixed and PTZ cameras; 1 (multiprocessor) computer for many cameras shared memory architecture Challenges: More precision, 3D reconstruction, consistent , labelingg in multiview, occlusion handling, people identification, beh avior analysis Distributed ( network camera) surveillance • • • loosely coupled acqusition and processing; and processing; potentially thousands of nodes with smart cameras and traditional network cameras message g p passing g architecture Challenges: Large coverage, communication, bandwidth, tradeoff‐ local computation and computation al power; less precision, multiple hypothesis generation , search for similarity Distributed surveillance and sensor network Multicamera surveillance + in addition freely moving cameras on vehicles, moving infrastructure, hand‐cameras Homography and data association and data association • In real context noise, errors in homography, p lack of planar constraints introduce uncertainly in the position From [A. C. Sankaranarayanan, A.Veeraraghavan, and R.Chellappa, Object Detection, Tracking and Recognition for Multiple Smart Camera Proceedings of the IEEE | Vol. 96, No. 10, October 2008 | V l 96 N 10 O b 2008 Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Multicamera surveillance Multicamera surveillance • Using multiple sensors/cameras l l / h many advantages: has d – Wider coverage of the scene – Multi‐modal Multi modal sensoring – Redundant data (improved accuracy) – Fault tolerance Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Data fusion for multicamera surveillance Data fusion multicamera surveillance Acquisition q Preprocessing Acquisition A i iti Preprocessing Fusion at Pixel level Calibration Homography. … segmentation tracking Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Data fusion for multicamera surveillance Data fusion multicamera surveillance Acquisition q Preprocessing Acquisition A i iti Preprocessing segmentation t ti segmentation t ki tracking tracking Fusion at Feature level (people axis) on Homography. … For consistent labeling Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it The solution at Imagelab HECOL (Homography and Epipolar-based COnsistent Labeling) Ground plane homography homography and epipolar and epipolar Ground‐plane geometry automatically computed from training videos Person’s main axis warped to the other view and Bayesian inference is used for y validate hypotheses [S. Calderara, A. Prati, R. Cucchiara, “Bayesian-competitive Bayesian-competitive Consistent Labeling for People Surveillance“ on IEEE Trans on PAMI, feb. 2008 [S. S Calderara, C A. Prati, R. Cucchiara, C “HECOL: Homography and Epipolarbased Consistent Labeling for Outdoor Park Surveillance" C Computer t Vision Vi i and d Image I Understanding, 2008 Automatic Homography computation • • Automatic learning phase to compute overlapping zones and ground‐plane homography. Take many correspondences among ground plane support points ( with ( a tracking algorithm for a single people) • Define the Entry Edge Field of Views E2oFoV using Least Square Optimization • Define the overlapping zones and the extremes points • Compute the homography from points correspondences E2oFoV EoFoV Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Video surveillance at ImageLab Video surveillance at ImageLab Modena Sakbot F F F Fixed camera Fixed camera Fixed/moving camera Segmentation Segmentation Tracking Tracking Ad‐hoc Geometry Recovery Hecol eco Homograpy & Epipoles Tracking ROI and model‐ based Tracking Sensors Sensor data acquisition Posture analysis Action analysis Trajectory Analysis Video surveillance Ontology PTZ Control HeadTracking Face selection Face Recognition & People Identification Bheavior recognition PTZ camera Mosaicing Segmentation & Tracking Consistent Labeling and Multicamera tracking VISOR Moses People detection Moving and mobile Moving and mobile camera P S M High Resolution Detection Face obscuration People Annotation Annotated video storage MPEG Streaming WEB Security control centers Mobile surveillance platforms l f Multicamera syst. Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Experiments in Surveillance 64 Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Esperiments in forensics in Modena… Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Example p Data correlation for manual identification Support of investigation Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it New.. New • • • Improving distributed d b d multi sensor l surveillance ll Motes CITRIC cameras and RFIDs Autocalibration of cameras and RFID sensor and RFID sensor network State of wearing tag person moving on a random pathway ICSDC’2010 boundary detected by RFID signal strength RFID reader operating area Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Agenda • • • • • • • Introduction Video Surveillance and Forensics Video Surveillance and Forensics Design Aspects for surveillance CV & PR for people surveillance People shape detection People shape detection People behavior by trajectory analysis Conclusion Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it From Moving visual objects to shape detection detection • People detection with 3D models • Pedestrian detection in still images with machine learning Zhao, Nevatia, Who, Segmentation and tracking on multiple humans in crowded environments IEEE T Trans. PAMI 2009 PAMI 2009 Dalal, Triggs, Histograms of oriented gradients for human detection CVPR2005 M. Enzweiler, d. Gravila M Enzweiler d Gravila Monoucluar pedestrian detection survey and experiments IEEE Trans PAMI dec. 2009 Dollar, P.; Wojek, C.; Schiele, B.; Perona, D ll P W j k C S hi l B P P.; Pedestrian detection: A benchmark CVPR 2009 • … Wojek, C.; Walk, S.; Schiele, B.; Multi‐ cue onboard pedestrian detection CVPR 2009 Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Pedestrian detection • When camera is moving • When Wh the environment th i t is i too t complex l • When the background is not available… People (pedestrian) detection with machine learning approach in still People (pedestrian) detection with in still images (and video) Feature detection Classification • • People No People Training set Test set ( on‐line data) Manyy features Many classifiers: – SVMS – Boosting classifiers with Sliding windows search Enzweiler, Gavrila, Monocular Pedestrian Detection: Survey and Experiments Trans on PAMI 2009 Cascade classifiers Biblio…… Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Histograms of Oriented Gradients Scan Input Image Resize window Divide into overlapping to 64x128 16x16 blocks Compute histogram of gradient over 9 directions Slides: courtesy of Slides: courtesy of www.andrew.cmu.edu 72 Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Hog + Deformable Part Model P. Felzenszwalb, D. McAllester, D. Ramanan “Object Detection with Discriminatively Trained part based Models” ,TPAMI‐2009 This work exploits the same HOG feature of This work exploits the same HOG feature of Dalal et al. The model of the target object is made of: a)) a coarse root filter (it corresponds to t filt (it d t Dalal model of pedestrian) b) several higher resolution part filters c) a spatial model for the location of each part relative to the root ((a)) ((b)) 73 Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it ((c)) Covariance and LogitBoost g Classifier O. Tuzel, F. Porikli, and P. Meer, “Pedestrian detection via classification on riemannian manifolds,” IEEE Trans. on PAMI, Oct. 08 F is a set of pixel pixel-wise wise features features. For generic region matching: For texture classification: For pedestrian detection: p Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Covariance and LogitBoost Classifier Casc 1 Casc 2 Casc N Extract Pixel‐wise Feature 1 Image g Sub Region R Mean, var E t t Pi l i F t Extract Pixel‐wise Feature 2 2 M Mean, var Extract Pixel‐wise Feature M Mean, var Covariance CR (MxM matrix, sym pos def) Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Covariance Descriptor LogitBoost Classifier on Riemannian Manifolds Casc 1 Casc 2 Linear Logistic Regressor Casc N on Riemannian Manifolds Euclidean Space needed Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Examples Dalal Tuzel Felzenszwalb 77 Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Ped. Detection with Sliding Window On each frame On each frame Apply pedestrian classifier on each window Exhaustive sliding window approach 78 Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Ped. Detection with Sliding Window • Two problems 1. Accuracy • Many false positives • Localization errors 2. Computation time Two approaches Exploiting other cues 1) Learning context 2) Using relevance feedback Exploiting statistics 1) Learning distribution 2) Multi‐stage search p g • Proportional to sliding windows size and overlap • Higher in Riemannian manifold Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it 1) Using context and relevance 1) Using and relevance feedback LogitBoost g Classifier on Riemannian Manifolds Casc 1 Casc 2 Casc N detection on negatives (precision) increases detection on positives (recall) decreases To increase precision without affecting recall.. Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Estimate the Size of Pedestrian within the Frame G.Gualdi, G G ldi A. Prati, R. A P i R Cucchiara, "Covariance Descriptors on Moving Regions for Human Detection in Very Complex Outdoor Scenes" in ACM/IEEE ICDSC 2009 Exploit Pedestrian detector to infer pedestrian size in the frame p p Sliding Slidi Window Pedestrian Detector RANSAC + LSQ with linear model of perspective Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it h(x,y) 8 1 Relevance Feedback (1/2) Training Dataset Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it 8 2 Relevance Feedback (2/2) RELEVANCE FEEDBACK (1) IMPLICIT (it is automatic) (2) EXPLICIT (it needs user assessment) Estimate background images => negative training set g g Response of the Ped Classifier Positives Training Dataset Negatives Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it 8 3 G. Gualdi, A. Prati, R. Cucchiara,"Perspective and Appearance Context for People pp p Surveillance in Open Areas" in Proceedings of the 2nd International Workshop on Use of Context in Video Processing (UCVP 2010), at CVPR 2010 Experimental Results Perspective Estimation 10% ~300K Windows 90% ~30K Windows Plain ped. detection ped detection With perspective With persp + rel. + rel feed + rel. feed Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it 8 4 Experimental p Results Perspective Estimation and Relevance Feedback P ii % Precision% R ll% Recall% Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it 8 5 2) Use statistics: multi‐stage 2) Use statistics: multi stage particle windows A probabilistic bayesian paradigm for object detection: Detection is achieved with multi‐stage search Estimate obj. detection as a pdf Use particle windows instead of sliding windows particle windows instead of sliding windows G. Gualdi, A. Prati, R. Cucchiara,"Multi‐stage Sampling with Boosting Cascades for Pedestrian Detection in Images and Videos” ECCV 2010 Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it 2) Use statistics: multi‐stage particle windows The measure of each sample is a function of the rejection level: d detection response i Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Region of support • Often there is a basin of attraction in position and size Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Face detection Face detection • Also with viola and jones face detector Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Experiments with m=5 stages Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it In Video: Exploit Bayesian Recursive Filter In Video: Exploit Bayesian prior i likelihood predicted di t d posterior measurements t sampled detections l d d t ti Final detections Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Experimental Results Miss Rate vs False Positives Per Image Dollar, P.; Wojek, C.; Schiele, B.; Perona, P.; Pedestrian detection: A benchmark CVPR 2009 Tuzel And Our solution Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it 9 4 Experiments • Save time • Or same time and Better accuracy Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Standard SW 96 Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it MS‐PW 97 Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Agenda • • • • • • • Introduction Video Surveillance and Forensics Video Surveillance and Forensics Design Aspects for surveillance CV & PR for people surveillance People shape detection People shape detection People behavior by trajectory analysis Conclusion Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it After people detection… People search multimedia Forensics Surveillance People search for forensics and surveillance; Why? 1) search people for answering a specific query ( i look for people with sun glasses and a blue jaked with a red luggage..) 2) search people similar to a given shape (similarity search, CBIR) 3) search people moving in an area or with a given behaviour ( people data and metadata annotation for search and mining) Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it After people detection… re‐identification People P l Re‐identification Search for similarity (in a plain database of (in a plain database of images and videos) multimedia Forensics Surveillance Extension of the tracking problem ( in videos) • The The tracking problem, aims at finding an association between tracking problem aims at finding an association between prediction and observation. • Tracking matchs Tracking matchs a previously seen target if it appears again in a previously seen target if it appears again in the same camera, after a short time, in a position close to the previous one, and with a similar appearance. Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Re identification: a short survey Re‐identification: a short survey • Many dimension ofPapers/year the problem 10 9 8 7 6 5 4 3 2 1 0 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it 2010 Many research works 2006 2008 2010 Sym m bod etry d ym r ode iven 3D l [Far bod 10] ym ode l [B al10 ] 2004 10-s lice mod el [B ir05 ] first bod ym ode l [G he0 6] 2002 Com disjoi mon nt bu t clo gro [Bla und pl se a 02, Jav0 ne disj oint 3] - CB IR l ike 3-sl [L ice mod an03] el [L an0 3] 2000 ove r Hom lapped ogra came phy ra base s d [C ai98 ] 1998 • [Cai98] [Cai98] Q. Cai Q Cai and J.K. Aggarwal, and J K Aggarwal “Automatic Automatic tracking of human motion in indoor scenes across multiple synchronized video streams, tracking of human motion in indoor scenes across multiple synchronized video streams ” Sixth Sixth International Conference on Computer Vision 1998, pp. 356‐362. • [Bla02] J. Black, T. Ellis, and P. Rosin, “Multi view image surveillance and tracking,” Proceedings of Workshop on Motion and Video Computing, 2002., IEEE Comput. Soc, 2002, pp. 169‐174. • [Jav03] O. Javed, Z. Rasheed, K. Shafique, and M. Shah, “Tracking across multiple cameras with disjoint views,” Proc. IEEE International Conference on Computer Vision,2003, pp. 952‐957 vol.2. • [Lan03] M. Lantagne, M. Parizeau, and R. Bergevin, “VIP: Vision tool for comparing Images of People,” Vision Interface, 2003. • [Bir05] N.D. Bird, O. Masoud, N.P. Papanikolopoulos, and A. Isaacs, “Detection of Loitering Individuals in Public Transportation Areas,” IEEE Transactions on Intelligent Transportation Systems, vol. 6, Jun. 2005, pp. 167‐177. • [Ghe06] N. Gheissari, T.B. Sebastian, and R. Hartley, “Person Reidentification [Gh 06] N Gh i i T B S b ti d R H tl “P R id tifi ti Using Spatiotemporal Appearance,” Conference on Computer U i S ti t lA ”C f C t Vision and Pattern Recognition 2006, pp. 1528‐1535. • [Far10] M. Farenzena, L. Bazzani, A. Perina, V. Murino, and M. Cristani, “Person re‐identification by symmetry‐driven accumulation of local features,” Conference on Computer Vision and Pattern Recognition, IEEE, 2010, pp. 2360‐2367. • [[Bal10] D. Baltieri, R. Vezzani, and R. Cucchiara, “3D Body Model Construction and Matching for Real Time People Re‐Identification,” Proc. ] , , , y g p , of Eurographics Italian Chapter Conference 2010 (EG‐IT 2010), 2010. Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Requirements for re‐identification re identification • • • • Object/People detection (bounding box) Obj t/P l d t ti (b di b ) Foreground detection (mask) Face/body‐part Face/body part detection (segmented regions) detection (segmented regions) Single‐camera tracking (temporal consistency and motion information) Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it A multi dimensional problem A multi‐dimensional • Camera positioning p g Same camera • • • Overl. cameras Disjoint j cameras Same camera: the system should be able to re‐detect the same person whenever he appears again in the same in the same camera. View camera View point and color and color correction problems can be neglected [Yan99] Overlapping cameras: geometrical information can be exploited; main p p p match are captured p at the veryy instant [[Cal08]] assumption: people to Disjoint cameras: much complex case [Far10] [Yan99] J. Yang, X. Zhu, R. Gross, J. Kominek, Y. Pan, and A. Waibel, [Yan99] J Yang X Zhu R Gross J Kominek Y Pan and A Waibel “Multimodal Multimodal people ID for a multimedia meeting browser, people ID for a multimedia meeting browser” International ACM Multimedia Conference, 1999, p. 159. [Cal08] S. Calderara, R. Cucchiara, and A. Prati, “Bayesian‐competitive consistent labeling for people surveillance.,” IEEE transactions on pattern analysis and machine intelligence, vol. 30, Feb. 2008, pp. 354‐60. [Far10] M. Farenzena, L. Bazzani, A. Perina, V. Murino, and M. Cristani, “Person re‐identification by symmetry‐driven accumulation of local features,” 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, IEEE, 2010, pp. 2360‐ fl lf t ” 2010 IEEE C t S i t C f C t Vi i d P tt R iti IEEE 2010 2360 2367. Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it A multi dimensional problem A multi‐dimensional • Single/multiple shots g p Single Shots • Single Shot methods associate pairs of images, each containing one instance of an individual. These methods are mostly similar to those proposed for image retrieval with some particular specialization to people. – • Multiple shots PROS: simple, fast. CONS: view dependent. Less stable with occlusions and noise Multiple shot: information coming from multiple frames (or images) containing the same person are used as training data. – PROS: more information gathered for the same person; CONS: alignment and increased data dimensionality Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it A multi dimensional problem A multi‐dimensional • Signature color • shape position texture soft‐biometry color – mean, histogram, Gaussian , g , model, Mixture , of Gaussians – color space RGB, rgb, HSV • shape: – width, height, h/w ratio, contour , g , / , • • • • Spatial feature position/trajectory: position in the image or in the ground plane texture: covariance matrix, SIFT/SURF texture: covariance matrix SIFT/SURF Soft‐biometry: face, gait Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it A multi dimensional problem A multi‐dimensional • The body model Th b d d l No body model 2D body model 3D body model • No body model • 2D body model 2D body model – Cylindrical model – LTH Leg torso head • 3D body model Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it No body model: search for similarity No body model: search Content based retrieval methods Global descriptors: Histograms , texture, Medioni’s circular hi t histograms, Mixture Mi t off gaussians, covariance i i matrix… ti SS. Calderara, R.Cucchiara, A. Prati Multimedia Surveillance: Content Calderara R Cucchiara A Prati Multimedia Surveillance: Content based Retrieval with Multicamera People Tracking Multicamera People Tracking Proc of workshop workshop VSSN at acm multimedia 2006 Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Cylindrical body model Cylindrical body • Cylindrical shape (or more generally as a solid of revolution) solid of revolution) – the horizontal variations of the people appearance are neglected, supposing that the color or texture – distribution along the vertical axis is the only important data important data. – [Bir05]: the person mask is divides into ten horizontal stripes and the mean color of each stripe is stored as representative feature. [ [Bir05] N.D. Bird, O. Masoud, N.P. Papanikolopoulos, and A. Isaacs, “Detection of Loitering Individuals in ] , , p p , , g Public Transportation Areas,” IEEE Transactions on Intelligent Transportation Systems, vol. 6, Jun. 2005, pp. 167‐177. Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Legs torso head model Legs‐torso‐head model • The reason of the legs‐torso‐head model, instead, is mainly due to the occidental traditional clothing. • The target silhouette is divided g into three horizontal parts, ideally corresponding to : – legs (and thus to the pants/skirt appearance) – torso (i.e., shirt or jacket) – head (i.e., hair). [[Lan03] M. Lantagne, M. Parizeau, and R. Bergevin, “VIP: Vision tool for comparing ] g , , g , p g Images of People,” Vision Interface, 2003. [Far10] M. Farenzena, L. Bazzani, A. Perina, V. Murino, and M. Cristani, “Person re‐ identification by symmetry‐driven accumulation of local features,” 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, IEEE, 2010, pp. 2360‐ 2367 2367. Fixed [Lan03] Estimated [Far10] Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it 3d body models (1) 3d body models • Panoramic Appearance Map: surface of a 3D cylindrical model [GAN06]. Signature g to compare [Gan06] T. Gandhi and M. Trivedi, “Panoramic Appearance Map (PAM) for Multi‐camera Based Person Re‐identification,” 2006 IEEE AVSS IEEE, 2006, pp. 78‐78. Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it 3D Body Models (2) 3D Body Models • 3D vertex model with features stored and related to each vertex [Bal10] [B l10] [ [Bal10] D. Baltieri, R. Vezzani, and R. Cucchiara, “3D Body Model Construction and Matching for Real Time ] , , , y g People Re‐Identification,” Proceedings of Eurographics Italian Chapter Conference 2010 (EG‐IT 2010), Genova, Italy: 2010. Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it A multi dimensional problem A multi‐dimensional • Spatial localization of features mapped local • • • Unmapped local Global Global features: global color histogram, shape descriptors [Orw99] U Unmapped d local l l features: features f t f t are computed t d on patches t h or blocks bl k but are unmapped to a body model or a relative position. E.g., Bag‐Of‐ Word with SIFT descriptors [Liu09] Mapped local features: features features: features are reffered are reffered to a human a human body model body model and and to specific regions [Lan03,Met10] [Orw99] J. Orwell, P. Remagnino, and G.A. Jones, “Multi‐camera colour tracking,” VS’99, pp. 14‐21. [Liu09] K. Liu and J. Yang, “Recognition of People Reoccurrences Using Bag‐Of‐Features Representation and Support Vector Machine,” Chinese Conference on Pattern Recognition, 2009, pp. 1‐5. [Lan03] M. Lantagne, M. Parizeau, and R. Bergevin, “VIP: Vision tool for comparing Images of People,” Vision Interface, 2003. [Met10]M. Metternich, M. Worring, and A. Smeulders, “Color Based Tracing in Real‐Life Surveillance Data,” Trans. on Data Hiding and Multimedia Security V, vol. 6010, 2010, pp. 18‐33. Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Multiple shots Single Shot color shape position texture soft-biometry Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Papers for Camera positioning and Adopted Feature 16 14 12 10 8 6 4 Disjoint 2 Close‐disjoint 0 Overlapping O l i Colour Shape Same Texture Trajectory P ii Position Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Example 3D body model 3D body model for ri‐identification ri identification • • • • • Camera positioning: disjoint p g j Body model: 3D vertex based body model about 600 vertices with scale factor Signature: local color histograms Requirements: calibration q Re‐identification is provided comparing i 3D models 3D d l or view‐specific i ifi projections of the model Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it ViSOR Re Identification Dataset ViSOR Re‐Identification • A new dataset designed for people re‐identification • • 50+ people : At least 4 snapshot for each person from different angles Position and orientation of each person w.r.t. the camera for correct 2d/3d alignment • Thanks to: A EU project to: A EU project in Prevention, Preparedness and Consequence Management in Prevention, Preparedness and Consequence Management of Terrorism and other Security‐related Risks Programme European Commission – DG: JLS http://www.openvisor.org Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Agenda • • • • • • • Introduction Video Surveillance and Forensics Video Surveillance and Forensics Design Aspects for surveillance CV & PR for people surveillance People shape detection People shape detection People behavior by trajectory analysis Conclusion Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it After people detection… recognition • Gait analysis • Posture analysis • Trajectory analysis • Action/interaction/activity analysis • Behavior analysis y • Motion in crowd • Anomaly detection.. M. S. Ryoo, J. K. Aggarwal Semantic Representation and Recognition of Continued and Recursive Human Activities Journal of Computer Vision 2009 Kaiqi Huang; Dacheng Tao; Yuan Yuan; Xuelong Li; Tieniu Tan; View‐ Yuan; Xuelong Li; Tieniu Tan; View Independent Behavior Analysis IEEE Trans SMC 2008 Cheriyadat, A.M.; Radke, R.J.; Cheriyadat, A.M.; Radke, R.J.; Detecting Dominant Motions in Dense Crowds, IEEE Journal of Selected Topics in Signal Processing 2008 Vijay Mahadevan, Weixin Li, Viral Bhalodia, Nuno VasconcelosAnomaly Detection in Vid Videos Using Mixtures of Dynamic U i Mi fD i Textures in Proc of CVPR 2010 Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it People trajectories in open space People trajectories in open space • Gi Given allll the h trajectories j i acquired i db by a video id surveillance ill system Which are the trajectories that share some specific location properties? Which are the trajectories that share some specific shape properties? Which are the most frequent Behaviors? Who did perform them? people l retrieval ti l Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Available datasets of trajectories Available datasets of trajectories Various time series (including trajectories): http://www.cis.temple.edu/~latecki/TestData/TS_Koegh/ http://www.cs.ucr.edu/~eamonn/time_series_data/ Character Trajectories Data Set: http://archive.ics.uci.edu/ml/datasets/Character+Trajectories Simone Calderara, Andrea Prati, Rita Cucchiara Body Part Tracking for Action Recognition J. Multimedia Intelligence and Secuirty 2010 Pen-Based Recognition of Handwritten Digits Data Set: http://archive.ics.uci.edu/ml/datasets/Pen-Based+Recognition+of+Handwritten+Digits Video surveillance ETISEO project: http://www-sop.inria.fr/orion/ETISEO/download.htm#video http://www sop.inria.fr/orion/ETISEO/download.htm#video_data data Soccer player trajectories: “T. D’Orazio, M.Leo, N. Mosca, P.Spagnolo, P.L.Mazzeo A Semi‐Automatic A Semi Automatic System for System for Ground Truth Ground Truth Generation of Generation of Soccer Video Sequences Soccer Video Sequences In the Proceeding of the 6th IEEE International Conference on Advanced Video and Signal Surveillance, Genoa, Italy Sep2‐4 2009” 1000 trajectory on soccer video ImageLab video surveillance dataset: More than 1000 trajectories of a video surveillance scenario Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Trajectory analysis in surveillance Trajectory analysis in surveillance • Trajectories are time series of data • Working on datasets of time series is a well studied data mining problem which requires: • A set of Features Feat res characterizing characteri ing trajectories • Similarity measure between two time series • A clustering technique to classify trajectories In video-surveillance video surveillance research • data availability is limited, discover a model of data • unprecise and noisy statistical methods • lack of reproducibility and high dinamicity adaptive methods for classification and clustering Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Literature on Trajectory analysis Literature on Trajectory analysis • Literature approaches on trajectory comparison can be classified: – Depending Depending on the Data Dimension (Complete vs on the Data Dimension (Complete vs Selected): Use all the temporal data or select a subset – Depending on the Representation (Original vs Transformed): Original feature space or a transformed f d) O i i l f f d space – Depending Depending on on the Feature (Point to Point vs the Feature (Point to Point vs Statistical): Statistical): Adopt a point‐to‐point comparison or exploit a statistical model for data representation – Depending on the similarity Measure B. Morris and M. Trivedi, “A survey of vision‐based trajectory learning and analysis for learning and analysis for surveillance,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 18, no. 8, pp. 1114–1127, Aug. 2008. Calderara S., Prati A. Cucchiara S Prati A Cucchiara R. R Mixtures of von Mises Distributions for People Trajectory Shape Analysis in press Trans. On Circuits and system for Video Technology 2010‐2011 Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Related Works Point to point Basharat08 UCF CVPR08 Hu06 Maybank PAMI06 Porikli04 CVPRWs04 Junejo04 UCF ICPR04 Bashir03 Shoenfeld ICIP03 Chen08 CVPR08 Shoenfeld Feature Statistical Representation Original Transformed x x Statistical Gaussian x x Statistical HMM x x HMM cross distance x x Hausdorf x Sampling x Piotto09 TMM09 x x Distance Gaussian x Ding08 VLD08 Shieh08 KDD08 Dimension Complete Selected PCA PCA Euclidean Null Space Projection Eigen decompositi on PCNSA(Pr. Comp Null Space analysis) distance x x SAX symbolic aggregate approximation Breakpoints LB_Keogh SAX symbol subspace symbol to symbol DTW distance Breakpoints quantization symbol to symbol Global Alignment distance Calderara09 ApproxWrapped MoAWLG x AVSS09 LinearGaussian Picciarelli09 x x Subsampling Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it TCMS09 GA KL‐divergence pdf distance SVM Learning References: (Basharat08) Basharat, A. Gritai, and M. Shah. Learning object motion patterns for anomaly detection and improved object detection. In Proc. of IEEE Int’l Conference on Computer Vision and Pattern Recognition, 2008 (Porikli04) F. Porikli and T. Haga. Event detection by eigenvector decomposition using object and frame features. In Proc. Of Computer Vision and Pattern Recognition (CVPR) Workshop,volume 7, pages 114–121, 2004. (Hu06)W. Hu, X. Xiao, Z. Fu, D. Xie, T. Tan, and S. Maybank. A system for learning statistical motion patterns. IEEE Trans. on PAMI, 28(9):1450– 1464, September 2006. (Junejo04) Junejo, O. Javed, and M. Shah, “Multi feature path modeling for video surveillance,” in Proc. of Int’l Conference on Pattern Recognition, vol. 2, Aug. 2004, pp. 716– 719. (Bashir03) F. I. Bashir, A. A. Khokhar, and D. Schonfeld, “Segmented trajectory based indexing and retrieval of video data,” in Proc. of IEEE Int’l Conference on Image Processing 2003 pp 623 626 Conference on Image Processing, 2003, pp. 623–626. (Chen08) X. Chen, D. Schonfeld, and A. Khokhar, “Robust null space representation and sampling for view invariant motion trajectory analysis,” in Proc. of IEEE Int’l Conference on Computer Vision and Pattern Recognition, 2008. (Ding08) H. Ding, G. Trajcevski, P. Scheuermann, X. Wang, and E. J. Keogh, “Querying and mining of time series data: experimental comparison of representations and distance measures,” Proceedings of the VLDB Endowment, vol. 1, no. 2, pp. 1542–1552, 2008. (Shieh08) Jin Shieh and Eamonn Keogh (2008). iSAX: Indexing and Mining Terabyte Sized Time Series. SIGKDD 2008. (Piotto09) N. Piotto, N. Conci, and F. De Natale. Syntactic matching of trajectories for ambient intelligence applications. IEEE Transactions on Multimedia, 11(7):1266–1275, l i di 11( ) 1266 12 Nov. 2009. 2009 (Calderara09)S. Calderara, A. Prati, and R. Cucchiara. Learning people trajectories using semi‐directional statistics. In Proceedings of IEEE International Conference on Advanced Video and Signal Based Surveillance (IEEE AVSS 2009), Genova, Italy, Sept. 2009. (Picciarelli08)Piciarelli, C.; Micheloni, C.; Foresti, G.L., "Trajectory‐Based (Picciarelli08)Piciarelli, C.; Micheloni, C.; Foresti, G.L., Trajectory Based Anomalous Event Detection, Anomalous Event Detection," Circuits and Systems for Video Circuits and Systems for Video Technology, IEEE Transactions on , vol.18, no.11, pp.1544‐1554, Nov. 2008 Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Ding Keogh 08 proposal (point to point) Ding‐Keogh 08 proposal (point to point) •The method proposed in (Ding‐Keogh 08) performs the comparison among time series in the original x‐y h i i i i h i i l data space. Tj xk , j , y k , j k 1...np •using dynamic programming and the Dynamic Time Warping • Inexact matching such as DTW are required to account g q for different lengths in time series and for temporal shifts . Ding, G. Trajcevski, P. Scheuermann, X. Wang, and E. J. h d Keogh, “Querying and mining of time series data: experimental comparison of representations and distance measures,” Proceedings of th VLDB E d the VLDB Endowment, vol. 1, no. 2, t l 1 2 pp. 1542–1552, 2008. Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it (Ding08) Point to point Complete Original (Ding08) Point‐to‐point Complete Original •DTW algorithm •The Method is effective when comparing similar sequences hence suitable when very large dataset is available Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Statistical models for trajectories • Instead of point‐to‐point comparison • create a statistical model of trajectory data To cope with noise To cope with the lack of large database To cope with the uncertainily of measure Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Trajectory shape analysis • • • Trajectory shape analysis for “abnormal behavior” recognition Trajectory Shape similarity; invariant to space shifts Not only space‐based or time‐based similarity Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it The approach of Imagelab The approach Statistical model: Mixture of representative pdfs TTrajectory j t t transformed f d as a sequence off Symbols S b l Corresponding to the most representatative pdf Trajectory alignement ( with global alignment) Similarity based on pdf similarity Clustering (k‐ medoids), classification detection anomalies.. or similarities…. Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it 1) Gaussian Model for spatial analysis x nj , yn j S Sequence off 2D spatial ti l coordinates di t Tj x2 , y2 x1 , y 1 x 1, j , y1, j , x 2 , j , y 2 , j , , x n j , j , y n j , j Advantages d a tages o of us using g spat spatial a coo coordinates: d ates Natural representation •Embodies additional information about velocity and acceleration • Some S paths th are more common then th other th depending d di on their th i position on the scene •Represent partially the reaction of people to the structure of the scenario Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Gaussian Model for spatial analysis Gaussian Model for spatial analysis • Due Due to the uncertainties on the measure of points to the uncertainties on the measure of points coordinates • Gaussian model to model every point location The simplest way: Bivariate Gaussian Centered on point coordinate having fixed variance. N i , k N ( x, y | i , k , ) Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Clustering Trajectories Clustering Trajectories Positional Gaussian Clustering • • • • • S. Calderara, A. Prati, R. Cucchiara, "Trajectory analysis in Video surveillance ill f multimedia for l i di forensic" in Proc of 1st ACM Workshop on Multimedia in Forensics (MiFOR 2009), Bejing, Chi 2009 China, 2009 Frequent and anomalous behaviors can be obtained by clustering trajectories by clustering trajectories according to positions and detect the most frequent activity zones (Gaussian model) Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Trajectory Shape Analysis by angles x nj , yn j S Sequence off 2D spatial ti l coordinates di t Tj x 1, j , y1, j , x 2 , j , y 2 , j , , x n j , j , y n j , j Sequence of 1D angles T j 1, j , 2 , j , , n j , j x2 , y2 Advantages of using angles: x1 , y 1 • more compact representation • invariant to spatial translations (both i 1 i local and global), thus describing trajectory shape Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Imagelab Proposal 1. Trajectory description with angle sequence T j 1,1 j , 2 , j , , n j , j 2. Statistical representation with a Mixture of Von Mises Distributions (MovM) Von Mises 1 V ( | 0 , m) e m cos( 0 ) 2 I 0 (m) I0 m 1 2 2 e m cos d 0 3. Coding with a sequence of selected vM pdf identifiers 4. Code Alignment g 5. Clustering with k with k‐medoids medoids A. Prati, S. Calderara, R. Cucchiara, "Using Circular Statistics for Trajectory Analysis" in Proceedings of CVPR 2008 Definition of EM algorithm for MovM Using Dynamic programming Definition f off Bhattacharyya distance fon vM and on-line EM Training Training set and on‐line classification set and on line classification MovM(T ) <S={S ..S },MovM(T )> T j 1, j , 2 , j , , n j , j j EM for MoVM 1j Coding with MAP njj j Alignement Clustering with Br distance Trajectory repository Trajectory clusters repository Normal/ abnorma l Surveillance system T j 1, j , 2 , j , , n j , j On-line EM for MoVM Classification with Br distance Coding with MAP Alignement Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Mixture of von Mises von Mises and Mixture and Mixture of Gaussians • MovM: MoG: K p ( x ) k x | μ k , Σ k K p ( ) kV | 0,k , mk k 1 k 1 14 1.4 1.2 m 1 0.8 0 2 0.6 1.2 9 0 5 0 1 m 1 1 0.3 0.5 0.6 1 0.4 0.2 0 0.5 0.8 m 1 0.4 9 5 0.3 03 2 0.2 0 1 2 1 0.2 3 2 0.5 4 5 3 0.3 6 7 0 0 1 2 1 0.2 3 4 2 0.5 Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it 5 6 3 0.3 7 Inexact matching • Since the symbols we are comparing correspond to pdf, match/mismatch should be proportional to the distance between the match/mismatch should be proportional to the distance between the two corresponding pdfs • Need to evaluate distance between two pdfs: Angular: Von Mises Distributions V ( | 0,a , ma ) V ( | 0,b , mb ) • Bhattacharyya distance bw pdfs (closed form)[Cal08] 1 d B 1 I0 I m I m ( ) ( ) 0 a 0 b ma2 mb2 2ma mb cos ( 0,a 0,b ) Spatial: Gaussians Distributions N ( x, y | a ,k , a ) N ( x, y | b ,m , b ) • Bhattacharyya distance bw pdfs ( ) a b 1 d B ( a b )T 1 ( a b ) 8 Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Comparison between VS and VLDB approaches Comparison between VS and VLDB approaches • Results on real dataset Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Experimental comparison • Clustering accuracy was measured using the same K‐medoids Clustering accuracy was measured using the same K medoids based clustering on based clustering on distance matrices computed with the different methods described Test ID Number of Trajectories j (Ding08) (Piotto09) Our Approach pp T1 140 78% 73% 95% T2 108 80% 87% 99% T3 145 94% 86% 96% T4 100 90% 80% 97% Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Available data set Available data set VISOR : Video Surveillance Online repository http://Imagelab.ing.unimore.it/visor http://www.openvisor.org Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Outdoor multicamera Outdoor multicamera Synchronized views Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Applications • The proposed system can be used for trajectory retrieval in forensic investigation: • Query by shape (a) • Location Filtering (b) • S Snapshot h t and d trajectory t j t retrieval ti l (c) ( ) Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Working on trajectory on trajectory at ImageLab at ImageLab • People trajectory analysis for: – Fetch Video Data from Raw and Annotated Video File – Find anomalies in people path – Compare trajectory in the dataset for retrieving similar elements: Compare trajectory in the dataset for retrieving similar elements: • Shape • Location • Clustering – Compare people appearances to retrieve similar elements – View Graphically Query Results View Graphically Query Results – View Video sequences associated to: • Trajectories • Snapshots Calderara S., Prati A. Cucchiara R. Mixtures of von Mises Distributions for People Trajectory Shape Analysis in press Trans. On Circuits and system for Video Technology 2010‐2011 Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Data Fetchingg and Main UI M. Aravecchia, S. Calderara, S. Chiossi and R. Cucchiara A Video Surveillance Data Browsing Software Architecture for Forensics: from Trajectories Similarities to Video Fragments. MIFOR 2010 at ACM Multimedia 2010 Query by Example (Shape) Query by Example (Location) Query by Drawing (Shape) Clustering (by Shape or Location) Query by Appearances Video Segment Retrieval Query Optimizer • Inside the query engine a query optimization module is designed • Query Q O ti i uses alternative feature Optimizer lt ti f t similarity i il it measures if provided. • Rule based optimization: – When providing the alternative technique also a rule can be provided to trigger the alternative procedure. • Index Based optimization: – If clusters of the data are available the optimizer limits the comparison to the cluster centers. Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Performances and Results The query optimizer automatically chooses the best performing query strategy according to th dataset the d t t cardinality and the user choices of investigation. Achieved Goals A Three Layer Architecture for video surveillance data browsing for the forensic analysis, designed for: •To be flexible with the possible addition of new feature models (Feature Model) •To be easy to use with a graphical user interface with presentation models that are feature specific p ((Presentation Layer) y ) •To allow to filter people trajectories to obtain few interesting samples and the related video sequences (Query Engine) •To exploit different l i diff similarity i il i measure to obtain b i a trade d off between ff b query time and accuracy (Query Optimizer Module) An extension: Action trajectories An extension: Action • space‐time trajectory (STT) Simone Calderara, Andrea Prati, Rita Cucchiara Body Part Tracking for Action Recognition J. Multimedia Intelligence and Secuirty 2010 Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Agenda • • • • • • • Introduction Video Surveillance and Forensics Video Surveillance and Forensics Design Aspects for surveillance CV & PR for people surveillance People shape detection People shape detection People behavior by trajectory analysis Conclusion Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Some Conclusion (1) Some Conclusion • SSmart Video surveillance, video analytics Vid ill id l i , multimedia l i di forensics are not only a research game anymore. it ‘s time to knowledge g transfer to companies p • Is the 80% done? Or is another effect of Pareto’s Principle 80‐20 ? • Processing Terabyte of videos is now straightforward. We b f d hf d need ( but we have) data. • Real‐time processing is not a chimera Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Some Conclusion (2) Some Conclusion The Future of Multimedia Surveillance&Forensics Architecture • Architecture for non IT experts. • With software solutions ft l ti f for: – Combining different analysis (real‐time knowledge extraction and data mining ) and data mining – Allow 3D‐4D world reconstruction – Presents data in innovative,intuitive and interactive , way (touch, y( , mobile..) – Allow traceability of operation (for legaly issues) – Deal with privacy • Focus on performance when analysing very large databases of data Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Some Conclusion (3) Some Conclusion • There Th i so many work to is k t do.. d Working on crowd Working on challanging partial crowd Working on moving sensors Working on 3D Working on multisensor, multimedia Working g on forecasting f g and pro‐active p bheavior analysis p before f 2020? • … will we solve the problems • • • • • • Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it Thanks. Thanks ..And thanks to Imagelab For any details http://Imagelab.ing.unimore.it Andrea Prati, Roberto Vezzani, A d P ti R b t V i Costantino Grana, Simone Calderara, Giovanni Gualdi, Paolo Piccinini, Paolo Santinelli, Daniele Borghesani, Davide Santinelli, Daniele Borghesani, Davide Baltieri, Sara Chiossi, Adnan Rashid, Michele Fornaciari, Manuel Aravecchia, Rudy Melli, Emanuele P i i Gi li Perini, Giuliano Pistoni.. Pi i Imagelab – University of Modena and Reggio Emilia – http://imagelab.ing.unimore.it