Here are a few things I've done that might be
of interest.
-
Audio and Video
I built one of the first systems using speech
recognition to retrieve video content. This work earned the Best
Paper awards at SIGIR
and ACM
Multimedia. Here's a video
demonstration of the system (8 minutes
MPG, 80 MB). Years before most people were thinking
about it, I built a system
to
retrieve news video from a keyword search of the subtitles.
-
Music Retrieval
I've been inventing ways to search and retrieve musical audio for
possibly longer than anyone else. I built the first content-based
search engine specifically for music. I wrote a widely-cited review
paper about audio information retrieval. I've invented ways
of
retrieving music by rhythmic
similarity as well as long-term
structure, and one of my better inventions automatically
segments popular music using matrix factorization.
-
Fundamental Algorithms
I invented an information-theoretic model for
high-dimensional
data
distributions, allowing audio and video data to be classified and ranked
by
similarity for retrieval. I've released the code on
SourceForge as TreeQ.
I developed a fundamental approach to analysing audio,
video
and
text based on similarity matrices. I've used this for many
interesting applications. Many others
now use this approach.
I came up with the beat
spectrum, a new approach to characterizing rhythm in musical
audio.
-
Panoramic Video Maps
For those interested in GIS and mapping, Don Kimber
and I linked GPS
data with recorded panoramic video for a photorealistic
virtual travel system called FlyAbout.
This grew out of a multiple-camera panoramic
video system I developed.
-
Novel Interfaces
I've come up with some interesting ways of
interacting with video and audio. A few examples:
- A video
browser showing keyword relevance on a timeline.
- Video
Manga
is an automatically generated printable summary of video
keyframes.
- "Reach-Through-The-Screen"
makes video an active interface for remote control.
- iLight
lets
remote users draw on real-world objects through an active video image.
- Pan-and-Scan
turns an image collection into a video animation synchronized to
music.
- The AMV
system
automatically generates short music videos with synchronized music
soundtracks.
-
Goofy Stuff
For fun, I
make
things. I'm unafraid of doing stupid
tricks
with higher mathematics. Finally, DON'T
FEAR THE
ROBOTS. For
somewhat less professional stuff, I have delusions
of talent.
Oh yeah, I have a bunch of patents
and papers
(a
few of the more interesting
ones are here),
a
vanity
web
page, and if you care about my degrees and awards, you can
find
them in my bio.
Credit and thanks to co-authors and collaborators too numerous
to list.