1

Digitization:
Image:
Image representation:
­­­ visual perception: The distinction between Human eye and photo camera 成像 imagation formation: Photo camera: * Lens has fixed local lenth.
* Adjust the focal length by varying the distance between the lens and the imaging plane. Human eye:
* Distance between the lens and retina is fixed.
* Focal length for proper focus obatained by varing the shape of the lense.

Question:
All the center squares hv exactly the same intensity. However, they appear to the eye to become progressively darker as the background becomes lighter. Why? Other example? Answer:
The human visual system tends to undershoot or overshoot around the boundary of regions of different intensities. e.g. Although the intensity of the stripes is constant, we actually perceive a brightness pattern that is strongly scalloped, especially near the boundaries. Therefore, perceived intensity is different from the actually intensity. e.g: a piece of paper that seems to be white when lying on a desk, but can appear totally black when used to shield the eyes while looking directly at a bright sky.

2

­­­ Matrix and Pixel:
● Matrix is the result of sampling quantization and
,
which is a matrix of real numbers. ● Each element of the matrix array is called an image element, picture element, pixel, or pel. Image representation:
● An image is referred to as a 2D light intensity function f(x,y) where 1. * (x,y) denotes the spatial coordinate(空间坐标), 2. *and f is a function of (x,y) and is propotional to the brightness or grey level of the image at that point (one pixel). f是把灰度级值赋予每个特定坐标（x,y）的函数。 ­­­ K­bit level:
The image digitization process requires decision abt values for
● M(rows),
● N(columns), and for the number,
● L, of discrete gray levels allowed for each pixel,

L = 2^K (2的整数幂) Dynamic range:
● The range of values spanned by the gray scale（灰度级的区间取值范围）. A image whose gray levels span a significant portion of the gray scale as having a high dynamic range (high contrast)
.（把占有灰度级全部有效段的图像叫做 高动态范围图像）
(large standard deviation,灰度Level跨度很大)

The number, b, of bits(位数) required to sotre a digitized image is b=M*N*k. When an image can hv 2^k gray levels>> k­bit image.
e.g 当一个图像有256个gray level时,我们称此图像为8­bit图（2^8=256） A general­purpose image processing system:
1. A sensor is sensitive to the energy radiated(辐射) by the object we wish to image. It produces an electrical ouput propotional (continuous voltage) to light intensity. 2. A digitizer is a device for converting the output of the physical sensing device into digital form. Digital image acquisition (image sampling):

3

(从传感器感知的数据中产生数字图像,需要把连续的感知数据转换为数字形式：取样和量化):

Digital image acquisation Process:
a) Energy (“illumination”) source
b) An element of a scene
c) Imaging system
d) Projection of the scene onto the image plane
e) Digitized image

3 ways of Image acquisition:
1) By Single sensor
2) By Sensor Strips (Linear Sensor strip & circular sensor strip)
3) By Sensor Arrays How to generate digital images from sensed data???：Image Sampling and Quantization

Sampling: Digitizing the coordinate values 数字化坐标值­­­ the principle factor determining the spatial resolution
.

Sampling rate: the number of samples (both spatial directions) takin per unit distance. (单位距
离的取样数目)

Spatial resolution: the smallest discernible detail in an image. can express as the number of pixels. the smallest change in spatial that can be recognized. Gray level resolution: the smallest discernible change in gray level.

4

Under sample: if under sampled, then a phenomenon called aliasing corrupts the sampled image. Aliasing: an effect that causes different signals to become indistinguishable Quantization: Digitizating the amplitute values 数字化幅度值 The quality of the image is determined to a large degree by the number of samples and discrete gray levels used in sampling and quantization. The result of sampling and quantization is a matrix of real numbers.

Video:
2 ways of generating video:
1) use a video camera to caputre a sequence of frames recording actual motion as it is occurring in the real world. 2)animition, create each frame individually. Video representation:
Project a 3D scene into a 2D plane. (3D: depth, texture, illumination, 2D: texture, illumination) Steps of digitization of video:
1) Signal acquisition and amplification(增强信号)
2) Spatial sampling & temporal sampling & quantization

5

spatial:
Spatial sampling:
* taking mearsurements of the underlying analog signal (模拟信号而不是数字信号,数字信号是
0,1) at a finite set of sampling points in a finite viewing area (frame) depending on a grid. Sampling resolution: number of sampling points. More Sampling resolution >> Higher visual quality of the video image >> require higher storage capacity. Raster Scanning:
● Transform the two­dimensional set of sampling points into a one­dimensional set.
● Two ways to achieve raster sacanning: Progressive & interlaced. 1. Progressive Scanning:
* The sampling points are scanned from left to right, top to bottom. * Typically used for film and computer displays. 2. Interlaced Scanning: * The sampling points are divided into even and odd scan lines. The odd lines are scanned first from left to right, and top to bottom, then the even lines are scanned.
* Commonly used for televison signals. temporal:
Temporal sampling:
* A moving video image is formed by sampling the video signal temporally, taking a rectangular “snapshot” of the signal at periodic time intervals. Frame rate:
16 frames/sec >>> illusion of motion
24 frames/sec >>> motion picture technology

Audio:
Audio reprsentation: wave form:

6

The vibration of materials triggers pressure气压 wave fluctuation in the air around the materials. The pressure waves propagate in the air. The pattern of the oscillation is called wave form. The amplitute 振幅 of sound: A measuring unit used to deviate the pressure wave from its mean value. The size of a sound pressure level (SPL) is measured in decibels(dB).
(human perceive subjectively as Loundness or Volume.) frequency: the number of pressure wava fluctuation per second. Unit hertz(Hz) or cycles per second(cps). (The general human accepted audible frequencies is 20 to 20,000 Hz.) spectrum:
Spectrum analysis allows you to examine sound by representing the sound in the frequency domain (amplitude vs. frequency), instead of time domain (amplitute vs. time)
* use FFT (fast fourier transform) to perfrom Spectrum analysis. Digitizing Sound:
2 steps:
1. Sampling: measure the signal’s value at discrete intervals;
(一段时间内,只取一个样本代表这段时间的data;)

2. Quantization: restric the value to a fixed set of level.
* use Analogue to digital converters (ADCs).

Audio sampling rate:
For high­fidelity (高保真): A sampling rate must be chosen that will preserve at least the full range of audible frequencies.

7

Recording sound:
To compromise the record, having the sampling rate is better than having the sampling size. Audio File Size: sample size rate r * sample size s /8 = data size each second of digitized sound (bytes)

Color: Different color models for image: RGB: 24­bit color cube
Used in images for multimedia.
● The different colors in this model are points on or inside the cube.
● Defined by vectors extending from the orgin.
● 8­bit hold 256 different colors. Therefore an RGB color can be represent in 3 bytes, or
24­bits. (每一副红,绿,蓝图像都是一副8­bit图像,每一副图像有2^8=256种颜色,因此每一
个RGB(R,G,B)图像中的颜色总数为2^8^3.)

● subset of colors: safe RGB colors: 216 colors
● FFFFFF:white, 000000:black CMY & CMYK:

8

Better for color printing.
: C = 1­R M = 1­G Y = 1­B
● Color Printing requires to input the CMY data or tranform the RBG data into CMY.
● Forming color by subtr
● ction of light, instead of addition: provides a color model appropriate to ink and paint.
● Cyan, magenata & yellow: subtractive primaries.
● CMYK: blackK. HSV:
Corresponds colsely with the way humans describe and interpret color.
(RGB, CMY, CMYK is suitable for harware)
Hue: basic color
Saturation: 纯度
Value: 亮度 the brightness of the color, Different video color system: YIQ: Y: Brightness, I: flesh tones(肉色), Q: Other colors YUV: Y: Brightness, YUV linearly related to the RGB components YDrDb: CCIR­601 Y:Cr:Cb
4:4:4
For every 2*2Y samples 4 Cr and 4 Cb sample (No subsampling)
4:2:2,
For every 2*2Y samples 2 Cr and 2 Cb sample (Subsampling by 2:1 horizontally only)
4:1:1,
For every 2*2Y sample 1 Cr and 1 Cb sample (Subsampling by 4:1 horizontally only)
4:2:0.
For every 2*2Y sample 1 Cr and 1Cb sample (Subsampling by 2:1 horizaontally and vertically) Human Visual & Hearing System:
Features of the human visual system: refer to the pg.1
* more sensitive to luminance
* more sensitive to high contrast
* more sensitive to image features that persist for a long duration

9

(Use these features to design a digital video system which produces pleaseant visual images for viewer.)

Psycho­acoustical model:
Scientific study of sound perception. ● Threshold of hearing：
The minimum level at which a sound can be heard. It varies non­linearly with frequency. e.g a very low/high­frequency sound must be much louder than a mid­range tone to be heard. ● Compression: It is meaningless to retain the sound that below the threshhold. ● Equal loudness curves:
Represent: sound pressure level (dB), over the range of audible 听得见的 frequencies.
The value get is : a constant loudness when presented with pure steady tones.

Image Processing:
Spartial domain and frequency domain processing techniques:
Image processing approaches fall into 2 broad categories:
1. Spatial domain methods / techniques: Direct manipulation of pixels in an image. ● g(x,y) = T[f(x,y)], here f(x,y) is a gray­level: the gray level of f(x,y) and g(x,y) at any point (x,y).
● It computes a pixel’s new value solely on the basis of its old value.
● Image enhancement: boundaries, edges, and contrast
● Gray Level Transformations:

10

Linear Gray­level transformation:
1. Image Negatives:图像反转
● useful in displaying medical images and film processing
● suited for enhancing white or gray detail emebedded in dark regiions.
● Reverse the order of pixel intensities (gray level)
2. Log transformation:
● 将图像的低灰度值部分扩展,将高灰度值部分压缩,以达到强调图像低灰度部分的
目的。(0 是黑,255白)
● allow to expand the values of dark pixels in an image while compressing the higher­level values.
3. Power­Law Transformation:
● As the gamma increases, the image will have a higher contrast. Non­linear Gray­level transformation:
1. Contrast stretching: Increase the dynamic range of the gray levels. under such circumanstence: poor illumination, lack of dynamic range in the imaging sensor, wrong setting of lens aperture during image acquisition. (narrow dynamic range: 亮的达不到最
亮。。暗的达不到最暗)

2. Histogram processing ­ hist.equalization:
● A histogram provides a view of the intensity profile of an image in bar char.

11

●
The Pixel intensities are plotted along the horizaontal x­axis while the number of occurences of each intensity are plotted along the vertical y­axis.
● An image ‘s pixels tend to occupy the entire range of possible gray levels and tend to be distributed uniformly, will have an high contrast.
● Histogram equalization: Streching its histogram to approximately uniformly distributed. Spatial Filtering:
Why use smmothing filters?(in contrast with the image sharpening)
● For blurrring and for noise reduction
● Blurring: used in pre­processing steps. Blends the smaller, less important object with the backgound, so that the larger, more important object easy to detect.
● Dis: may blurred the edge of the object
● Bridging of small gaps in lines or curves.
Linear smoothing filtering: Box filter, weighted average filter 1. Smoothing Filters ­­­ Averageing filters
Simply the average of the pixels contained in the neighborhood of the filter mask. box filter: the coefficient are equal. Weighted average filter: pixels are multiplied by different coefficient.

Non­Linear Smoothing filters: Median filter, max filter, min filter 1. Median Filter: replaces the value of a pixel by the median of the gray level in the neighborhood of that pixel. (The original value of the pixel is included in the computation of the median)
● Less blurring than linear smoothing filters of similar size.

Image Sharpening: Spatial differentiation
● The prinicipal object of sharpening is to
1. Highlight fine details in an image, featureless the background
2. Highlight gray­level discountinuities in an image
3. De­emphasized regions with slowly varing gray levels
4. enhance detail that has been blurred
● Edge­detection

2. Frequency domain methods / techniques: Modifying the fourier (or other) transform of an image.
● Step1 : D Fourier transform is to convert a signal from spatial domain to frequency domain. ● Inverse Fourier transform is to convert a spectral signal into spartial signal.

12

Geometric Transform: Modify the spartial relationship between pixels in an image, which are often called Rubber­sheet transformation.
1. image filtering: change range of image (intensity interpolation)
2. image warping: change domain of image (global warping) (tranlation, rotation…

Image Segmentation: divides an image into its constituent组成 regions.
Segments can be formed by:
1. Uniform image intensity
2. Particular texture pattern
3. Particular edge patterns
4. Perception knowledge
Segmentation should stop when the objects of interest in an application have been isolated分
离.

Threshhold segmentation: Inexpansive, fast and simpe.
● Input: gray image
● Output: binary image

13

● Steps:
1. Observe the histogram
2. Find the dynamic range for the object,
3. Transform the dynamic range for non important background to be 0.
● Unlikely to be effective: even in a simple image, there are likely to be various between objects and background.

Color image manipulation:
Different color models / channels:
RGB, CMY&CMYK, HSV. Function curves and color manupulation:
● The function curves are graphs that represent and control different attributes of an image, such as brightness or color.
● These attributes can be easily modified/ by manipulating the function curves/ without having to alter the image directly with a retouching tool. ● Color­correction operators can be either applied to all the channels of an image equally, or to individual channels in varying amt. ● Multiplication: Can result in a visual color shift when different amt are applied to different channels.

● Tone Correction: LUT curves
Can multify all the three channels by the set of three­user­define curves.

Whats the distinction between LUT and multiplication??? ­­­ LUT­manipulation provide extremely better amount of color control, even allow specific color correction across a narrow range of values.

14

● Tone Correction: Muldify the saturation.

e.g de­saturate anything that is blue, supersaturate a narrow range of green values. (saturate: 值为满的时候,纯色,值为0的时候,白色。hue控制颜色)

High Dynamic Range Imaging:
Advantage:
● More details of a scene
● Great potential to create much better photographs
● More dynamic range ● Steps of software based imaging: step1: take multiple images with diffrenent amt of exposure step2: recover the respons function of the imaging processing step3: construct high dynmaic range radiance map

Graphics & Animation
Concepts & Applications:
● Computer Graphics: Allows us to generate, modify and display still pictures, also underlies成为..的基础 the display of moving pictures and text.
● 2 different approaches:
○ Bitmapped image (位图): Must record the values of each pixels
○ Vector graphics: Stored as mathmatical descrpitons/ of a collection of individual lines, curves and shapes/ making up the image. Modeling: The main difference between 2D and 3D is depth. 2D drawings:
Even it appears to be 3D, if you want to change the viewpoint, you have to re­draw the object. 3D Modeling:
● Use of primitives to construct objects
● mesh reconstruction 曲面重构for surface simplification

15

● Complex objects, such as trees, can be constructed with automatic procedural modelling algorithms. Primitive parameters:
● Lines
● Polylines
● Polygons
● Vetex: 顶点
● Edge: 边缘
● Face: 面 3D modeling with primitive:
3D objects are made up for polygons, which are arranged by the computer into the form you desire. Low­poly Modeling:
(Useful in 3D real­time applications, game) Question: Why low­poly modeling?? Why dont we model with primitive? Answer: Although delicate(精致的) model can be obtained through detailed modeling with high number of polygons. BUT！Modeling with too many poly count will decrease the performance of the game even make it unplayable!! How to use Low­poly modeling:
● Building a low­poly version of an object, and using it as a stand­in during animation.
● low­poly stand­ins enable you to preview the animation in real­time, and test renders.
● Refine the animation, until it reaches your expectation, than you can replace it with the hi­res mesh. Level of Detail (LOD):
● Create multiple versions of the model, each with different poly count, or LOD!
● Swap out object with hi­res with the less­detail versions when the it is far from the camera. Procedural Modeling­­­ Fractal分形体:
(
Use to generate trees, mountains, clouds...
)
● Each image is formed from transformed copies of itself.

Particle质点 System:

16

Question: How to create a natural­looking form modelling,that is hard to generate with primitives? Answer:
Use Praticle system: A method which is use to create a complex model that is difficult to obatian with geometric 几何 (Modeling with primitives) modeling. How?:
This system allows for randomness, recursion, and accidents of shape like those typical of natural shapes.
● The system is based on employing simple shapes.
● These shapes have growth attributes (属性).
● The system will simulate模仿 these attributes, the shapes will grow in a specific track.
As the time past by, it will result in a 3D models. 3D Rendering:
It is a process that the computer interpretes: all the “object and light data”, and creates a finished image from the viewport(视角) you have selected. Represent the animated model with the aid of the a simulated camera. How it works?
● Normal: Polygon face has a normal, which represented by a little arrow sticking out of a face. And it indicates which direction its facing.
● The conputer considers: camera position, light position, intensity, colo. Caulculate, and paint the result image. Animation Control:
Animation is the creation of moving pictures one frame at a time.
24 drawings for each second of film. How to reduce the enormous amt of labor drawing???
● Cel animation:
The elements in a scene that migh move are drawn on a transparent material “Cel”, and laid over a background.
In producing the a sequence, only the moving elements on the “cel” need to be redrawn for each frame.
● Clay animation:
Use mini 3D sets, like stage sets, on which objects are moved carefully between shots(镜头). They made out of a malleable modelling material.

Animatronics：

17

The computer­controlled models that can be animated in real time. To bring the virtual character into to life: Control method:
● Explicitly declared control: Specify simple transformations for all events in an animation. Use: Key frame interpolation(更改) techniques: create sequence of still frames and insert key pose in the keyframe. ● Live action control: Examing the motions of objects in the real world, to animate the same movement by creating corresponding sequences ob object. ● Kinematics control:
Most useful in connection with jointed(关节) strutures. Like the various parts of the arm, can only move in certian ways, for 3D models, it has to obey the same kinematics constraints as a real arm.
● Forward kinematics: Determine the angles of joines
● Inverse kinematics: dealing with complex models with a large number of joints. ● Dynamic control: takins into account the physical laws that govern kinematics.
● Facial animation control:
Control facial expression of animation.

Object Recognition:
Background:
● Definition & Applications
Object Recognition:
● In computer vision, it refers to the task of finding and identifying objects in an image or video sequence. Applications:
1. Object detection
2. Object verification
3. Object classification
4. Human­computer interaction
5. Robert navigation
6. User interface
7. Location recognition Application: Automobile driver assistance­­­ Mobile eye, Smart eye.

18

●
1.
2.
3.
4.
5.
6.

Challenges:
Illumination variation
View point variation
Occlusion
Scale variation
Deformation
Complicated background

Components of object recognition system:
● Representation
1. How to represent an object category:
● Appearance only or location and appearance
● Invariances: View point, illlumination, Occlusion, Scale, Deformation,
● Use set of feartures or each pixel in image.
● Learning
1. How to form the classifier,given training data
○
Learn “ distinguishes them”, rather than manually specify the difference. what ○ 2 Methods of traning: 1. Generative 总的
2. Discriminative 分的 ○ Level of supervision: Mannual segmentation; bounding box; image labels;noisy labels; ○ Batch/increment
○ Tranining image:
1. Issue of over fitting
2. Negative images for iscriminative methods
○ Priors
● Classification How the classifier to be used on novel新奇的 data Object recoginition as pattern classification 1. Minimum­Distance­based classifiers: (Nearest neighbor Classifier)离哪个类最近就是哪
个类
2. Neural networks: 输入 features, 分类算法
3. Bayes classifiers: 推测一个物体是这个东西的可能性
4. Object recognition in practice:
General object recoginiton systems

Object recogonition: end to end （怎么训练,训练流程 step1: traning step2: tesing）

19

Video Processing:
Y: Caputures the brightness information, can be compatbly decode by a black­and­white television.
Chrominance (color) components: I and Q.
● Video standard *
CCIR­601: 4:2:2 A standard representation of video in terms of digital YCrCb(SECAM) color components. The luminance component is sampled at a rate of 13.5MHz with 720 active samples per line. Cr and Cb each are sample at 6.75MHz with 360 active samples per line. * SIF: Y, Cr, Cb. 4:2:0 image size 353 * 288 * CIF: * QCIF: A scaled down version of CIF 4:2:0

● Hierarchical structure
*
Sequence(Video): An unstructured data stream, consisting a sequence of scenes. * Scence: A collection of semantically (特征) related and temporally close shots, discrpting 描绘 and transfer a high­level concept or story.
特定时间特定地点内的一系列动作就一个场景，包括的属性就是其内容。

20

* Shot: An unbroken sequence of frames representing continous action in time and space, a physical entity which can be detected automatically.
强调拍摄，一系列的帧就是一个镜头，包括的属性有角度、轨迹、时间等等。

* Key frames: The frame representing the special content of the shot, depending on the content complexity of the shot, one or more key frames can be extracted from a single shot. Frame: A still image at a particular instant in time ● Video segmentation
Partition a video stream into a set of meaningful and manageable segments, which then serve as the basic unit for indexing.

Manual segmentation: Labor intensive, and time consuming. Semi­automatic / Automatic segmentation:
Split up a film into shots. Provide an efficient access to huge video archives. Camera vedio shot transitions (shot­cut detection): Abrupt (生硬的) and gradual
(平缓的)
Abrupt: occurs when the two individual shot are connect simply by pasted together. Gradual: connect two shots smoothly by applying special editing.(fade, dissolve) application: finding the positions in a video in that one scene is replaced by another one with different visual content.

Detecting method ­­­ differnce measure, e.g. pixel intensity difference. histogram difference. ● Video object segmentation
* Background modelling: “To model the background in a manner, so as to distinguish it from the rest of the images”. Can be used to identify moving objects. How can we do this???
1. ­­­ Image Subtraction
○ Video consist of a sequence of images
○ Assign the first frame to be the background.
○ Subtract intensities, pixel by pixel for all successive(连续的) images.

21

○ Non­zero differnces represent movement (everything else = background).
These entities can be thresholded and grouped. 2. ­­­ Median filtering preserves edges while removing noise

● otion Estimation ­ concept
M
ovement of objects and changes of camera settings during the capturing of
M
image sequences result in temporal content variations
.
* Objects ­­­ translation, rotation, shape variations * Camera ­­­ zooming, tilting, panning, traveling

Motion estimation is the estimation of the parameters of a video model that describes the temporal variations, usally from cosecutive frames. Application
1. Motion compensated prediction (video compresion)
2. Motion compensated interpolation (改写) (video stabilization)
3. Video image sequence analysis (computer vision)

Audio Processing:
● Major audio editing techniques
* Trimming/fade­ins and fade­outs…
○ Trimming: Remove the “dead air” or blank space from the start of a recording, and any
○
➢
➢
○
○
○
○

unnecessary extra time at the end of the recording. (This is the first sound editing task)
Fade­ins and Fade­outs:
Fade­ins: the volume of the selected section,gradually increase from silence at the beginning. Fade­outs: the volume of the selected section, gradually reduced to silence at its end.
Delay / Echo (重复): generates echo effects by repeating the sound within an interval.
Morphying: (变形) allow the selected section to transofrm into another by matching the related frequencies.
Time Correction: shorten or extend
Dynamics: changes the amplitude.

● MIDI vs. sampled audio MIDI: usical Instrument Digital Interface
M
MIDI standards, each word describing an action of a musical performace is assigned a specific binary.

22

1. a musician plays a key
2. the MIDI interface generates a MIDI message (defines the start of each note score配乐, and its intensity.)
3. this machine translate this message to connected system
4. the musician release the key
5. another message is created and tranlated… ● MIDI files requires less storage than the sampled audio files. And its size is independent of the playback quality (播放质量). 因为它是数字信号存储,而不是模
拟信号。
● MIDI files are generally 200 to 1000 times smaller than CD­quality sampled audio files.
● MIDI file can change the its length by manipulate the smallest detail of the composition (instead of change degrading the audio quality), this cannot be apply on digital audio.
● Dis:
● MIDI data does not represent sound, but musical instrucments, the playback can be accurate only if the MIDI playback device is identical to the device used for production. ● Adv of Sampled Audio:
The preparation and programming required for creating sampled audio do not demand a knowledge of music theory. ● Digital speech processing
* Speech synthesis 合成 ­ concept machine generation of speech
➢ Structure analysis
○ process the input text to determine where paragraphs, sentences and other structures start and end.

➢ Text pre­processing
○ analyze the input text for special constructs of the language.(e.g the abbreviations) ➢ Text­to­phoneme 音素 conversion
○ “
Phoneme is a basic unit of sound in a language.”
○ convert the word to phonemes.(different lauguage have different set of sound)

➢ Prosody韵律 analysis
○
Process the sentence structure, words and phonemes to determine appropriate prosody for the sentence.

23

○ includes: pitch(melody), timing (rythem), the pausing, speaking rate, emphasis on word...

➢ Wavaform production
○
The phonemes and prosody information are used to produce the audio
●

○
○
○
●

○
○

waveform for each sentence.
Advantage of Speech synthese:
Freedom of body movement Disable people Emotion detection
Problems

Interference from background noise Natural lauguage difficult for computer to interpret

● * Speech recognition ­ concept
○ Feature Extraction:
Convert the audio wave into a sequence of feature vectors
○ odeling:
M
acousitic modeling­­­ Analyze the human speech generated wave, describes the sounds that make up speech. (Frequency: pitch, amplitude: volumn)
○ Language modeling­­­ Describes the likelihood of various sequences of words being spoken. ○ Speaker­independent speech recognition: system do not use training.
○ Speaker­dependent: system analyze the person’s specific voice and use it to fine­tune the recogonition of that person’s speech.


Image Compression

●
Compression
➢ Background:
○ Enormous amt of multimedia information is stored, processed, and transmitted digitally over the internet
○ Compression to reduce storage requirement, reduce transmission time
○ Retain the amount of data required to represent digital media
○ The key of the reduction process is the removal of redundant多余的data
■ The aim of digital media compression is to represent media data with the smallest possible number of bits, for reducing the space required to store it, or reducing the bandwidth required to tranmit it.
➢ Redundancy
○ If the data that provide no relevant info, it is said that the data contain data redundancy. 24

○

○ Coding redundancy：
○ Each piece of information or event is assigned a sequence of code symbols, called a code word.
○ If the gray levels of an image are coded in a way that uses more code symbols than absolutely necessary to represent each grey level, the resulting image is said to contain coding redundancy.
■

■

25

■

○ Spatial Redundancy
■ The pixels in an image have statistical dependencies between each other, eapecially between neighboring pixels. Much of the visual contribution of a single pixel to an image is redundant, since it could have been guessed on the basis of the values of its neighbors. These inter­pixel dependencies are refer to spatial redundancy. ○
Psycho­visual Redundancy (Irreversible operation: visual information is lost )
■ Human visual perception: The brightness of a region perceived by the human eye, may not repond equal sensitive to all visual information. Pyscho­visual redundancy: the information iteself is not essential for human visual processing.

Phycho­visual Redundancy can be eliminated without large impact on the qulity of image perception.

26

● Lossless and lossy compression
○ Lossless compression:
Reduce only coding redundancy. Coding redundancy: minimize the average number of bits required to represent each pixel. 2 Method to achieve this: 1. Huffman coding: Assign the shortest possible code words to the most probable grey level.

27

2. LZW coding: Assigns the fixed­length code words to a variable length of symbol sequence. 28

➢ Source encoder & decoder models
➢ Reversible and irreversible operations ○ Lossy compression Lossless compression is the error­free compression. Lossy compression is based on the sacrifice the accuray of the reconstructed image, to increase the level of compression. as long as the distortion can be accepted, the compression can be achieve significantly.

ethods??? : Quantization, transoform coding (DCT, and Wavelet)
M
➢
Quantization:
The quantization method generally maps a broad range of input values to a limited number of output values. (Operate directly on the pixels of an image, which is spatial doman methods)

29

2^8 = 256, 2^4=16 ➢ Transform coding Compression techniques that are based on modifying the transform of an image. why? The human vision system has some specific limitations, which transform coding takes advantage of to achieve high rates of compression. human are much less likely to notice loss of very high­spatial­frequency components than lower­frequency components. (very deitaled image, many changes occur within a few pixels.) Reversible, linear transform: map the original image into a set of transform cofficients. DCT: discrete cosine transform (DCT).
○ JPEG: DCT based transform coding to compress bit­map images
(bitmapped image: record the values of each pixels
).
■ Designed for effcient compressing.
■ Good for photographs, artwork. Not good for simple cartoons, lettering, line drawings.
■ Designed to exploit limitations of the human eye: small color changes are perceived less accurately than small changes in brighteness. ■ Intended for compressing images that have a good visual result for human. 30

■ Degree of lossy compression can be varied by adjusting compression parameters. The image maker can trade off file size against output image quality.
Steps:
● Color space conversion:
Essential for True­color images.
RGB to YUV color space.
● DCT conversion (domain transformation) apply to each individual blocks independently.
● Quantization This is a lossy process. ○ JPEG2000: wavelet coding: coefficients of a transform that de­correlates the pixels of an image, can be coded more efficiently than the original pixels themselves. ● High compression efficiency
● Lossless color transformations
● Lossy and lossless coding in one algorithm JPEG VS JPEG2000
● JPEG 2000 is better.
JPEG VS GIF JPEG does better when images containing lots of color, GIF does better whith images containing few color(support up to 256 colors) JPEG is not suitable for use with iamges containing straight edges of contrast. JPEG is not good for images containing fine text.

Video Compression:
● Spatial compression/Intra­frame compression
○ Redundancy that exist within a single frame of a video.
○ Reducing spatial redundancy has been the focus of many digital image compression methods. It is directly applyed to the video frames.

● Temporal compression/Inter­frame compression
○ Exists between consecutive frames within a video sequence.
○ Since the frame rate of the video is often relatively high, while the camera parameters usually do not change rapidly between frames. Therefore, the

31

contents of consecutive frames are usually similar, which is temporal frame redundancy. ○ Frame differencing
Temporal compression methods exploit the temporal redundancies due to similarity between neighboring frames. The computer checks the difference between two video frames, if the pixels hv changed there apparently was something changing in the image (e.g moving.) Intra frame: a self­contained frame, that can be independently decoded without any reference to other images.
* the first image of a vedio sequence is always an I­frame. Can be used to implement fast­forward, rewind and other radom access functions. I­frames are inserted at regular intervals or on demand to insert by the viewer. Predictive frame: The prediction is made from an earlier picture, mainly I­frame, require less coding data than I­frame,. data needed to do this are:
Motion vectors. transofrm coeffcients describing predition correction. Involeves the use of motion compensation.

○ Motion compensation
■ Definition and procedure

32

● Motion compensation is used to improving the prediction, uses a model of the motion of objects between frames to form a prediction.
● Predict a frame, given the previous or future frames by accounting for motion of the objects in the video.
● Since much of the information that represents one frame will be the same as the inforamtion used in the next frame. Use motion compensation, the only information stored for the frames in between would be the information needed to transform the previous frame into the next frame. Audio Compression:
The complex and unpredictable nature of sound waveforms makes them difficult to compress using lossless methods.
● Predictor­based compression
○ PCM coding
■ Linear
● Differential (DPCM):If the sampling rate is very high, the difference between successive samples are likely to be small.
● Therefore: sotre the same audio signal using fewer bits per sample by sotring the difference between successive samples rather than the samples themselves. ■ non linear
● Human hearing:
○ Loud sound: insensitive to small errors
○ Quiet sound: sensitive to similar error
○ draw back of linear PCM coding: allows the same degree of error regardless of the amplitute(which decides the volume)

33

● therefore, use more bits for quiet sounds, which data loss is more audible.
■ A­law coding
● Europe
■ u­law coding
● Cannot be used directly because logarithms are time consuming to compute.
● North amercia, japan.

● Subband coding
○ Transform coding, breaks a signal into a number of different frequency bands, and encodes each one independently.(usaually the first step of compression) ● Perceptually­based compression
○ If an audio is digitized directely, the inaudible sounds data may also be included in the file.
○ a sound may be too quiet to be heard
○ may be covered by some other sound
○ when compressing sound, sound below the threshold of hearing should be eliminated. ■ Threshold of hearing: minimum level at which a sound can be heard. ■ Very low or very high frequency sound: much louder than a mid­range volumn to be heard. (refer to equal louness curve)
■ Use phycho acoustical听觉的 model
● Speech compression
○ silence encoding
■ The pauses during the speech, such as between words, phrases, between sentence, or when changing speakers.
○ Analysis engine
■ Produce a series of parameter. To decompress, we feed the parameters into the vocal tract model to recreate the sounds.(significant compression, while retain good quality.)

Digital photography:
● Exposure
➢ Aperture 光圈: f1, f1.4, f2.8…

34

○ Larger openings = more light
○ Moving from f16 to f8 is: two stops brighter (f8 is 4 times brighter than f16)
➢ Shutter: ○ a curtain in front of sensor, controls the exposure time, the length of time a curtian is open.
○ Blocks all light from exposing the file until you press the button. ○ Longer shutter speeds = more light
○ A half second exposure is ONE STOP(or 2 times) darker than a one second exposure. 1:2
○ Since f­stop and shutter are both measured in stops, keeping balance is easy. If you take away 2 stops from the aperture, you can give 2 stops back with the shutter and end up with the same exposure level.
Basic Exposure Mode:
● Mode P: Programe auto, shutter speed and aperture both selected by camera. ● Mode S: Shutter­priority auto
● Mode A: Aperture­priority auto
● Mode M: Manual ➢ ISO
○ Measure of a photographic film’s sensitivity to light.
➢ Sensor
● Lense:
○ Focal length:

35

■ ○ depth of field: if the exposure is made with a wide aperture (f2.8), then objects farther away from the subject are thrown farther out of focus. ■ Increase with f­numbers: low f­number will tend to have subjects at one distance in focus, with the rest of the image
(nearer and farther elements) out of focus. (But it is also depend on other parameters, such as focal length, subject distance) ● Focus:
○ Metering: Matrix metering, Center metering, spot metering
○ White Balance: white color is the reference color to distinguish the color.