Sound Morphing using the FFT
The purpose of this project is to expose Digital Culture students to several practical
aspects of DSP algorithm implementation. In particular, this project provides hands-on
programming experiences with sound processing functions, signal framing, windowing, FFT
processing, analysis and reconstruction, and subjective and objective evaluation of sound
morphing. We have gone through the project and software in our class sessions. We want the all
the EEE407/591 students to type up the code and run it. Below we have information on the
operation of the algorithm.
Sound morphing is often used in entertainment applications such as movies, video games,
electronic music synthesis, etc. It is accomplished by impressing characteristics of one sound
onto another. This can be done either in the time domain using modulation techniques or in the
frequency domain using FFT-based operations. When two audio signals are represented in the
frequency domain, there are many different opportunities to manipulate them in order to create
“hybridized” sounds. One interesting possibility involves transforming one sound into another.
In this project, we will attempt to operate on the spectral content of two signals using the shorttime Fourier Transform (STFT). Figure 1 shows a basic block diagram of a sound morphing
scheme.
Figure 1. Sound Morphing using the FFT.
-1-
As can be seen, the signal is first divided into frames using windows. This is
accomplished by a “running” or sliding window as shown in Figure 2. In MATLAB this can be
accomplished by taking the signal vector that contains the wave file and dividing it into smaller
N-point vectors that are processed one at a time.
Figure 2. Segmentation using a sliding rectangular window.
The sliding N-point window divides the signal into frames. If the window is shifted N
points at a time we have disjoint frames. When the signal is shifted by overlapping windows
then we have frames that have common content. Overlapping tapered windows are very
common in signal processing of speech and audio signals. One common signal framing scheme
that involves triangular windows is shown below in Figure 3.
past frame current frame next frame
Figure 3. Signal framing using overlapping triangular windows.
In Figure 1, sound morphing is accomplished by first forming the N-point signal frame.
The frame may be tapered by multiplying it (sample-by-sample) with a tapered window such as a
triangular or a Hamming window. If no tapering is used then the signal frame will have spectral
properties associated with the rectangular window. After signal frames are formed, the FFT is
taken – one frame at a time. This combination of a sliding window with an FFT forms a running
spectrum of the signal and the overall process is formally called short-time Fourier transform
(STFT). The STFT is both a function of time and frequency, X(k,m), where k is the frequency
-2-
index and m is the frame index. An STFT taken using the DFT and a window w(n) is
represented by the following equation.
X (k , m)
N 1
w(n) x(mN
n) e
j 2 kn
n 0
where m is the frame index, k is the frequency index The operation above is taken using the FFT
as follows
signal
Process next
window
Other DSP
Operations
FFT
Y
next
frame?
N
Stop
Figure 4. Frame-by-frame FFT processing.
For this exercise, initially, we will form N-point frames with windows. Then, the FFT
will be taken to obtain the spectrum. Once this is obtained, then the complex frequency vector is
broken up into magnitude and phase spectra. Note that for sound 2 only the magnitude spectrum
is obtained and the phase is discarded. The next step for sound 2 is the spectral fitting, and that
involves interpolating the magnitude spectrum to the size of the magnitude spectrum for sound 1
if it is larger. This will happen in the case where one sound is longer than the other (it will not
be necessary here). Next normalization of the magnitude spectrum is performed by dividing X(k)
by a global average magnitude. Before the spectra are multiplied, they are compressed according
to a compression envelope that also has the effect of controlling their relative contribution. The
final step is to take the inverse FFT using the computed hybridized magnitude spectrum and the
phase spectrum from sound 1 to get the output frame.
Another way to combine two sounds is a process similar to sound morphing called
vocoding. In vocoding, the magnitude spectrum of one sound is scaled by the magnitude
spectrum of another, thereby creating a sound that has characteristics from both sounds. In
Figure 4 we give a diagram which can be seen as a simplification of the diagram in Figure 1.
-3-
Figure 5. Simplified sound morphing system.
As in morphing, sound 1 is FFTed and the complex FFT components are
separated into magnitude and phase spectra. Sound 2 is transformed and only the magnitude
spectrum is calculated and retained. The phase of sound 2 is discarded. Sound 1 is often
referred to as the ‘carrier’ signal and sound 2 is referred to as the ‘modulating’ signal, or ‘voice’
signal. This is because the ‘voice’ is in essence impressed or coded into the ‘carrier’ signal,
hence the name vocoder1 which is short for voice-coder. We provide all the software code
which you will need to run and obtain results.
Use the attached program and the sound files to implement sound morphing.
1
Vocoder is somewhat of an unused terminology here. Vocoder is a term usually used in cellular phone
technologies and refers to the model and associated speech compression algorithm that is used to reduce the bit rate.
-4-
DELIVERABLES:
You need to prepare a typed, printed report (about 2 pages) and turn in on Tuesday June 23rd in
class. Your report must include all the items listed below, graphs, tables, and textual
explanations. Use figure captions and refer to the figure number. We will need this uploaded on
the Blackboard.
The typed report must have the following sections
1. Introduction
The introduction must contain a summary of the project. It should cite the application
(find sites from the web and research or application papers that appeared in the literature
and cite them). For example, you may find a cool web site that has an innovative
morphing algorithm. Also find when and where these types of algorithms have first
appeared and cite the paper (it may be IEEE or Audio engineering or some audio
magazines). Also cite here any commercial product that maybe available for sound
morphing.
2. Algorithm description
The algorithm must be described in a way that it shows that you understood its major
components. Why do we divide the record in signal frames? Try to explain why is the
phase of sound 2 discarded? Why is the processing done only with the magnitude and
not with the phase? Does the magnitude contain more information than the phase?
3. Simulations and Results
In this section you need to run the software and give results that demonstrate the
performance of the algorithm. Use MATLAB’s spectrogram function and compute the
spectrograms (we covered the spectrogram command in previous homework – revisit
those examples) of all the files and then the spectrogram of the resultant sound. Also
you need to describe qualitatively the performance of the algorithm. Address the
following questions:
Question 1: What are the similarities between the output spectrum with respect to the
carrier and modulator spectrums?
Question 2: Which sounds work best for the vocoding effect? Try testing and comparing
using the two carrier signals ‘guitar_chugg.wav’ and ‘guitar_chordriff.wav’ as the input
sounds. Can you identify distinctly any of the two sounds at the output? Can you hear a
guitar?
4. Concluding Remarks
The following points should be addressed:
-5-
First a summary of what you have learned? How did the project help you digest some
practical aspects of DSP and FFT algorithms? What would you do to improve this project?
Answer the following questions (a paragraph each):
– What did you learn about framing and processing signals?
– What would you do to improve the performance of the algorithm?
5. References
Include references and web sites that provided info. Use IEEE style.
6. Appendix: Listing of the MATLAB software.
PROJECT REPORT DUE:
06/23/2015 in class (submissions after the deadline will be penalized 25%)
Please bring a hard copy of your report.
Upload a zip file to the Bb, which contains the report and m files.
We will not accept late reports after 06/26/2015.
Type carefully the code below and run it. Make sure you have the wav files in the working
directory. Please do not leave the work for the last minute.
-6-
-7-
-8-
Place this order with us and get 18% discount now! to earn your discount enter this code: special18 If you need assistance chat with us now by clicking the live chat button.