{ "cells": [ { "cell_type": "markdown", "id": "8376dde5-688a-4d0f-af02-1e6e6b435651", "metadata": { "tags": [ "remove-nb-cell" ] }, "source": [ "# Lab 7 Primer\n", "\n", "[Slides](https://real-time-dsp.github.io/lab/lab7/primer.slides.html)\n", "\n", "With all of the tools previously developed, we are equipped to build a plethora of real-time systems with creative applications. In particular, most musical effects can be implemented with a relatively small computational load. This makes it feasible to run many different effects on cheap, reprogrammable systems.\n", "\n", "Here, we will focus on four types of effects." ] }, { "cell_type": "markdown", "id": "838192db-20e3-48ce-9221-a805e4a38010", "metadata": { "slideshow": { "slide_type": "slide" }, "tags": [ "remove-cell" ] }, "source": [ "# Audio effects" ] }, { "cell_type": "markdown", "id": "e5a1be4e-44d1-4ada-87f7-c778b6185cae", "metadata": { "slideshow": { "slide_type": "slide" }, "tags": [ "remove-cell" ] }, "source": [ "## Overview" ] }, { "cell_type": "markdown", "id": "cc7abe36-dd7a-43c2-9902-c9343f2810e0", "metadata": {}, "source": [ "* Comb filters\n", "* Harmonic Distortion\n", "* Amplitude Modulation\n", "* Vocoder" ] }, { "cell_type": "markdown", "id": "48c76203-17dc-4713-80e9-dc5dfb22d228", "metadata": { "tags": [ "remove-nb-cell" ] }, "source": [ "Many of the most well-known audio effects are variations or combinations of these basic systems." ] }, { "cell_type": "markdown", "id": "594f37d9-19ef-426c-9a08-40eb593922fc", "metadata": { "tags": [ "remove-nb-cell" ] }, "source": [ "## Comb filters\n", "\n", "The effect of adding signal with a delayed version of itself is often referred to as a \"comb filter.\" These filters can be further categorized as either feedforward (FIR) or feedback (IIR) comb filters." ] }, { "cell_type": "markdown", "id": "aa23db62-371b-48a1-af65-3c90ff7880cc", "metadata": { "slideshow": { "slide_type": "slide" }, "tags": [] }, "source": [ "### Feedforward" ] }, { "cell_type": "markdown", "id": "5e84cdd3-742c-457a-a4fd-30b9a005b5c2", "metadata": { "tags": [ "remove-nb-cell" ] }, "source": [ "Consider the effect of adding an input signal $x[n]$ with a scaled and delayed version of itself $\\alpha x[n-K]$\n", "\n", "![](../img/comb1.svg)" ] }, { "cell_type": "markdown", "id": "0aa3dd6e-7afa-4462-ba46-ed88ff932cc9", "metadata": { "tags": [ "remove-cell" ] }, "source": [ "

\n", "\n", "

" ] }, { "cell_type": "markdown", "id": "ace1a218-069d-4517-b684-b42c5a4c4b99", "metadata": { "tags": [ "remove-nb-cell" ] }, "source": [ "It is very easy to describe this system using a difference equation and to find the impulse response $h[n]$" ] }, { "cell_type": "markdown", "id": "e5173fe2-c1db-4166-a069-3c63f63984ab", "metadata": {}, "source": [ "$$y[n] = x[n] + \\alpha x[n-K]$$\n", "\n", "$$h[n] = \\delta [n] + \\alpha \\delta[n-K]$$" ] }, { "cell_type": "markdown", "id": "e75fe45f-2346-41b3-8247-56e710b1fb41", "metadata": { "tags": [ "remove-nb-cell" ] }, "source": [ "Equivalently, we could express this as a set of filter coefficients:\n", "\n", "```\n", ">> K = 5; b = [1, zeros(1,K), 1]\n", "\n", "b =\n", "\n", " 1 0 0 0 0 0 1\n", "```\n", "\n", "The magnitude response of this system has several nulls at equally spaced frequencies.\n", "\n", "```\n", "K = 39; b = [1, zeros(1,K), 1]; freqz(b,1,n,16000)\n", "```\n", "\n", "\n", "![](../img/comb1_freq.svg)" ] }, { "cell_type": "markdown", "id": "fa9f4271-11bf-49f6-8c23-0e4db1f1e8f3", "metadata": { "slideshow": { "slide_type": "slide" }, "tags": [ "remove-cell" ] }, "source": [ "

\n", "\n", "

" ] }, { "cell_type": "markdown", "id": "93ea6d07-8430-4a81-a34a-44f6a38c44bb", "metadata": { "tags": [ "remove-nb-cell" ] }, "source": [ "Recall that a notch filter can be created by placing zeros at an angle of the desired notch frequency. The feedforward comb filter can be thought of as cascading several notch filters, resulting in zeros at equally spaced angles around the unit circle:\n", "\n", "```\n", "zplane(h,1);\n", "```\n", "\n", "![](../img/comb1_zplane.svg)" ] }, { "cell_type": "markdown", "id": "46d20104-9f7c-4128-95b2-c89ab71c3be8", "metadata": { "slideshow": { "slide_type": "slide" }, "tags": [ "remove-cell" ] }, "source": [ "

\n", "\n", "

" ] }, { "cell_type": "markdown", "id": "ada9819a-0a75-4a2a-96d4-5dff554fe510", "metadata": { "tags": [ "remove-nb-cell" ] }, "source": [ "Although this example is a 40th order FIR filter, the computational complexity is low because nearly all of the filter coefficients are zero." ] }, { "cell_type": "markdown", "id": "23557411-a06e-43af-abd4-92a82772cd61", "metadata": { "slideshow": { "slide_type": "slide" }, "tags": [] }, "source": [ "### Feedback" ] }, { "cell_type": "markdown", "id": "04c2d030-ce1d-46ed-b40f-ad1dceaf8bd4", "metadata": { "tags": [ "remove-nb-cell" ] }, "source": [ "If an input signal $x[n]$ is instead added with a delayed version of the output, it results in an all-pole IIR filter:\n", "\n", "![](../img/comb2.svg)" ] }, { "cell_type": "markdown", "id": "36dd0244-b6c9-4e6d-a4b2-176c8be0bad3", "metadata": { "tags": [ "remove-cell" ] }, "source": [ "

\n", "\n", "

" ] }, { "cell_type": "markdown", "id": "866f77c2-1f3b-4bb4-a07a-95b414f56603", "metadata": {}, "source": [ "$$ y[n] = x[n] + \\alpha y[n-K] $$" ] }, { "cell_type": "markdown", "id": "d6b3daea-9b0d-4b8f-a9ae-0894d7843aac", "metadata": { "tags": [ "remove-nb-cell" ] }, "source": [ "```\n", "K = 39; a = [1, zeros(1,K), 1]; freqz(1,a,n,16000)\n", "```\n", "\n", "![](../img/comb2_freq.svg)" ] }, { "cell_type": "markdown", "id": "847b3a0c-7b08-4954-9bf3-1b3d66619757", "metadata": { "slideshow": { "slide_type": "slide" }, "tags": [ "remove-cell" ] }, "source": [ "

\n", "\n", "

" ] }, { "cell_type": "markdown", "id": "6332d91b-896e-47ea-8a47-7f4847aa3664", "metadata": { "tags": [ "remove-nb-cell" ] }, "source": [ "### Echo/Delay\n", "\n", "With sufficiently high delay $K$, the comb filter becomes an audible echo. If the IIR form is used, it will create multiple echos.\n", "\n", "### Allpass\n", "\n", "If the same delay $K$ is used for both a feedforward and a feedback term, the result in a perfectly flat magnitude response but a nonlinear phase response, These effects are often referred to as \"Allpass\" filters." ] }, { "cell_type": "markdown", "id": "a858808c-0c0f-40ac-8b04-ed9a765aa542", "metadata": { "slideshow": { "slide_type": "slide" }, "tags": [] }, "source": [ "### Flanger\n", "\n", "A flanger is a comb filter whose delay $K[n]$ slowly oscillates within some range $\\pm R$\n", "\n", "$$y[n] = x[n] + \\alpha x\\left[n-K[n]\\right]$$\n", "\n", "$$K[n] = R \\cos(\\omega_{\\text{flanger}}) + \\frac{R}{2}$$" ] }, { "cell_type": "markdown", "id": "b7dd452b-b73a-4800-8e2a-11dfde54e0bb", "metadata": { "tags": [ "remove-nb-cell" ] }, "source": [ "The time-varying nature of the delay characterizes the flanger. Typically, the overall delay is very short. Although the example above uses a cosine, any periodic function could be used.\n", "\n", "Since the delay must be an integer, $K[n]$ must be rounded.\n", "\n", "## Chorus\n", "\n", "Typically, a \"chorus\" effect consists of adding three or more flangers running in parallel each with it's own set of parameters. \n", "\n", "## Phaser\n", "\n", "The \"phaser\" effect can be implemented by slowly varying the frequency of a notch filter that is applied to the input and added back to the original signal.\n", "\n", "## Reverb\n", "\n", "Reverb simulates acoustic propagation effects. For example, a reverb effect might emulate the wall reflections in a large concert hall. Since the acoustic effects they emulate can be complex, their implementations vary widely. Reverb effects often consist of several comb filters combined with one or more allpass filters. However, since they attempt to replicate acoustic effects of some environment, they are time-invariant." ] }, { "cell_type": "markdown", "id": "cc256fc7-e2de-462a-b86d-2f0ac545b08a", "metadata": { "slideshow": { "slide_type": "slide" }, "tags": [] }, "source": [ "## Harmonic Distortion" ] }, { "cell_type": "markdown", "id": "a55fb0ef-e9c3-415f-9e76-d9ac8f87a6a8", "metadata": { "tags": [ "remove-nb-cell" ] }, "source": [ "Recall that a Fourier series can represent any periodic signal as a combination of a fundamental frequency and equally spaced harmonics. However, if the signal or its derivative contain discontinuities, it requires an infinite number of harmonics.\n", "\n", "For example, when the linear range of an amplifier is exceeded, clipping will occur which introduces additional harmonics.\n", "\n", "![](../img/clipping.svg)" ] }, { "cell_type": "markdown", "id": "d14e41f0-a399-49fb-a621-4b5cb8160f12", "metadata": { "tags": [ "remove-cell" ] }, "source": [ "

\n", "\n", "

" ] }, { "cell_type": "markdown", "id": "5c751ae3-b47c-4c2d-b4e4-f389bda90ae0", "metadata": { "tags": [ "remove-nb-cell" ] }, "source": [ "For most systems, the introduction of new frequencies is a type of impairment. For musical applications, addition of harmonics can make an instrument have greater presence and warmth.\n", "\n", "Many types of nonlinearities can produce harmonic distortion. As a result, this category of effects often goes by many names:\n", "\n", "* Distortion\n", "* Fuzz\n", "* Overdrive\n", "* Bit crusher" ] }, { "cell_type": "markdown", "id": "5d8ecd6b-6c4b-485d-b20b-059d13920d5a", "metadata": { "tags": [ "remove-nb-cell" ] }, "source": [ "## Amplitude Modulation\n", "\n", "Any effect which only modifies the amplitude and not the phase can be considered \"amplitude modulation\"\n", "\n", "### Tremolo\n", "\n", "Tremolo refers to the effect of scaling the input by $\\beta[n]$ which osillates between some positive range. For example:\n", "\n", "$$ y[n] = x[n]\\beta[n] $$\n", "\n", "$$\\beta[n] = \\frac{1}{2} \\left(1 + \\cos(\\omega_{\\text{tremolo}}n) \\right)$$\n", "\n", "### Ring modulation\n", "\n", "If the input is scaled by an oscillating factor which is allowed to be negative, the effect is instead referred to as \"ring modulation\"\n", "\n", "$$\\beta[n] = \\cos(\\omega_{\\text{ring}}n)$$\n", "\n", "Tremolo can be considered a specific type of ring modulation." ] }, { "cell_type": "markdown", "id": "e76a29df-d558-491f-953e-237ee87d0ebb", "metadata": { "tags": [ "remove-nb-cell" ] }, "source": [ "## Vocoder\n", "\n", "Originally, the vocoder (voice encoder) was intended for applications of speech synthesis and compression. However, the term has since become synonymous with various musical effects used to combine a speech signal with a musical instrument." ] }, { "cell_type": "markdown", "id": "36ee8631-81fe-4543-a1df-3b7f75df16fe", "metadata": { "slideshow": { "slide_type": "slide" }, "tags": [] }, "source": [ "### Modeling speech" ] }, { "cell_type": "markdown", "id": "5baddc97-92e4-4ffd-8c86-324791e6991e", "metadata": { "tags": [ "remove-nb-cell" ] }, "source": [ "The [source-filter model](https://en.wikipedia.org/wiki/Source–filter_model) of speech is ubiquitous for speech synthesis. Most operating systems include speech synthesizers utilizing this model which can be accessed easily in MATLAB.\n", "\n", "```\n", "NET.addAssembly('System.Speech');\n", "speech_synth = System.Speech.Synthesis.SpeechSynthesizer;\n", "Speak(speech_synth,'Put your hands up. This is a filter bank robbery');\n", "Dispose(speech_synth);\n", "```\n", "\n", "```{raw} html\n", "\n", "```\n", "\n", "The source is typically modeled on the pressure wave that occurs when the vocal folds repeatedly open and close. One possible model is a pulse which is modulated by an impulse train.\n", "\n", "![](../img/gauss_shah.svg)" ] }, { "cell_type": "markdown", "id": "ea531029-0e9f-4b9d-828b-0283bcd745d2", "metadata": { "tags": [ "remove-cell" ] }, "source": [ "

\n", "\n", "

" ] }, { "cell_type": "markdown", "id": "60e364ad-c173-40a3-b5dc-6fd32eccaac5", "metadata": { "tags": [ "remove-nb-cell" ] }, "source": [ "The pressure wave can be modeled as the derivative of this signal.\n", "\n", "![](../img/ddt_gauss_shah.svg)" ] }, { "cell_type": "markdown", "id": "2507d0b5-b8b2-4c94-bfd2-e13cce8da34f", "metadata": { "tags": [ "remove-cell" ] }, "source": [ "

\n", "\n", "

" ] }, { "cell_type": "markdown", "id": "c4c6f85c-4d70-4b3b-85ce-3463d1c9352d", "metadata": { "tags": [ "remove-nb-cell" ] }, "source": [ "In the frequency domain, the result is concentration of energy at evenly spaced harmonics\n", "\n", "![](../img/gauss_shah_freq.svg)" ] }, { "cell_type": "markdown", "id": "920408f5-dbd3-4c43-9291-a7dab29eabc7", "metadata": { "tags": [ "remove-cell" ] }, "source": [ "

\n", "\n", "

" ] }, { "cell_type": "markdown", "id": "5d883f01-6c34-4b9c-862f-dd81743bfd69", "metadata": { "tags": [ "remove-nb-cell" ] }, "source": [ "The spacing of these harmonics is 'pitch', and their envelope contains the remaining phonetic information.\n", "\n", "A [vocoder](https://en.wikipedia.org/wiki/Vocoder) attempts to to find the time-varying parameters to a speech model from a recording. Since the sophistication of speech models varies widely, so does the sophistication of vocoders. However, the main principles can be demonstrated with just a three subsystems: (1) a source model (above), (2) [pitch detection](https://en.wikipedia.org/wiki/Pitch_detection_algorithm), and (3) an [adaptive filter](https://en.wikipedia.org/wiki/Adaptive_filter).\n", "For a musical applications, the vocoder might rencode the speech with a new frequency $f_c$ (autotune), or apply the learned filter model to a separate input signal, like a guitar.\n", "\n", "![](../img/vocoder.svg)" ] }, { "cell_type": "markdown", "id": "865dd147-08ec-4564-9e05-25bfb387de39", "metadata": { "slideshow": { "slide_type": "slide" }, "tags": [ "remove-cell" ] }, "source": [ "## Vocoder\n", "\n", "

\n", "\n", "

" ] }, { "cell_type": "markdown", "id": "9116dafb-1306-4ec7-9566-e29567a4a81d", "metadata": { "tags": [ "remove-nb-cell" ] }, "source": [ "### Autocorrelation for pitch detection\n", "\n", "Suppose that we want to estimate the time-varying fundamental frequency $f_0$ of a speech recording with an update rate of about 25 Hz. We can divide the input signal into frames which are roughly $\\frac{1}{25 \\text{ Hz}} = 40 \\text{ ms}$ long. For a sampling rate of 48 kHz, this would mean that we want to produce an estimate every $\\frac{48 \\text{ kHz}}{25 \\text{ Hz}} = 1920$ samples, though for convenience we will round this to to a power of two (2048).\n", "\n", "```\n", "[y,fs] = audioread('example.wav'); y = resample(y,320,147); fs = 48e3;\n", "L = 2048; N = floor(length(y)/L); y(N*L+1:end) = []; y = reshape(y,L,N);\n", "```\n", "\n", "The fundamental period of the signal is $T_0 = 1/f_0$. Therefore, the autocorrelation should have a peak at this offset. This fact leads to a simple pitch detection algorithm.\n", "\n", "1. Find the offset $N_0$ (in samples) where the peak autocorrelation occurs (subject to a constraint).\n", "\n", "2. Estimate that the signal in the frame is periodic with fundamental frequency $\\omega_0 = \\frac{1}{N_0} \\frac{\\text{radians}}{\\text{sample}}$.\n", "\n", "The constraint in step one is that the estimated frequency should be in the normal vocal range of about 50 to 250 Hz. Or equivalently, the offset should be between $\\frac{48 \\text{kHz}}{250 \\text{Hz}} \\approx 200$ samples and $\\frac{48 \\text{kHz}}{50 \\text{Hz}} \\approx 1000$ samples.\n", "\n", "```\n", "R = @(x,n) sum(x(n+1:end) .* x(1:end-n));\n", "Rn = []; N0 = [];\n", "for i_frame = 1:N\n", "R_max = 0; n_max = 0;\n", "for n = 200:1000\n", " Rn(i_frame) = R( y(:,i_frame), n);\n", " if Rn(i_frame) > R_max\n", " R_max = Rn(i_frame); n_max = n;\n", " end\n", " N0(i_frame) = n_max;\n", " Rn(i_frame) = R_max;\n", " end\n", "end\n", "```" ] }, { "cell_type": "markdown", "id": "57e4ce1b-5919-46ad-a6f6-1cb8e4cfc36f", "metadata": { "tags": [ "remove-nb-cell" ] }, "source": [ "### Simple model of vocal source\n", "\n", "Any waveform that matches the desired fundamental frequency and has energy widely distributed among harmonics is suitable to simulate the vocal source. For this example, we will use the (derivative of) a gaussian pulse which has been modulated by an impulse train of the desired period. We will use the magnitude of the autocorrelation to control the volume and avoid changing frequencies when no voiced speech is present.\n", "\n", "![](../img/ddt_gauss_shah.svg)\n", "\n", "```\n", "x1 = [];\n", "period = 600;\n", "for i_frame = 1:N\n", " if Rn(i_frame)>2\n", " period = N0(i_frame);\n", " end\n", " waveform1 = diff(gausswin(period,30));\n", " for i_sample = 1:L\n", " i_x = L*(i_frame-1) + i_sample;\n", " i_w = mod(i_sample,length(waveform1));\n", " i_w = i_w + (i_w == 0);\n", " x1(i_x) = (waveform1(i_w)) * abs(Rn(i_frame));\n", " if i_sample > L - period\n", " x1(i_x) = x1(i_x) * ((L - i_sample)/period);\n", " end\n", " end\n", "end\n", "```" ] }, { "cell_type": "markdown", "id": "bf5a75ca-0fc0-4726-9468-ce4b25a1df77", "metadata": { "tags": [ "remove-nb-cell" ] }, "source": [ "### Modeling the vocal tract\n", "\n", "Now that we have a model of the source, we can attempt to learn (for each new sample) an LTI approximation of the articulatory filter (lips, tounge, pharynx, etc). Many adaptive filtering techniques exist for this task. Though it has some limitations, [Least mean square equalizer](https://en.wikipedia.org/wiki/Least_mean_squares_filter), could be used for this application.\n", "\n", "```\n", "y = y(1:length(x1));\n", "n=513; h=zeros(n,1);\n", "mu=.001; delta=256;\n", "p1 = []; p2 = [];\n", "for i_sample=n+1:length(y)\n", " t1 = x1(i_sample : -1 : i_sample-n+1)';\n", " p1(end+1) = dot(t1,h);\n", " e = y(i_sample-delta) - p1(end);\n", " h = h + mu*e*t1;\n", "end\n", "soundsc(p1,fs)\n", "```\n", "\n", "```{raw} html\n", "\n", "```" ] }, { "cell_type": "markdown", "id": "3fb24b9f-0f7f-4a22-bf18-74d36befb869", "metadata": { "tags": [ "remove-nb-cell" ] }, "source": [ "## Applications\n", "\n", "The example above encodes the speech into a time-varying frequeny $f_0$ and a time-varying filter $h$. With this description of the signal, many applications are possible:\n", "\n", "* Replace the source with a musical instrument with energetic harmonics, (like a guitar, violin, or synthesizer), but use the learned time-varying filter to make the instrument 'speak'. (demonstrated in the lab exercise)\n", "\n", "* Exploit the fact that the parameterized representation contains fewer bits for speech compression.\n", "\n", "* Modify the parameters based on other information about the signal to perform denoising or source separation." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.1" } }, "nbformat": 4, "nbformat_minor": 5 }