G B

QuantLib Python CookBook Is Almost Done

Goutham Balaraman — Mon, 27 Aug 2018 00:00:00 -0700

More than a year ago, Luigi Ballabio and I made an initial release of the QuantLib Python Cookbook. Now we are mostly finished with the content of the book. The new version of the book is out, and those who bought a previous version can now log into LeanPub to download the new version of the book and the accompanying python code in the form of notebooks.

This version includes some content update and preliminary support for Python 3. We are working to convert IPython notebooks into Jupyter notebooks and that will be the next release milestone for the book. You should expect to hear from us in about a month or so.

Optimizing Python Code: Numba vs Cython

Goutham Balaraman — Thu, 03 Aug 2017 00:00:00 -0700

I came across an old post by jakevdp on Numba vs Cython. I thought I will revisit this topic because both Numba and Cython has matured significantly over this time period. In this post I am going to do two examples:

Pairwise distance estimation example that Jake discusses. The intention is to see how the maturity of these projects has contributed to improvements.
A simple cashflow payment calculation of an amortizing bond or mortgage payments. This a calculation that cannot be vectorized in a numpy sense. So the speedups would have to come from optimizing loops using tools like Numba or Cython.

In [1]:

import numpy as np
import numba 
import cython
%load_ext cython
import pandas as pd

numba.__version__, cython.__version__, np.__version__

Out[1]:

('0.31.0', '0.24.1', '1.11.3')

Pairwise Distance Estimation¶

In [2]:

X = np.random.random((1000, 3))

In [3]:

def pairwise_python(X):
    M = X.shape[0]
    N = X.shape[1]
    D = np.empty((M, M), dtype=np.float)
    for i in range(M):
        for j in range(M):
            d = 0.0
            for k in range(N):
                tmp = X[i, k] - X[j, k]
                d += tmp * tmp
            D[i, j] = np.sqrt(d)
    return D

%timeit -n10 pairwise_python(X)

10 loops, best of 3: 2.47 s per loop

In [4]:

def pairwise_numpy(X):
    return np.sqrt(((X[:, None, :] - X) ** 2).sum(-1))

%timeit -n10 pairwise_numpy(X)

10 loops, best of 3: 38.3 ms per loop

In [5]:

pairwise_numba = numba.jit(pairwise_python)

%timeit -n10 pairwise_numba(X)

The slowest run took 13.98 times longer than the fastest. This could mean that an intermediate result is being cached.
10 loops, best of 3: 4.04 ms per loop

In [6]:

%%cython
import numpy as np
cimport cython
from libc.math cimport sqrt

@cython.boundscheck(False)
@cython.wraparound(False)
def pairwise_cython(double[:, ::1] X):
    cdef int M = X.shape[0]
    cdef int N = X.shape[1]
    cdef double tmp, d
    cdef double[:, ::1] D = np.empty((M, M), dtype=np.float64)
    for i in range(M):
        for j in range(M):
            d = 0.0
            for k in range(N):
                tmp = X[i, k] - X[j, k]
                d += tmp * tmp
            D[i, j] = sqrt(d)
    return np.asarray(D)

In [7]:

%timeit -n10 pairwise_cython(X)

10 loops, best of 3: 6.6 ms per loop

The timiing for the results in Jake's post (2013) and the results from this post (2017) are summarized below.

In [8]:

df1 = pd.DataFrame({"Time (ms)": [13400,111, 9.12, 9.87], "Speedup": [1, 121, 1469, 1357]}, 
             index=["Python", "Numpy", "Numba", "Cython"])
df2 = pd.DataFrame({"Time (ms)": [2470, 38.3, 4.04, 6.6], "Speedup": [1, 65, 611, 374]}, 
             index=["Python", "Numpy", "Numba", "Cython"])
df = pd.concat([df1, df2], axis = 1, keys=(["2013", "2017"]))
df

Out[8]:

	2013		2017
	Speedup	Time (ms)	Speedup	Time (ms)
Python	1	13400.00	1	2470.00
Numpy	121	111.00	65	38.30
Numba	1469	9.12	611	4.04
Cython	1357	9.87	374	6.60

The timings are speedup number for the 2013 and 2017 runs are very different due to differences in versions and perhaps even the python version. This post is using Py35 running in Windows. The take away here is that the numpy is atleast 2 orders of magnitude faster than python. And the numba and cython snippets are about an order of magnitude faster than numpy in both the benchmarks.

I will not rush to make any claims on numba vs cython. It is unclear what kinds of optimizations is used in the cython magic. I would expect the cython code to be as fast as C and perhaps some tweaking will help us get there. It is really interesting how easy it is to get performance boost from numba. From an ease of use point of view, numba is hands down winner in this simple example.

Amortizing Payments

Here lets look at one more example. This is an amortizing payment calculation, such as in mortgage payments.

In [9]:

def amortize_payments_py(B0, R, term, cpr=0.0):
    smm = 1. - pow(1 - cpr/100., 1/12.)
    r = R/1200.
    S = np.zeros(term)
    P = np.zeros(term)
    I = np.zeros(term)
    B = np.zeros(term)
    Pr = np.zeros(term)
    Bt = B0
    pow_term = pow(1+r, term)
    A = Bt*r*pow_term/(pow_term - 1)
    for i in range(term):
        n = term-i
        
        I[i] = Bt * r
        Pr[i] = smm*Bt
        S[i] = A-I[i] if Bt>1e-2 else 0.
        P[i] = S[i] + Pr[i]
        Bt = max(Bt - P[i], 0.0)
        
        B[i] = Bt
    return S,I, Pr,P, B

Here is the equivalent Cython function.

In [10]:

%%cython
cimport cython 
import numpy as np
from libc.math cimport pow

@cython.boundscheck(False)
@cython.wraparound(False)
def amortize_payments_cy(double B0,double R,int term,double cpr=0.0):
    cdef double smm = 1. - pow(1 - cpr/100., 1/12.)
    cdef double r = R/1200.
    cdef double[:] D = np.empty(term, dtype=np.float64)
    cdef double[:] S = np.empty(term, dtype=np.float64)
    cdef double[:] P = np.empty(term, dtype=np.float64)
    cdef double[:] I = np.empty(term, dtype=np.float64)
    cdef double[:] B = np.empty(term, dtype=np.float64)
    cdef double[:] Pr = np.empty(term, dtype=np.float64)
    cdef double Bt = B0
    cdef double pow_term = pow(1+r, term)
    cdef double A = Bt*r*pow_term/(pow_term - 1.)
    cdef double n = term
    cdef int i=0
    for i in range(term):
        n = term-i       
        I[i] = Bt * r
        Pr[i] = smm*Bt
        S[i] = A-I[i] if Bt>1e-2 else 0.
        P[i] = S[i] + Pr[i]
        Bt = max(Bt - P[i], 0.0)        
        B[i] = Bt
    return np.asarray(S),np.asarray(I), np.asarray(Pr),np.asarray(P), np.asarray(B)

Here is the Numba version

In [11]:

amortize_payments_nb = numba.njit(cache=True)(amortize_payments_py)

Let's compare the performance of the three function types.

In [12]:

B0 = 500000.
R = 4.0
term = 360

Python

In [17]:

%timeit -n1000 S,I, Pr,P, B = amortize_payments_py(B0, R, term, cpr=10)

1000 loops, best of 3: 369 µs per loop

Numba

In [18]:

%timeit -n1000 S,I, Pr,P, B = amortize_payments_nb(B0, R, term, cpr=10)

1000 loops, best of 3: 8.64 µs per loop

Cython

In [19]:

%timeit -n1000 S,I, Pr,P, B = amortize_payments_cy(B0, R, term, cpr=10)

1000 loops, best of 3: 28.7 µs per loop

Once again we see that cython and numba methods are orders of magnitude (roughly 15X) faster than python version. The numba version is almost 3X faster than Cython. There are probably some performance cython tweaks that I am missing perhaps.

Conclusion

In this post, I have redone Jake's pairwise distance example to see how the same code performs 3 years forward. I also discuss a common finance example of amortizing cashflow generation. The numba and cython snippets are orders of magnitude faster than a pure python version. Surprisingly, numba is 20% to 300% faster than cython on these examples. There may very well be some cython tweaks I might be missing. But nevertheless these examples show how one can easily get performance boost using numba module.

American Option Pricing with QuantLib and Python

Goutham Balaraman — Sun, 23 Jul 2017 00:00:00 -0700

I wrote about pricing European options using QuantLib in an earlier post. Since then, I have received many questions from readers on how to extend this to price American options. So here is a modified example on pricing American options using QuantLib. The idea is very similar to European Option construction. Lets take a look at the details below. In this post, I will price both an European option and an American option side by side.

In [1]:

import QuantLib as ql 
import matplotlib.pyplot as plt
%matplotlib inline
ql.__version__

Out[1]:

'1.9.2'

Let us consider a European and an American call option for AAPL with a strike price of \$130 maturing on 15th Jan, 2016. Let the spot price be \$127.62. The volatility of the underlying stock is known to be 20%, and has a dividend yield of 1.63%. Lets value these options as of 8th May, 2015.

In [2]:

# option data
maturity_date = ql.Date(15, 1, 2016)
spot_price = 127.62
strike_price = 130
volatility = 0.20 # the historical vols or implied vols
dividend_rate =  0.0163
option_type = ql.Option.Call

risk_free_rate = 0.001
day_count = ql.Actual365Fixed()
calendar = ql.UnitedStates()

calculation_date = ql.Date(8, 5, 2015)
ql.Settings.instance().evaluationDate = calculation_date

We construct the European and American options here. The main difference here is in the Exercise type. One has to use AmericanExercise instead of EuropeanExercise to pass into the VanillaOption to construct an American option.

In [3]:

payoff = ql.PlainVanillaPayoff(option_type, strike_price)
settlement = calculation_date

am_exercise = ql.AmericanExercise(settlement, maturity_date)
american_option = ql.VanillaOption(payoff, am_exercise)

eu_exercise = ql.EuropeanExercise(maturity_date)
european_option = ql.VanillaOption(payoff, eu_exercise)

The Black-Scholes-Merton process is constructed here.

In [4]:

spot_handle = ql.QuoteHandle(
    ql.SimpleQuote(spot_price)
)
flat_ts = ql.YieldTermStructureHandle(
    ql.FlatForward(calculation_date, risk_free_rate, day_count)
)
dividend_yield = ql.YieldTermStructureHandle(
    ql.FlatForward(calculation_date, dividend_rate, day_count)
)
flat_vol_ts = ql.BlackVolTermStructureHandle(
    ql.BlackConstantVol(calculation_date, calendar, volatility, day_count)
)
bsm_process = ql.BlackScholesMertonProcess(spot_handle, 
                                           dividend_yield, 
                                           flat_ts, 
                                           flat_vol_ts)

The value of the American option can be computed using a Binomial Engine using the CRR approach.

In [5]:

steps = 200
binomial_engine = ql.BinomialVanillaEngine(bsm_process, "crr", steps)
american_option.setPricingEngine(binomial_engine)
print (american_option.NPV())

6.84210328728556

For illustration purpose, lets compare the European and American option prices using the binomial tree approach.

In [6]:

def binomial_price(option, bsm_process, steps):
    binomial_engine = ql.BinomialVanillaEngine(bsm_process, "crr", steps)
    option.setPricingEngine(binomial_engine)
    return option.NPV()

steps = range(5, 200, 1)
eu_prices = [binomial_price(european_option, bsm_process, step) for step in steps]
am_prices = [binomial_price(american_option, bsm_process, step) for step in steps]
# theoretican European option price
european_option.setPricingEngine(ql.AnalyticEuropeanEngine(bsm_process))
bs_price = european_option.NPV()

In the plot below, the binomial-tree approach is used to value American option for different number of steps. You can see the prices converging with increase in number of steps. The European option price is plotted along with BSM theoretical price for comparison purposes.

In [7]:

plt.plot(steps, eu_prices, label="European Option", lw=2, alpha=0.6)
plt.plot(steps, am_prices, label="American Option", lw=2, alpha=0.6)
plt.plot([5,200],[bs_price, bs_price], "r--", label="BSM Price", lw=2, alpha=0.6)
plt.xlabel("Steps")
plt.ylabel("Price")
plt.ylim(6.7,7)
plt.title("Binomial Tree Price For Varying Steps")
plt.legend()

Out[7]:

<matplotlib.legend.Legend at 0x85b24e0>

Variance Reduction in Hull-White Monte Carlo Simulation Using Moment Matching

Goutham Balaraman — Mon, 26 Jun 2017 00:00:00 -0700

In an earlier blog post on how the Hull-White Monte Carlo simulations are notorious for not coverging with some of the expected moments. In this post, I would like to touch upon a variance reduction technique called moment matching that can be employed to fix this issue of convergence.

The idea behind moment matching is rather simple. Lets consider the specific example of short rate model. For the short rate model, it is known that the average of stochastic discount factors generated from each path has to agree with the model (or the give yield curve) discount factors. The idea of moment matching is to correct the short rates generated by the term structure model such that the average of stochastic discount factors from the simulation matches the model discount factors.

In [1]:

import QuantLib as ql
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
from scipy.integrate import cumtrapz
ql.__version__

Out[1]:

'1.11'

For simplicity, we use a constant forward rate as the given interest rate term structure. The method discussed here would work with any market yield curve as well.

In [2]:

sigma = 0.01
a = 0.001
timestep = 360
length = 30 # in years
forward_rate = 0.05
day_count = ql.Thirty360()
todays_date = ql.Date(15, 1, 2015)

In [3]:

ql.Settings.instance().evaluationDate = todays_date

yield_curve = ql.FlatForward(
    todays_date, 
    ql.QuoteHandle(ql.SimpleQuote(forward_rate)), 
    day_count)
spot_curve_handle = ql.YieldTermStructureHandle(yield_curve)

Here, I setup the Monte Carlo simulation of the Hull-White process. The result of the generate_paths function below is the time grid and a matrix of short rates generated by the model. This is discussed in detaio in the Hull-White simulation post.

In [4]:

hw_process = ql.HullWhiteProcess(spot_curve_handle, a, sigma)
rng = ql.GaussianRandomSequenceGenerator(
    ql.UniformRandomSequenceGenerator(
        timestep, 
        ql.UniformRandomGenerator(125)))
seq = ql.GaussianPathGenerator(hw_process, length, timestep, rng, False)

In [5]:

def generate_paths(num_paths, timestep):
    arr = np.zeros((num_paths, timestep+1))
    for i in range(num_paths):
        sample_path = seq.next()
        path = sample_path.value()
        time = [path.time(j) for j in range(len(path))]
        value = [path[j] for j in range(len(path))]
        arr[i, :] = np.array(value)
    return np.array(time), arr

Here is a plot of the generated short rates.

In [6]:

num_paths = 128
time, paths = generate_paths(num_paths, timestep)
for i in range(num_paths):
    plt.plot(time, paths[i, :], lw=0.8, alpha=0.6)
plt.title("Hull-White Short Rate Simulation")
plt.show()

The model zero coupon bond price $B(0, T)$ is given as: $$B(0, T) = E\left[\exp\left(-\int_0^T r(t)dt \right) \right]$$ where $r(t)$ is the short rate generated by the model. The expectation of the stochastic discount factor at time $T$ is the price of the zero coupon bond at that time. In a simulation the paths are generated in a time grid and the discretization introduces some error. The empirical estimation of the zero coupon bond price from a Monte Carlo simulation $\hat{B}(0, t_m)$ maturing at time $t_m$ is given as:

$$\hat{B}(0, t_m) = \frac{1}{N}\sum_{i=1}^{N} \exp\left(-\sum_{j=0}^{m-1} \hat{r}_i(t_j)[t_{j+1}-t_j] \right)$$

where $\hat{r}_i(t_j)$ is the short rate for the path $i$ at time $t_j$ on the time grid. The expression for the moment matched short rates is given as [1]:

$$ r^c_i(t_j) = \hat{r}_i(t_j) + \frac{\log \hat{B}(0, t_{j+1}) - \log \hat{B}(0, t_{j})}{t_{j+1} - t_j} - \frac{\log B(0, t_{j+1}) - \log B(0, t_{j})}{t_{j+1} - t_j}$$

In [7]:

def stoch_df(paths, time):
    return np.mean(
        np.exp(-cumtrapz(paths, time, initial=0.)),axis=0
    )
B_emp = stoch_df(paths, time)
logB_emp = np.log(B_emp)
B_yc = np.array([yield_curve.discount(t) for t in time])
logB_yc = np.log(B_yc)

In [8]:

deltaT = time[1:] - time[:-1]
deltaB_emp = logB_emp[1:]-logB_emp[:-1]
deltaB_yc = logB_yc[1:] - logB_yc[:-1]

In [9]:

new_paths = paths.copy()
new_paths[:,1:] += (deltaB_emp/deltaT - deltaB_yc/deltaT)

The plots below show the zero coupon bond price and mean of short rates with and without the moment matching.

In [10]:

plt.plot(time,
         stoch_df(paths, time),"r-.", 
         label="Original", lw=2)
plt.plot(time,
         stoch_df(new_paths, time),"g:",
         label="Corrected", lw=2)
plt.plot(time,B_yc, "k--",label="Market", lw=1)
plt.title("Zero Coupon Bond Price")
plt.legend()

Out[10]:

<matplotlib.legend.Legend at 0x7f78aa2f1990>

In [11]:

def alpha(forward, sigma, a, t):
    return forward + 0.5* np.power(sigma/a*(1.0 - np.exp(-a*t)), 2)

avg = [np.mean(paths[:, i]) for i in range(timestep+1)]
new_avg = [np.mean(new_paths[:, i]) for i in range(timestep+1)]
plt.plot(time, avg, "r-.", lw=3, alpha=0.6, label="Original")
plt.plot(time, new_avg, "g:", lw=3, alpha=0.6, label="Corrected")
plt.plot(time,alpha(forward_rate, sigma, a, time), "k--", lw=2, alpha=0.6, label="Model")
plt.title("Mean of Short Rates")
plt.legend(loc=0)

Out[11]:

<matplotlib.legend.Legend at 0x7f78a8a3d2d0>

Reference

[1] Paul Glasserman, Monte Carlo Methods in Financial Engineering, Chapter 4

Multi-Processing with Pandas and Dask

Goutham Balaraman — Thu, 04 May 2017 00:00:00 -0700

I wrote a post on multiprocessing with pandas a little over 2 years back. A lot has changed, and I have started to use dask and distributed for distributed computation using pandas. Here I will show how to implement the multiprocessing with pandas blog using dask.

For this example, I will download and use the NYC Taxi & Limousine data. In this example I will use the January 2009 Yellow tripdata file (2GB in size), and run on my laptop. Extending to multiple data files and much larger sizes is possible too.

We start by importing dask.dataframe below.

In [1]:

import dask.dataframe as dd

Any large CSV (and other format) file can be read using a pandas like read_csv command.

In [2]:

df = dd.read_csv(r"C:\temp\yellow_tripdata_2009-01.csv")

It is important to understand that unlike the pandas read_csv, the above command does not actually load the data. It does some data inference, and leaves the other aspects for later.

Using the npartitions attribute, we can see how many partitions the data will be broken in for loading. Viewing the raw df object would give you a shell of the dataframe with column and datatypes inferred. The actual data is not loaded yet.

In [3]:

df.npartitions

Out[3]:

You can infer the columns and datatypes.

In [4]:

df.columns

Out[4]:

Index(['vendor_name', 'Trip_Pickup_DateTime', 'Trip_Dropoff_DateTime',
       'Passenger_Count', 'Trip_Distance', 'Start_Lon', 'Start_Lat',
       'Rate_Code', 'store_and_forward', 'End_Lon', 'End_Lat', 'Payment_Type',
       'Fare_Amt', 'surcharge', 'mta_tax', 'Tip_Amt', 'Tolls_Amt',
       'Total_Amt'],
      dtype='object')

In [5]:

df.dtypes

Out[5]:

vendor_name               object
Trip_Pickup_DateTime      object
Trip_Dropoff_DateTime     object
Passenger_Count            int64
Trip_Distance            float64
Start_Lon                float64
Start_Lat                float64
Rate_Code                float64
store_and_forward        float64
End_Lon                  float64
End_Lat                  float64
Payment_Type              object
Fare_Amt                 float64
surcharge                float64
mta_tax                  float64
Tip_Amt                  float64
Tolls_Amt                float64
Total_Amt                float64
dtype: object

Computing the length of the dataset can be done by using the size attribute.

In [6]:

size = df.size
size, type(size)

Out[6]:

(dd.Scalar<size-ag..., dtype=int32>, dask.dataframe.core.Scalar)

As you can see above, the size does not return a value yet. The computation is actually defferred until we compute it.

In [7]:

%%time
size.compute()

Wall time: 48 s

Out[7]:

253663434

This computation comes back with 25MM rows. This computation actually took a while. This is because when we compute size, we are not only calculating the size of the data, but we are also actually loading the dataset. Now you think that is not very efficient. There are a couple of approaches you can take:

If you have access to a (cluster of )computers with large enough RAM, then you can load and persist the data in memory. The subsequent computations will compute in memory and will be a lot faster. This also allows you to do many computations much like using pandas but in a distributed paradigb.
Another approach is to setup a whole bunch of deferred computations, and to compute out of core. Then dask will intelligently load data and process all the computations once by figuring out the various dependencies. This is a great approach if you don't have a lot of RAM available.

Now the way to load data in memory is by using the persist method on the df object.

In [8]:

df = df.persist()

The above persist call is non-blocking and you need to wait a bit for the data to load. Once it is loaded, you can compute the size as above.

In [9]:

%%time
df.size.compute()

Wall time: 35 ms

Out[9]:

253663434

That computed instantly. Now you can scale to much larger data sizes and compute in parallel.

Numpy Vs Pandas Performance Comparison

Goutham Balaraman — Tue, 14 Mar 2017 00:00:00 -0700

Pandas and Numpy are two packages that are core to a lot of data analysis. In this post I will compare the performance of numpy and pandas.

tl;dr:

numpy consumes less memory compared to pandas
numpy generally performs better than pandas for 50K rows or less
pandas generally performs better than numpy for 500K rows or more
for 50K to 500K rows, it is a toss up between pandas and numpy depending on the kind of operation

In [1]:

import pandas as pd
import matplotlib.pyplot as plt
plt.style.use("seaborn-pastel")
%matplotlib inline
import seaborn.apionly as sns
import numpy as np
from timeit import timeit 
import sys

In [2]:

iris = sns.load_dataset('iris')

In [3]:

data = pd.concat([iris]*100000)
data_rec = data.to_records()

In [4]:

print (len(data), len(data_rec))

15000000 15000000

Here I have loaded the iris dataset and replicated it so as to have 15MM rows of data. The space requirement for 15MM rows of data in a pandas dataframe is more than twice that of a numpy recarray.

In [5]:

MB = 1024*1024
print("Pandas %d MB " % (sys.getsizeof(data)/MB))
print("Numpy %d MB " % (sys.getsizeof(data_rec)/MB))

Pandas 1506 MB 
Numpy 686 MB

A snippet of the data shown below.

In [6]:

data.head()

Out[6]:

	sepal_length	sepal_width	petal_length	petal_width	species
0	5.1	3.5	1.4	0.2	setosa
1	4.9	3.0	1.4	0.2	setosa
2	4.7	3.2	1.3	0.2	setosa
3	4.6	3.1	1.5	0.2	setosa
4	5.0	3.6	1.4	0.2	setosa

In [7]:

Expand Code

In this post, performance metrics for a few different categories are compared between numpy and pandas:

operations on a column of data, such as mean or applying a vectorised function
operations on a filtered column of data
vector operations on a column or filtered column

Operations on a Column¶

Here some performance metrics with operations on one column of data. The operations involved in here include fetching a view, and a reduction operation such as mean, vectorised log or a string based unique operation. All these are O(n) calculations. The mean calculation is orders of magnitude faster in numpy compared to pandas for array sizes of 100K or less. For sizes larger than 100K pandas maintains a lead over numpy.

In [8]:

bench(data, "data.loc[:, 'sepal_length'].mean()", 
      data_rec, "np.mean(data_rec.sepal_length)",
     title="Mean on Unfiltered Column")

Below, the vectorized log operation is faster in numpy for sizes less than 100K but pandas costs about the same for sizes larger than 100K.

In [9]:

bench(data, "np.log(data.loc[:, 'sepal_length'])",
      data_rec, "np.log(data_rec.sepal_length)",
     title="Vectorised log on Unfiltered Column")

The one differentiating aspect about the test below is that the column species is of string type. The operation demonstrated is a unique calculation. We observe that the unique calculation is roughly an order of magnitude faster in pandas for sizes larger than 1K rows.

In [10]:

bench(data, "data.loc[:,'species'].unique()",
      data_rec, "np.unique(data_rec.species)", 
      grid=np.array([100, 1000, 10000, 100000, 1000000]),
     title="Unique on Unfiltered String Column")

Operations on a Filtered Column¶

Below we perform the same tests as above, except that the column is not a full view, but is instead a filtered view. The filters are simple filters with an arithmetic bool comparison for the first two and a string comparison for the third below.

Below, mean is calculated for a filtered column sepal_length. Here performance of pandas is better for row sizes larger than 10K. In the mean on unfiltered column shown above, pandas performed better for 1MM or more. Just having selection operations has shifted performance chart in favor of pandas for even smaller number of records.

In [11]:

bench(data, "data.loc[(data.sepal_width>3) & \
             (data.petal_length<1.5), 'sepal_length'].mean()",
      data_rec, "np.mean(data_rec[(data_rec.sepal_width>3) & \
                 (data_rec.petal_length<1.5)].sepal_length)",
      grid=np.array([1000, 10000, 100000, 1000000]),
     title="Mean on Filtered Column")

For vectorised log operation on a unfiltered column shown above, numpy performed better than pandas for number of records less than 100K while the performance was comparable for the two for sizes larger than 100K. But the moment you introduce a filter on a column, pandas starts to show an edge over numpy for number of records larger than 10K.

In [12]:

bench(data, "np.log(data.loc[(data.sepal_width>3) & \
            (data.petal_length<1.5), 'sepal_length'])",
      data_rec, "np.log(data_rec[(data_rec.sepal_width>3) & \
                 (data_rec.petal_length<1.5)].sepal_length)",
      grid=np.array([1000, 10000, 100000, 1000000]),
     title="Vectorised log on Filtered Column")

Here is another example of a mean reduction on a column but with a string filter. We see a similar behavior where numpy performs significantly better at small sizes and pandas takes a gentle lead for larger number of records.

In [13]:

bench(data, "data[data.species=='setosa'].sepal_length.mean()",
      data_rec, "np.mean(data_rec[data_rec.species=='setosa'].sepal_length)",
      grid=np.array([1000, 10000, 100000, 1000000]),
     title="Mean on (String) Filtered Column")

Vectorized Operation on a Column¶

In this last section, we do vectorised arithmetic using multiple columns. This involves creating a view and vectorised math on these views. Even when there is no filter, pandas has a slight edge over numpy for large number of records. For smaller than 100K records, numpy performs significantly better.

In [14]:

bench(data, "data.petal_length * data.sepal_length + \
      data.petal_width * data.sepal_width",
      data_rec, "data_rec.petal_length*data_rec.sepal_length + \
      data_rec.petal_width * data_rec.sepal_width",
     title="Vectorised Math on Unfiltered Columns")

In the following figure, the filter involves vectorised arithmetic operation, and mean reduction is computed on the filtered column. The presence of a filter makes pandas significantly faster for sizes larger than 100K, while numpy maitains a lead for smaller than 10K number of records.

In [16]:

bench(data, "data.loc[data.sepal_width * data.petal_length > \
             data.sepal_length, 'sepal_length'].mean()",
      data_rec, "np.mean(data_rec[data_rec.sepal_width * data_rec.petal_length \
                 > data_rec.sepal_length].sepal_length)",
     title="Vectorised Math in Filtering Columns",
     grid=np.array([100, 1000, 10000, 100000, 1000000]))

Conclusion¶

Pandas is often used in an interactive environment such as through Jupyter notebooks. In such a case, any performance loss from pandas will be in significant. But if you have smaller pandas dataframes (<50K number of records) in a production environment, then it is worth considering numpy recarrays.

numpy consumes (roughtly 1/3) less memory compared to pandas
numpy generally performs better than pandas for 50K rows or less
pandas generally performs better than numpy for 500K rows or more
for 50K to 500K rows, it is a toss up between pandas and numpy depending on the kind of operation

Profiling Python Code in Jupyter Notebooks

Goutham Balaraman — Thu, 02 Mar 2017 00:00:00 -0800

Sometimes you want to quickly identify performance bottlenecks in your code. You can use some of these recipes while using the Jupyter notebook environment. Jupyter allows a few magic commands that are great for timing and profiling a line of code or a block of code. Let us take a look at a really simple example with these functions:

import numpy as np

def func(n, a):
  y = np.arange(n)
  y = np.exp(-y*a)
  return y

def gunc(n, a):
  y = np.exp(-n*a)
  return y

def hunc(n, a):
  y1 = func(n, a)
  y2 = gunc(n, a)
  return y1, y2

timeit

Now to see which one of these is faster, you can use the %timeit magic command:

%timeit -n 3 func(10,0.5)

3 loops, best of 3: 14.1 µs per loop

Here the -n 3 denotes the number of loops to execute. The default is 100000 loops. Now let's say you have both func and gunc in a cell, and you want to measure the time taken, then you can use %%timeit in the block. Notice the double percentage sign:

%%timeit -n 3
func(10, 0.5)
gunc(10, 0.5)

3 loops, best of 3: 18.2 µs per loop

time

On some occasions, you can get the time taken using the %time magic command for the line or %%time for the cell block.:

%%time
func(60000, 0.5)
gunc(10, 0.5)

Wall time: 10 ms

prun

The above recipes are more of timing code than profiling code. There is a profiler magic command %prun and %%prun that does function level code profiling. For example

%prun hunc(50000, 0.5)

7 function calls in 0.010 seconds

 Ordered by: internal time

 ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      1    0.009    0.009    0.009    0.009 <ipython-input-23-5c434cf7f3ad>:3(func)
      1    0.001    0.001    0.001    0.001 {built-in method numpy.core.multiarray.arange}
      1    0.000    0.000    0.010    0.010 <string>:1(<module>)
      1    0.000    0.000    0.010    0.010 {built-in method builtins.exec}
      1    0.000    0.000    0.000    0.000 <ipython-input-23-5c434cf7f3ad>:8(gunc)
      1    0.000    0.000    0.009    0.009 <ipython-input-23-5c434cf7f3ad>:12(hunc)
      1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

lprun

Lastly, you can install the line_profiler if you want to dig a little deep to understand what line in the code is slow. You can install as:

pip install line_profiler

This extension be loaded as:

%load_ext line_profiler

Let's say we have a hunch that the func call in hunc is the bottleneck, but we are wondering which line in func is the culprit, then here is how %lprun can help.

%lprun -f func hunc(50000, 0.5)

Timer unit: 3.00459e-07 s

Total time: 0.00950652 s
File: <ipython-input-23-5c434cf7f3ad>
Function: func at line 3

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     3                                           def func(n, a):
     4         1         1288   1288.0      4.1      y = np.arange(n)
     5         1        30340  30340.0     95.9      y = np.exp(-y*a)
     6         1           12     12.0      0.0      return yTimer unit: 3.00459e-07 s

If you need to profile some function in a python package, then import that function and stick it after -f flag. Happy profiling!

Introducting Introspection in QTK

Goutham Balaraman — Thu, 01 Sep 2016 00:00:00 -0700

Last couple of months, I have been working on the infrastructure of QTK. I annouced the first point release a couple of weeks back after I implemented the core internals. Recently, I implemented the workflow paradigm for qtk. I will highlight some of the core concepts here. The Template in QTK is like a blueprint for what you want to accomplish. Let us see how to create a flat forward curve using some of the introspection concepts introduced in 0.1.3 onwards.

In [1]:

from qtk import Template as T, Controller

The above statement is used to import the template. Now you can use the tab functionality in ipython to get all the different types of templates. Now if you want to get the description of how to create this template, then all you do is execute that template as shown below. This is going to produce a rather nice output for you to view with the required variables and such.

In [2]:

T.TS_YIELD_FLAT

Out[2]:

Description

A template to create a flat forward yield curve.

Required Fields

Template [Template]: 'TermStructure.Yield.FlatCurve'
ForwardRate [Float]: Forward rate from an yield curve in decimal
Currency [String]: Currency

Optional Fields

ObjectId [String]: A unique name or identifier to refer to this dictionary data
AsOfDate [Date]: As of date for the yield curve to create a curve fixed to a given reference date. Alternately, one can provide settlement days and settlement calendar to do relative to calculation date.
DiscountBasis [DayCount]: Discount Basis
SettlementDays [Integer]: Settlement days, and is used if asof date is not provided.
SettlementCalendar [Calendar]: Settlement Calendar
Compounding [Compounding]: Compounding
CompoundingFrequency [Frequency]: Compounding Frequency
Extrapolation [Boolean]: Enable Extrapolation

Or if you use a simple python console, then you can use the help to print the info in Markdown format. If you just need the help string, then you can use the info method.

In [3]:

T.TS_YIELD_FLAT.help()

**Description**

A template to create a flat forward yield curve.

**Required Fields**

 - `Template` [*Template*]: 'TermStructure.Yield.FlatCurve'
 - `ForwardRate` [*Float*]: Forward rate from an yield curve in decimal
 - `Currency` [*String*]: Currency

**Optional Fields**

 - `ObjectId` [*String*]: A unique name or identifier to refer to this dictionary data
 - `AsOfDate` [*Date*]: As of date for the yield curve to create a curve fixed to a given reference date. Alternately, one can provide settlement days and settlement calendar to do relative to calculation date.
 - `DiscountBasis` [*DayCount*]: Discount Basis
 - `SettlementDays` [*Integer*]: Settlement days, and is used if asof date is not provided.
 - `SettlementCalendar` [*Calendar*]: Settlement Calendar
 - `Compounding` [*Compounding*]: Compounding
 - `CompoundingFrequency` [*Frequency*]: Compounding Frequency
 - `Extrapolation` [*Boolean*]: Enable Extrapolation

In order to create a yield curve, you can use the sample_data method attached to the template to get a sample dictionary object. This is what it looks like.

In [4]:

T.TS_YIELD_FLAT.sample_data()

Out[4]:

{'AsOfDate': 'Optional (Date)',
 'Compounding': 'Optional (Compounding)',
 'CompoundingFrequency': 'Optional (Frequency)',
 'Currency': 'Required (String)',
 'DiscountBasis': 'Optional (DayCount)',
 'Extrapolation': 'Optional (Boolean)',
 'ForwardRate': 'Required (Float)',
 'ObjectId': 'Optional (String)',
 'SettlementCalendar': 'Optional (Calendar)',
 'SettlementDays': 'Optional (Integer)',
 'Template': 'TermStructure.Yield.FlatCurve'}

You can prune the optional fields to just pass the required items. It is worth pointing out that sometimes when one of two keywords need to be provided, both would be marked as optional. Looking at the documentation as listed by info or help methods will clarify the usage. For example, the fixed rate bond template INSTRUMENT_BOND_TBOND has Coupon and ListOfCoupon both marked as optional fields. But this is because you can provide either one of the two as inputs, but atleast one of the two has to be provided. This is clarified in the documentation.

In [5]:

zero_curve = {"Template": T.TS_YIELD_FLAT.id, "ForwardRate": 0.02, "Currency": "USD", "ObjectId": "FlatCurve.USD"}
data = [zero_curve]

In [6]:

c = Controller(data)
out = c.process("1/15/2016")
ts = c.object("FlatCurve.USD")

In [7]:

ts.referenceDate()

Out[7]:

Date(20,1,2016)

In [8]:

import QuantLib as ql
time_grid = range(30)
zero = [ts.zeroRate(t, ql.Compounded).rate() for t in time_grid]

In [9]:

import matplotlib.pyplot as plt
%matplotlib inline

In [10]:

plt.scatter(time_grid, zero)
plt.xlabel("Time", size=12)
plt.ylabel("Zero Rate", size=12)

Out[10]:

<matplotlib.text.Text at 0x7f1f95768690>

Valuing Callable Bonds Using QuantLib Python

Goutham Balaraman — Tue, 30 Aug 2016 00:00:00 -0700

In this post, I will walk you through on how to value callable bonds in QuantLib Python. The approach to construct a callable bond is lot similar to creating a fixed rate bond in QuantLib. The one additional input that we need to provide here is the details on the callable schedule. If you follow the fixed rate bond example already, this should be fairly straight forward.

As always, we will start with some initializations and imports.

In [1]:

import QuantLib as ql
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
calc_date = ql.Date(16,8,2016)
ql.Settings.instance().evaluationDate = calc_date

For simplicity, I am going to assume that the interest rate term structure is a flat yield curve at 3.5%.

In [2]:

day_count = ql.ActualActual(ql.ActualActual.Bond)
rate = 0.035
ts = ql.FlatForward(calc_date, 
                    rate, 
                    day_count, 
                    ql.Compounded, 
                    ql.Semiannual)
ts_handle = ql.YieldTermStructureHandle(ts)

The call and put schedules for the callable bond is created as shown below. I create a container for holding all the call and put dates using the CallabilitySchedule class. You can add each call using Callability class and noting as Callability.Call or Callability.Put for either a call or put.

In [3]:

callability_schedule = ql.CallabilitySchedule()
call_price = 100.0
call_date = ql.Date(15,ql.September,2016);
null_calendar = ql.NullCalendar();
for i in range(0,24):
    callability_price  = ql.CallabilityPrice(
        call_price, ql.CallabilityPrice.Clean)
    callability_schedule.append(
            ql.Callability(callability_price, 
                           ql.Callability.Call,
                           call_date))

    call_date = null_calendar.advance(call_date, 3, ql.Months);

What follows next is very similar to the Schedule that we created in the vanilla fixed rate bond valuation.

In [4]:

issue_date = ql.Date(16,ql.September,2014)        
maturity_date = ql.Date(15,ql.September,2022)
calendar = ql.UnitedStates(ql.UnitedStates.GovernmentBond)
tenor = ql.Period(ql.Quarterly)
accrual_convention = ql.Unadjusted

schedule = ql.Schedule(issue_date, maturity_date, tenor,
                       calendar, accrual_convention, accrual_convention,
                       ql.DateGeneration.Backward, False)

The callable bond is instantiated using the CallableFixedRateBond class, which accepts the bond inputs and the call / put schedule.

In [5]:

settlement_days = 3
face_amount = 100
accrual_daycount = ql.ActualActual(ql.ActualActual.Bond)
coupon = 0.025


bond = ql.CallableFixedRateBond(
    settlement_days, face_amount,
    schedule, [coupon], accrual_daycount,
    ql.Following, face_amount, issue_date,
    callability_schedule)

In order to value the bond, we need an interest rate model to model the fact that the bond will get called or not in the future depending on where the future interest rates are at. The TreeCallableFixedRateBondEngine can be used to value the callable bond. Below, the value_bond function prices the callable bond based on the Hull-White model parameter for mean reversion and volatility.

In [6]:

def value_bond(a, s, grid_points, bond):
    model = ql.HullWhite(ts_handle, a, s)
    engine = ql.TreeCallableFixedRateBondEngine(model, grid_points)
    bond.setPricingEngine(engine)
    return bond

The callable bond value for a 3% mean reversion and 12% volatility is shown below.

In [7]:

value_bond(0.03, 0.12, 40, bond)
print "Bond price: ",bond.cleanPrice()

Bond price:  68.3769646975

The price sensitivity of callable bonds to that of volatility parameter is shown below. As volatility increases, there is a higher chance of it being callable. Hence the value of the bond decreases.

In [8]:

sigmas = np.arange(0.001, 0.15, 0.001)
prices = [value_bond(0.03, s, 40, bond).cleanPrice() 
          for s in sigmas]

In [9]:

plt.plot(sigmas, prices)

Out[9]:

[<matplotlib.lines.Line2D at 0x7fa0abbdbd90>]

The static cashflows can be accessed using the cashflows accessor.

In [10]:

for c in bond.cashflows():
    print c.date(), "     ", c.amount()

December 15th, 2,014       0.618131868132
March 16th, 2,015       0.625
June 15th, 2,015       0.625
September 15th, 2,015       0.625
December 15th, 2,015       0.625
March 15th, 2,016       0.625
June 15th, 2,016       0.625
September 15th, 2,016       0.625
December 15th, 2,016       0.625
March 15th, 2,017       0.625
June 15th, 2,017       0.625
September 15th, 2,017       0.625
December 15th, 2,017       0.625
March 15th, 2,018       0.625
June 15th, 2,018       0.625
September 17th, 2,018       0.625
December 17th, 2,018       0.625
March 15th, 2,019       0.625
June 17th, 2,019       0.625
September 16th, 2,019       0.625
December 16th, 2,019       0.625
March 16th, 2,020       0.625
June 15th, 2,020       0.625
September 15th, 2,020       0.625
December 15th, 2,020       0.625
March 15th, 2,021       0.625
June 15th, 2,021       0.625
September 15th, 2,021       0.625
December 15th, 2,021       0.625
March 15th, 2,022       0.625
June 15th, 2,022       0.625
September 15th, 2,022       0.625
September 15th, 2,022       100.0

Conclusion

Here we explored a minimal example on pricing a callable bond.

Heston Model Calibration Using QuantLib Python and Scipy Optimize

Goutham Balaraman — Sun, 31 Jul 2016 00:00:00 -0700

I have discussed parameter calibration in a couple of my earlier posts. In this post I want to show how you can use QuantLib Python and Scipy to do parameter calibration. In order to run this, you will need to build the QuantLib github master and the latest SWIG code with my pull request. Alternately, this should get merged into version 1.9 and you should be able to use it when it is released. This pull request adds some of the moethods of the CalibratedModel such as calibrationError that we will use in calibrating models using Scipy. QuantLib's strength is all financial models. Scipy's strength is all the solvers and numerical methods. So here, I will show you how you can make the best of both worlds. We will start as usual by importing the modules.

In [1]:

import QuantLib as ql
from math import pow, sqrt
import numpy as np
from scipy.optimize import root

Let's construct some of the basic dependencies such as the yield and dividend term structures.

In [2]:

day_count = ql.Actual365Fixed()
calendar = ql.UnitedStates()
calculation_date = ql.Date(6, 11, 2015)

spot = 659.37
ql.Settings.instance().evaluationDate = calculation_date

risk_free_rate = 0.01
dividend_rate = 0.0
yield_ts = ql.YieldTermStructureHandle(
    ql.FlatForward(calculation_date, risk_free_rate, day_count))
dividend_ts = ql.YieldTermStructureHandle(
    ql.FlatForward(calculation_date, dividend_rate, day_count))

Following is a sample grid of volatilities for different expiration and strikes.

In [3]:

expiration_dates = [ql.Date(6,12,2015), ql.Date(6,1,2016), ql.Date(6,2,2016),
                    ql.Date(6,3,2016), ql.Date(6,4,2016), ql.Date(6,5,2016), 
                    ql.Date(6,6,2016), ql.Date(6,7,2016), ql.Date(6,8,2016),
                    ql.Date(6,9,2016), ql.Date(6,10,2016), ql.Date(6,11,2016), 
                    ql.Date(6,12,2016), ql.Date(6,1,2017), ql.Date(6,2,2017),
                    ql.Date(6,3,2017), ql.Date(6,4,2017), ql.Date(6,5,2017), 
                    ql.Date(6,6,2017), ql.Date(6,7,2017), ql.Date(6,8,2017),
                    ql.Date(6,9,2017), ql.Date(6,10,2017), ql.Date(6,11,2017)]
strikes = [527.50, 560.46, 593.43, 626.40, 659.37, 692.34, 725.31, 758.28]
data = [
[0.37819, 0.34177, 0.30394, 0.27832, 0.26453, 0.25916, 0.25941, 0.26127],
[0.3445, 0.31769, 0.2933, 0.27614, 0.26575, 0.25729, 0.25228, 0.25202],
[0.37419, 0.35372, 0.33729, 0.32492, 0.31601, 0.30883, 0.30036, 0.29568],
[0.37498, 0.35847, 0.34475, 0.33399, 0.32715, 0.31943, 0.31098, 0.30506],
[0.35941, 0.34516, 0.33296, 0.32275, 0.31867, 0.30969, 0.30239, 0.29631],
[0.35521, 0.34242, 0.33154, 0.3219, 0.31948, 0.31096, 0.30424, 0.2984],
[0.35442, 0.34267, 0.33288, 0.32374, 0.32245, 0.31474, 0.30838, 0.30283],
[0.35384, 0.34286, 0.33386, 0.32507, 0.3246, 0.31745, 0.31135, 0.306],
[0.35338, 0.343, 0.33464, 0.32614, 0.3263, 0.31961, 0.31371, 0.30852],
[0.35301, 0.34312, 0.33526, 0.32698, 0.32766, 0.32132, 0.31558, 0.31052],
[0.35272, 0.34322, 0.33574, 0.32765, 0.32873, 0.32267, 0.31705, 0.31209],
[0.35246, 0.3433, 0.33617, 0.32822, 0.32965, 0.32383, 0.31831, 0.31344],
[0.35226, 0.34336, 0.33651, 0.32869, 0.3304, 0.32477, 0.31934, 0.31453],
[0.35207, 0.34342, 0.33681, 0.32911, 0.33106, 0.32561, 0.32025, 0.3155],
[0.35171, 0.34327, 0.33679, 0.32931, 0.3319, 0.32665, 0.32139, 0.31675],
[0.35128, 0.343, 0.33658, 0.32937, 0.33276, 0.32769, 0.32255, 0.31802],
[0.35086, 0.34274, 0.33637, 0.32943, 0.3336, 0.32872, 0.32368, 0.31927],
[0.35049, 0.34252, 0.33618, 0.32948, 0.33432, 0.32959, 0.32465, 0.32034],
[0.35016, 0.34231, 0.33602, 0.32953, 0.33498, 0.3304, 0.32554, 0.32132],
[0.34986, 0.34213, 0.33587, 0.32957, 0.33556, 0.3311, 0.32631, 0.32217],
[0.34959, 0.34196, 0.33573, 0.32961, 0.3361, 0.33176, 0.32704, 0.32296],
[0.34934, 0.34181, 0.33561, 0.32964, 0.33658, 0.33235, 0.32769, 0.32368],
[0.34912, 0.34167, 0.3355, 0.32967, 0.33701, 0.33288, 0.32827, 0.32432],
[0.34891, 0.34154, 0.33539, 0.3297, 0.33742, 0.33337, 0.32881, 0.32492]]

I have abstracted some of the repetitive methods into python functions. The function setup_helpers will construct the Heston model helpers and returns an array of these objects. The cost_function_generator is a method to set the cost function and will be used by the Scipy modules. The calibration_report lets us evaluate the quality of the fit. The setup_model method initializes the HestonModel and the AnalyticHestonEngine prior to calibration.

In [4]:

def setup_helpers(engine, expiration_dates, strikes, 
                  data, ref_date, spot, yield_ts, 
                  dividend_ts):
    heston_helpers = []
    grid_data = []
    for i, date in enumerate(expiration_dates):
        for j, s in enumerate(strikes):
            t = (date - ref_date )
            p = ql.Period(t, ql.Days)
            vols = data[i][j]
            helper = ql.HestonModelHelper(
                p, calendar, spot, s, 
                ql.QuoteHandle(ql.SimpleQuote(vols)),
                yield_ts, dividend_ts)
            helper.setPricingEngine(engine)
            heston_helpers.append(helper)
            grid_data.append((date, s))
    return heston_helpers, grid_data

def cost_function_generator(model, helpers,norm=False):
    def cost_function(params):
        params_ = ql.Array(list(params))
        model.setParams(params_)
        error = [h.calibrationError() for h in helpers]
        if norm:
            return np.sqrt(np.sum(np.abs(error)))
        else:
            return error
    return cost_function

def calibration_report(helpers, grid_data, detailed=False):
    avg = 0.0
    if detailed:
        print "%15s %25s %15s %15s %20s" % (
            "Strikes", "Expiry", "Market Value", 
             "Model Value", "Relative Error (%)")
        print "="*100
    for i, opt in enumerate(helpers):
        err = (opt.modelValue()/opt.marketValue() - 1.0)
        date,strike = grid_data[i]
        if detailed:
            print "%15.2f %25s %14.5f %15.5f %20.7f " % (
                strike, str(date), opt.marketValue(), 
                opt.modelValue(), 
                100.0*(opt.modelValue()/opt.marketValue() - 1.0))
        avg += abs(err)
    avg = avg*100.0/len(helpers)
    if detailed: print "-"*100
    summary = "Average Abs Error (%%) : %5.9f" % (avg)
    print summary
    return avg
    
def setup_model(_yield_ts, _dividend_ts, _spot, 
                init_condition=(0.02,0.2,0.5,0.1,0.01)):
    theta, kappa, sigma, rho, v0 = init_condition
    process = ql.HestonProcess(_yield_ts, _dividend_ts, 
                           ql.QuoteHandle(ql.SimpleQuote(_spot)), 
                           v0, kappa, theta, sigma, rho)
    model = ql.HestonModel(process)
    engine = ql.AnalyticHestonEngine(model) 
    return model, engine
summary= []

Comparing Different Calibration Methods

Solvers such as Levenberg-Marquardt find local minimas and are very sensitive to the initial conditions. Depending on the starting conditions for your solver, you could end up with a good set of parameters with good convergence or not so good set of parameters. We will look at two initial conditions for different solvers and see how the local minima solvers perform. We will compare this with differential evolution that looks for global minima.

We will setup the Heston model with two different initial conditions:

- theta, kappa, sigma, rho, v0 = (0.02, 0.2, 0.5, 0.1, 0.01)
- theta, kappa, sigma, rho, v0 = (0.07, 0.5, 0.1, 0.1, 0.1)

Local Solvers

Using QuantLib Levenberg-Marquardt Solver

As a first step, let's look at QuantLib's Levenberg-Marquardt solver. The initial condition considered is theta, kappa, sigma, rho, v0 = (0.02,0.2,0.5,0.1,0.01)

In [5]:

model1, engine1 = setup_model(
    yield_ts, dividend_ts, spot, 
    init_condition=(0.02,0.2,0.5,0.1,0.01))
heston_helpers1, grid_data1 = setup_helpers(
    engine1, expiration_dates, strikes, data, 
    calculation_date, spot, yield_ts, dividend_ts
)
initial_condition = list(model1.params())

In [6]:

%%time
lm = ql.LevenbergMarquardt(1e-8, 1e-8, 1e-8)
model1.calibrate(heston_helpers1, lm, 
                 ql.EndCriteria(500, 300, 1.0e-8,1.0e-8, 1.0e-8))
theta, kappa, sigma, rho, v0 = model1.params()
print "theta = %f, kappa = %f, sigma = %f, rho = %f, v0 = %f" % \
    (theta, kappa, sigma, rho, v0)
error = calibration_report(heston_helpers1, grid_data1)
summary.append(["QL LM1", error] + list(model1.params()))

theta = 0.125748, kappa = 7.915000, sigma = 1.887854, rho = -0.364942, v0 = 0.055397
Average Abs Error (%) : 3.015268051
CPU times: user 4.86 s, sys: 0 ns, total: 4.86 s
Wall time: 4.86 s

Methods like Levenberg-Marquardt solve for local minimas and do not search for global minimas. The solver is very sensitive to the initial conditions. Let's set a different set of initial conditions, and see what happens below. The initial condition considered is theta, kappa, sigma, rho, v0 = (0.07,0.5,0.1,0.1,0.1)

In [7]:

model1, engine1 = setup_model(
    yield_ts, dividend_ts, spot, 
    init_condition=(0.07,0.5,0.1,0.1,0.1))
heston_helpers1, grid_data1 = setup_helpers(
    engine1, expiration_dates, strikes, data, 
    calculation_date, spot, yield_ts, dividend_ts
)
initial_condition = list(model1.params())

In [8]:

%%time
lm = ql.LevenbergMarquardt(1e-8, 1e-8, 1e-8)
model1.calibrate(heston_helpers1, lm, 
                 ql.EndCriteria(500, 300, 1.0e-8,1.0e-8, 1.0e-8))
theta, kappa, sigma, rho, v0 = model1.params()
print "theta = %f, kappa = %f, sigma = %f, rho = %f, v0 = %f" % \
    (theta, kappa, sigma, rho, v0)
error = calibration_report(heston_helpers1, grid_data1)
summary.append(["QL LM2", error] + list(model1.params()))

theta = 0.084523, kappa = 0.000000, sigma = 0.132289, rho = -0.514278, v0 = 0.099928
Average Abs Error (%) : 11.007433024
CPU times: user 4.97 s, sys: 0 ns, total: 4.97 s
Wall time: 4.98 s

We see that the solver produces a 11% average of absolute error. This is not particularly great.

Using Scipy Levenberg-Marquardt Solver

Here we are going to try the same exercise but using Scipy. Scipy has far more optimization, minimization and root finding algorithms that are very robust. So by leveraging Scipy, we can take advantage of this rich set of options at hand.

In [9]:

model2, engine2 = setup_model(
    yield_ts, dividend_ts, spot, 
    init_condition=(0.02,0.2,0.5,0.1,0.01))
heston_helpers2, grid_data2 = setup_helpers(
    engine2, expiration_dates, strikes, data,
    calculation_date, spot, yield_ts, dividend_ts
)
initial_condition = list(model2.params())

In [10]:

%%time
cost_function = cost_function_generator(model2, heston_helpers2)
sol = root(cost_function, initial_condition, method='lm')
theta, kappa, sigma, rho, v0 = model2.params()
print "theta = %f, kappa = %f, sigma = %f, rho = %f, v0 = %f" % \
    (theta, kappa, sigma, rho, v0)
error = calibration_report(heston_helpers2, grid_data2)
summary.append(["Scipy LM1", error] + list(model2.params()))

theta = 0.125747, kappa = 7.915687, sigma = 1.887934, rho = -0.364944, v0 = 0.055394
Average Abs Error (%) : 3.015252651
CPU times: user 4.68 s, sys: 12 ms, total: 4.69 s
Wall time: 4.65 s

The solution for this particular case seems to be fairly robust. Both solvers (QuantLib and Scipy) seem to have landed on more or less the same solution for this particular initial condition. Let's see how Scipy does for the second initial condition considered above - theta, kappa, sigma, rho, v0 = (0.07,0.5,0.1,0.1,0.1)

In [11]:

model2, engine2 = setup_model(
    yield_ts, dividend_ts, spot,
    init_condition=(0.07,0.5,0.1,0.1,0.1))
heston_helpers2, grid_data2 = setup_helpers(
    engine2, expiration_dates, strikes, data,
    calculation_date, spot, yield_ts, dividend_ts
)
initial_condition = list(model2.params())

In [12]:

%%time
cost_function = cost_function_generator(model2, heston_helpers2)
sol = root(cost_function, initial_condition, method='lm')
theta, kappa, sigma, rho, v0 = model2.params()
print "theta = %f, kappa = %f, sigma = %f, rho = %f, v0 = %f" % \
    (theta, kappa, sigma, rho, v0)
error = calibration_report(heston_helpers2, grid_data2)
summary.append(["Scipy LM2", error] + list(model2.params()))

theta = 0.048184, kappa = -0.548903, sigma = 0.197958, rho = -0.999547, v0 = 0.090571
Average Abs Error (%) : 7.019499509
CPU times: user 20.7 s, sys: 56 ms, total: 20.7 s
Wall time: 20.5 s

For this particular case, Scipy solver has performed significantly better. It would be inappropriate to make loud claims about Scipy's superiority based on one observation. Perhaps this calls for a more detailed study for later.

Using Least Squares Method

If you want to use a simpler approach like least squares, you can do that with Scipy. Here is how you would use it.

In [13]:

from scipy.optimize import least_squares

In [14]:

model3, engine3 = setup_model(
    yield_ts, dividend_ts, spot, 
    init_condition=(0.02,0.2,0.5,0.1,0.01))
heston_helpers3, grid_data3 = setup_helpers(
    engine3, expiration_dates, strikes, data,
    calculation_date, spot, yield_ts, dividend_ts
)
initial_condition = list(model3.params())

In [15]:

%%time
cost_function = cost_function_generator(model3, heston_helpers3)
sol = least_squares(cost_function, initial_condition)
theta, kappa, sigma, rho, v0 = model3.params()
print "theta = %f, kappa = %f, sigma = %f, rho = %f, v0 = %f" % \
    (theta, kappa, sigma, rho, v0)
error = calibration_report(heston_helpers3, grid_data3)
summary.append(["Scipy LS1", error] + list(model3.params()))

theta = 0.125747, kappa = 7.915814, sigma = 1.887949, rho = -0.364944, v0 = 0.055394
Average Abs Error (%) : 3.015251175
CPU times: user 4.54 s, sys: 8 ms, total: 4.55 s
Wall time: 4.55 s

With the second initial condition:

In [16]:

model3, engine3 = setup_model(
    yield_ts, dividend_ts, spot, 
    init_condition=(0.07,0.5,0.1,0.1,0.1))
heston_helpers3, grid_data3 = setup_helpers(
    engine3, expiration_dates, strikes, data,
    calculation_date, spot, yield_ts, dividend_ts
)
initial_condition = list(model3.params())

In [17]:

%%time
cost_function = cost_function_generator(model3, heston_helpers3)
sol = least_squares(cost_function, initial_condition)
theta, kappa, sigma, rho, v0 = model3.params()
print "theta = %f, kappa = %f, sigma = %f, rho = %f, v0 = %f" % \
    (theta, kappa, sigma, rho, v0)
error = calibration_report(heston_helpers3, grid_data3)
summary.append(["Scipy LS2", error] + list(model3.params()))

theta = 3.136774, kappa = 0.000005, sigma = -0.000245, rho = -0.000010, v0 = 1.597904
Average Abs Error (%) : 5.096414042
CPU times: user 28.1 s, sys: 16 ms, total: 28.2 s
Wall time: 28.1 s

Global Solvers

Using Differential Evolution

The above methods are more suited to finding local minimas. One method that makes an attempt at searching for global minima is the differential evolution. Both QuantLib and Scipy have implementations of this method. Scipy however has a lot more bells and whistles to tune and calibrate the methodology. Let's take a look at the Scipy's differential_evolution methodology.

In [18]:

from scipy.optimize import differential_evolution

In [19]:

model4, engine4 = setup_model(yield_ts, dividend_ts, spot)
heston_helpers4, grid_data4 = setup_helpers(
    engine4, expiration_dates, strikes, data,
    calculation_date, spot, yield_ts, dividend_ts
)
initial_condition = list(model4.params())
bounds = [(0,1),(0.01,15), (0.01,1.), (-1,1), (0,1.0) ]

In [20]:

%%time
cost_function = cost_function_generator(
    model4, heston_helpers4, norm=True)
sol = differential_evolution(cost_function, bounds, maxiter=100)
theta, kappa, sigma, rho, v0 = model4.params()
print "theta = %f, kappa = %f, sigma = %f, rho = %f, v0 = %f" % \
    (theta, kappa, sigma, rho, v0)
error = calibration_report(heston_helpers4, grid_data4)
summary.append(["Scipy DE1", error] + list(model4.params()))

theta = 0.123221, kappa = 5.012199, sigma = 0.950309, rho = -0.570340, v0 = 0.078440
Average Abs Error (%) : 2.859113482
CPU times: user 1min 39s, sys: 144 ms, total: 1min 39s
Wall time: 1min 39s

In the above example, I am setting the variable maxiter in order to limit the time taken. In production scenarios, you may want to try a larger number or not provide any value and default to 1000. This can help search a larger area of the parameter space.

In [21]:

model4, engine4 = setup_model(yield_ts, dividend_ts, spot)
heston_helpers4, grid_data4 = setup_helpers(
    engine4, expiration_dates, strikes, data,
    calculation_date, spot, yield_ts, dividend_ts
)
initial_condition = list(model4.params())
bounds = [(0,1),(0.01,15), (0.01,1.), (-1,1), (0,1.0) ]

In [22]:

%%time
cost_function = cost_function_generator(
    model4, heston_helpers4, norm=True)
sol = differential_evolution(cost_function, bounds, maxiter=100)
theta, kappa, sigma, rho, v0 = model4.params()
print "theta = %f, kappa = %f, sigma = %f, rho = %f, v0 = %f" % \
    (theta, kappa, sigma, rho, v0)
error = calibration_report(heston_helpers4, grid_data4)
summary.append(["Scipy DE2", error] + list(model4.params()))

theta = 0.122251, kappa = 4.996804, sigma = 0.849266, rho = -0.637706, v0 = 0.079484
Average Abs Error (%) : 2.876087363
CPU times: user 1min 57s, sys: 200 ms, total: 1min 57s
Wall time: 1min 57s

Basin Hopping Algorithm

Here we will use the Basin Hopping (annealing like) method to solve for the parameters. A couple things to make note here. The Basin Hopping method works best when used wiht a minimizer. Here I played with various minimizers and finally decided to use something that supports bounds checking. Without bounds checking, I often ended with nan and did not have a meaningful solution in the end.

I have chosen bounds based on a very basic reasoning. One needs careful reasoning to use appropriate bounds for the problem at hand.

In [23]:

from scipy.optimize import basinhopping

In [24]:

class MyBounds(object):
     def __init__(self, xmin=[0.,0.01,0.01,-1,0], xmax=[1,15,1,1,1.0] ):
         self.xmax = np.array(xmax)
         self.xmin = np.array(xmin)
     def __call__(self, **kwargs):
         x = kwargs["x_new"]
         tmax = bool(np.all(x <= self.xmax))
         tmin = bool(np.all(x >= self.xmin))
         return tmax and tmin
bounds = [(0,1),(0.01,15), (0.01,1.), (-1,1), (0,1.0) ]

In [25]:

model5, engine5 = setup_model(
    yield_ts, dividend_ts, spot,
    init_condition=(0.02,0.2,0.5,0.1,0.01))
heston_helpers5, grid_data5 = setup_helpers(
    engine5, expiration_dates, strikes, data,
    calculation_date, spot, yield_ts, dividend_ts
)
initial_condition = list(model5.params())

In [26]:

%%time
mybound = MyBounds()
minimizer_kwargs = {"method": "L-BFGS-B", "bounds": bounds }
cost_function = cost_function_generator(
    model5, heston_helpers5, norm=True)
sol = basinhopping(cost_function, initial_condition, niter=5,
                   minimizer_kwargs=minimizer_kwargs,
                   stepsize=0.005,
                   accept_test=mybound,
                   interval=10)
theta, kappa, sigma, rho, v0 = model5.params()
print "theta = %f, kappa = %f, sigma = %f, rho = %f, v0 = %f" % \
    (theta, kappa, sigma, rho, v0)
error = calibration_report(heston_helpers5, grid_data5)
summary.append(["Scipy BH1", error] + list(model5.params()))

theta = 0.123455, kappa = 5.074067, sigma = 0.995095, rho = -0.562106, v0 = 0.079009
Average Abs Error (%) : 2.850972273
CPU times: user 2min 15s, sys: 456 ms, total: 2min 16s
Wall time: 2min 14s

In [27]:

model5, engine5 = setup_model(
    yield_ts, dividend_ts, spot,
    init_condition=(0.07,0.5,0.1,0.1,0.1))
heston_helpers5, grid_data5 = setup_helpers(
    engine5, expiration_dates, strikes, data,
    calculation_date, spot, yield_ts, dividend_ts
)
initial_condition = list(model5.params())

In [28]:

%%time
mybound = MyBounds()
minimizer_kwargs = {"method": "L-BFGS-B", "bounds": bounds}
cost_function = cost_function_generator(
    model5, heston_helpers5, norm=True)
sol = basinhopping(cost_function, initial_condition, niter=5,
                   minimizer_kwargs=minimizer_kwargs,
                   stepsize=0.005,
                   accept_test=mybound,
                   interval=10)
theta, kappa, sigma, rho, v0 = model5.params()
print "theta = %f, kappa = %f, sigma = %f, rho = %f, v0 = %f" % \
    (theta, kappa, sigma, rho, v0)
error = calibration_report(heston_helpers5, grid_data5)
summary.append(["Scipy BH2", error] + list(model5.params()))

theta = 0.123403, kappa = 4.791319, sigma = 0.904397, rho = -0.593711, v0 = 0.079215
Average Abs Error (%) : 2.863355647
CPU times: user 1min 50s, sys: 332 ms, total: 1min 50s
Wall time: 1min 49s

Summary

Here is a summary of all the results with the calibration error overall, and the respective parameters. All the local minima methods give parameters that are very different based on the initial condition that we start with. This is different in contrary with the global minimization methods that all end up in more or less the same proximity of each other.

The global solvers such as Differential Evolution and Basin Hopping are capable of finding the global minima and it is sometimes a question of computation resources. Here, I have lower "iterations" set for these routines for faster solving. Even with such a short threshold, we get fairly good solution set. I think it is premature to compare the effectiveness of different global solvers just based on the results here. The scipy optimize package has detailed documentation with various tuning parameters. I haven't exploited the nuances much, and is left as an exercise for the reader.

Hope you find this useful!

In [29]:

from pandas import DataFrame
DataFrame(
    summary,
    columns=["Name", "Avg Abs Error","Theta", "Kappa", "Sigma", "Rho", "V0"],
    index=['']*len(summary))

Out[29]:

Name	Avg Abs Error	Theta	Kappa	Sigma	Rho	V0
QL LM1	3.015268	0.125748	7.915000e+00	1.887854	-0.364942	0.055397
QL LM2	11.007433	0.084523	1.625740e-08	0.132289	-0.514278	0.099928
Scipy LM1	3.015253	0.125747	7.915687e+00	1.887934	-0.364944	0.055394
Scipy LM2	7.019500	0.048184	-5.489029e-01	0.197958	-0.999547	0.090571
Scipy LS1	3.015251	0.125747	7.915814e+00	1.887949	-0.364944	0.055394
Scipy LS2	5.096414	3.136774	4.896844e-06	-0.000245	-0.000010	1.597904
Scipy DE1	2.859113	0.123221	5.012199e+00	0.950309	-0.570340	0.078440
Scipy DE2	2.876087	0.122251	4.996804e+00	0.849266	-0.637706	0.079484
Scipy BH1	2.850972	0.123455	5.074067e+00	0.995095	-0.562106	0.079009
Scipy BH2	2.863356	0.123403	4.791319e+00	0.904397	-0.593711	0.079215

Valuing Bonds with Credit Spreads in QuantLib Python

Goutham Balaraman — Tue, 26 Jul 2016 00:00:00 -0700

In an earlier example on pricing fixed rate bonds I demonstrated how to construct and value bonds using the given yield curve. In this example, let us take a look at valuing bonds with credit spreads. We will show how to add credit spreads to the give yield curve using different approaches.

As usual, let us start by importing the QuantLib library and pick a valuation date and set the calculation instance evaluation date.

In [1]:

import QuantLib as ql
calc_date = ql.Date(26, 7, 2016)
ql.Settings.instance().evaluationDate = calc_date

For simplicity, let us imagine that the treasury yield curve is flat. This makes it easier to construct the yield curve easily. This also allows us to directly shock the yield curve, and provides a validation for the more general treatment of shocks on yield curve.

In [2]:

flat_rate = ql.SimpleQuote(0.0015)
rate_handle = ql.QuoteHandle(flat_rate)
day_count = ql.Actual360()
calendar = ql.UnitedStates()
ts_yield = ql.FlatForward(calc_date, rate_handle, day_count)
ts_handle = ql.YieldTermStructureHandle(ts_yield)

Now let us construct the bond itself. We do that by first constructing the schedule, and then passing the schedule into the bond.

In [3]:

issue_date = ql.Date(15, 7, 2016)
maturity_date = ql.Date(15, 7, 2021)
tenor = ql.Period(ql.Semiannual)
calendar = ql.UnitedStates()
bussiness_convention = ql.Unadjusted
date_generation = ql.DateGeneration.Backward
month_end = False
schedule = ql.Schedule (issue_date, maturity_date, 
                        tenor, calendar, 
                        bussiness_convention,
                        bussiness_convention, 
                        date_generation, 
                        month_end)

In [4]:

settlement_days = 2
day_count = ql.Thirty360()
coupon_rate = .03
coupons = [coupon_rate]

# Now lets construct the FixedRateBond
settlement_days = 0
face_value = 100
fixed_rate_bond = ql.FixedRateBond(
    settlement_days, 
    face_value, 
    schedule, 
    coupons, 
    day_count)

Now that we have the fixed_rate_bond object, we can create a DiscountingBondEngine and value the bond.

In [5]:

bond_engine = ql.DiscountingBondEngine(ts_handle)
fixed_rate_bond.setPricingEngine(bond_engine)
fixed_rate_bond.NPV()

Out[5]:

114.18461651948999

So far, we have valued the bond under the treasury yield curve and have not incorporated the credit spreads. Let us assume that the market prices this bond with a 50BP spread on top of the treasury yield curve. Now we can, in this case, directly shock the flat_rate used in the yield term structure. Let us see what the value is:

In [6]:

flat_rate.setValue(0.0065)
fixed_rate_bond.NPV()

Out[6]:

111.5097766266561

Above we shocked the flat_rate and since the yield term structure is an Observer observing the Observable flat_rate, we could just shock the rate, and QuantLib behind the scenes recalculates all the Observers. Though, this approach is not always viable, in cases such as a bootstrapped bond curve. So let us look at two different approaches that can be used. Before we do that, we need to reset the flat_rate back to what it was.

In [7]:

flat_rate.setValue(0.0015)
fixed_rate_bond.NPV()

Out[7]:

114.18461651948999

Parallel Shift of the Yield Curve

The whole yield curve can be shifted up and down, and the bond revalued with the help of the ZeroSpreadedTermStructure. The constructor takes the yield curve and the spread as argument.

In [8]:

spread1 = ql.SimpleQuote(0.0050)
spread_handle1 = ql.QuoteHandle(spread1)
ts_spreaded1 = ql.ZeroSpreadedTermStructure(ts_handle, spread_handle1)
ts_spreaded_handle1 = ql.YieldTermStructureHandle(ts_spreaded1)

bond_engine = ql.DiscountingBondEngine(ts_spreaded_handle1)
fixed_rate_bond.setPricingEngine(bond_engine)

# Finally the price
fixed_rate_bond.NPV()

Out[8]:

111.50977662665609

Once we have constructed the spreaded term structure, it is rather easy to value for other spreads. All we need to do is change the SimpleQuote object spread1 here.

In [9]:

spread1.setValue(0.01)
fixed_rate_bond.NPV()

Out[9]:

108.89999943320038

Non-Parallel Shift of the Yield Curve

The above method allows only for parallel shift of the yield curve. The SpreadedLinearZeroInterpolatedTermStructure class allows for non parallel shock. First, let us mimic a parallel shift using this class. For the constructor, we need to pass the yield term structure that we wish to shift, and the a list of spreads and a list of the corresponding dates.

In [10]:

spread21 = ql.SimpleQuote(0.0050)
spread22 = ql.SimpleQuote(0.0050)
start_date = calc_date
end_date = calendar.advance(start_date, ql.Period(50, ql.Years))
ts_spreaded2 = ql.SpreadedLinearZeroInterpolatedTermStructure(
    ts_handle,
    [ql.QuoteHandle(spread21), ql.QuoteHandle(spread22)],
    [start_date, end_date]
)
ts_spreaded_handle2 = ql.YieldTermStructureHandle(ts_spreaded2)

bond_engine = ql.DiscountingBondEngine(ts_spreaded_handle2)
fixed_rate_bond.setPricingEngine(bond_engine)

# Finally the price
fixed_rate_bond.NPV()

Out[10]:

111.50977662665609

Here, once again we can change the value of spread2 to value for other shocks.

In [11]:

spread21.setValue(0.01)
spread22.setValue(0.01)
fixed_rate_bond.NPV()

Out[11]:

108.89999943320038

We can easily do non-parallel shifts just by shocking one end.

In [12]:

spread21.setValue(0.005)
spread22.setValue(0.01)
fixed_rate_bond.NPV()

Out[12]:

111.25358792334083

The SpreadedLinearZeroInterpolatedTermStructure is a very powerful class and can be used to implement key-rate duration calculations.

Announcing qtk for QuantLib Python

Goutham Balaraman — Wed, 20 Jul 2016 00:00:00 -0700

I have blogged about using QuantLib Python in this tutorial series. Learning QuantLib Python involves a knowledge of the various syntax and paying a lot of attention to detail such as the various conventions. I have started to work on the package qtk that adds a layer on top of QuantLib python by making it data driven. What do I mean by that?

The qtk package relies on data in the form of Python dict and then uses this data to form QuantLib objects behind the scenes. Lets take a look at a small example that replicates my example on bond modeling in QuantLib Python

In [1]:

from qtk import Controller, Field as F, Template as T

The Template class directly maps to the different kinds of QuantLib objects that are supported. For instance the Template.TS_YIELD_ZERO corresponds to the zero yield term structure. The Field class has all the properties that are needed by the various Templates. The Controller class is what does all the processing to create QuantLib objects behind the scenes. Let us look at a minimal example below, and then explain the different features in detail.

In [2]:

data = [
    {
        F.LIST_OF_DATES.id: ["1/15/2015", "7/15/2015", "1/15/2016"],
        F.LIST_OF_ZERO_RATES.id: [0.0, 0.005, 0.007],
        F.DISCOUNT_BASIS.id: "30/360",
        F.DISCOUNT_CALENDAR.id: "UnitedStates",
        F.COMPOUNDING.id: "Compounded",
        F.COMPOUNDING_FREQ.id: "Annual",
        F.CURRENCY.id: "USD",
        F.TEMPLATE.id: T.TS_YIELD_ZERO.id,
        F.OBJECT_ID.id: "USD.Zero.Curve"},
    {
        F.OBJECT_ID.id: "BondEngine",
        F.DISCOUNT_CURVE.id: "->USD.Zero.Curve",
        F.TEMPLATE.id: T.ENG_BOND_DISCOUNTING.id},
    {
        F.ASOF_DATE.id: '2016-01-15',
        F.COUPON.id: 0.06,
        F.COUPON_FREQ.id: "Semiannual",
        F.CURRENCY.id: 'USD',
        F.PAYMENT_BASIS.id: '30/360',
        F.ISSUE_DATE.id: '2015-01-15',
        F.MATURITY_DATE.id: '2016-01-15',
        F.ACCRUAL_CALENDAR.id: "UnitedStates",
        F.ACCRUAL_DAY_CONVENTION.id: "Unadjusted",
        F.DATE_GENERATION.id: "Backward",
        F.END_OF_MONTH.id: False,
        F.OBJECT_ID.id: "USD.TBond",
        F.PRICING_ENGINE.id: "->BondEngine",
        F.TEMPLATE.id: T.INST_BOND_TBOND.id}
]

res = Controller(data)
asof_date = "1/15/2015"

ret = res.process(asof_date)
tbond = res.object("USD.TBond")
print tbond.NPV()

105.276539925

Now let us disect the code above to understand the specifics. The Controller class takes a list of dict as input, provided here by the variable data. In the above example, we have used qtk mnemonics such as Field.TEMPLATE.id instead of directly using the text "Template". The content of data is as shown below:

In [3]:

data

Out[3]:

[{'Compounding': 'Compounded',
  'CompoundingFrequency': 'Annual',
  'Currency': 'USD',
  'DiscountBasis': '30/360',
  'DiscountCalendar': 'UnitedStates',
  'ListOfDate': ['1/15/2015', '7/15/2015', '1/15/2016'],
  'ListOfZeroRate': [0.0, 0.005, 0.007],
  'ObjectId': 'USD.Zero.Curve',
  'Template': 'TermStructure.Yield.ZeroCurve'},
 {'DiscountCurve': '->USD.Zero.Curve',
  'ObjectId': 'BondEngine',
  'Template': 'Engine.Bond.Discounting'},
 {'AccrualCalendar': 'UnitedStates',
  'AccrualDayConvention': 'Unadjusted',
  'AsOfDate': '2016-01-15',
  'Coupon': 0.06,
  'CouponFrequency': 'Semiannual',
  'Currency': 'USD',
  'DateGeneration': 'Backward',
  'EndOfMonth': False,
  'IssueDate': '2015-01-15',
  'MaturityDate': '2016-01-15',
  'ObjectId': 'USD.TBond',
  'PaymentBasis': '30/360',
  'PricingEngine': '->BondEngine',
  'Template': 'Instrument.Bond.TreasuryBond'}]

Every dict corresponds to an object, with the key value pairs being the attributes required in the creation of that object. The value of the key Template denotes what object is created. So for instance, in the above example the value Engine.Bond.Discounting denotes the DiscountingBondEngine. In the above data, every dict has a key ObjectId which is a unique name required for every object. Objects can be cross referenced by using the special prefix "->" along with the ObjectId. You can see this usage in the last dict where the PricingEngine references the BondEngine object, and the BondEngine object references the USD.Zero.Curve yield term structure.

Once the data is prepared, the API call to process is extremely simple because qtk does the major brunt of interfacing with QuantLib.

In [4]:

res = Controller(data)
asof_date = "1/15/2015"

ret = res.process(asof_date)
tbond = res.object("USD.TBond")
print tbond.NPV()

105.276539925

The Controller accepts the data and the process method processes as of the calculation date. Once processed, the QuantLib objects can be accessed and queried for output.

This project is alpha and has a few templates written as of now. Would love to hear your thoughts. Checkout the qtk project page to get involved.

QuantLib Python Notebooks On Docker

Goutham Balaraman — Sat, 02 Jul 2016 00:00:00 -0700

The typical way to get QuantLib python running in your computer is by compiling QuantLib and the SWIG bindings in your computer. I have heard from some of my blog readers that they run into issues with getting QuantLib-Python running in their system. Here I will show how you can easily get started on QuantLib-Python using Docker.

Docker

Docker is a platform for containerization of software. In other words, Docker lets one to wrap an application in a complete ecosystem needed to run the application (aka image), such as system libraries and filesystem. This means that irrespective of what operating system your computer runs, a Docker image will run reliably on the Docker containers. Docker containers are very lightweight, and it starts up in a matter of couple of seconds.

Docker and QuantLib

Every now and then, I would run into conflicts between the development version of QuantLib and the release version. So in order to not have such conflicts, Docker containers are a perfect solution. Every image is isolated and it is easy to load a specific version and not have any conflicts. In order to keep my Docker images minimal, I use Alpine Linux as a base instead of Ubuntu.

Getting Up and Running

In order to get started on using Docker, you need to first install Docker. The first step is to get Docker itself. You can get the Docker client by going to getting started page.

Once you have done the installation, you can get quantlib-python by doing the following on a shell:

docker pull gbalaraman/quantlib-notebook

This is going to pull the quantlib-notebook image from Docker Hub. This process takes a few seconds. Think of this as the same as downloading an installer for a software.

Once you have the image downloaded, you can run quantlib-python notebooks by doing the following:

docker run -d -p 8888:8888 --mount src=<your notebook folder>,target=/home/notebooks,type=bind --name ql_notebook gbalaraman/quantlib-notebook

If you are using latest docker images published by Luigi, you should mount to /notebooks folder on the container. So in all it would look like:

docker run -d -p 8888:8888 --mount src=<your notebook folder>,target=/notebooks,type=bind --name ql_notebook2 lballabio/quantlib-notebook

Those using older docker versions, you had to do this:

docker run -d -p 8888:8888 -v <your notebook folder>:/home/notebooks/ --name ql_notebook gbalaraman/quantlib-notebook

The above command runs the image "gbalaraman/quantlib-notebook" as a daemon (-d) and will forward the port 8888 in the container to the port 8888 in localhost (-p 8888:8888). We are calling the container where this image runs as ql_notebook (--name ql_notebook). In "-v :/home/notebooks/", you should put the folder in you machine where you want the notebooks to be stored. This takes care of mapping your local folder to the volume in the container where the notebook will run from. You can visit http://localhost:8888 to access the QuntLib python notebook environment to play with. This image also has a few other packages included in it, such as numpy, pandas, scipy, matplolib and seaborn. This is most of all you would need to run the notebooks that I publish as part of my QuantLib python tutorial series.

When you are done you can stop the Docker container by using the command:

docker stop ql_notebook

This is equivalent to shutting down the notebook. If you want to start docker again, this time around you can start it by saying:

docker start ql_notebook

Give this a try and drop a word about your experience with it. You can also take a look at Luigi Ballabio's presentation on QuantLib using Docker for another take on this subject.

Valuing Interest Rate Caps and Floors Using QuantLib Python

Goutham Balaraman — Thu, 23 Jun 2016 00:00:00 -0700

In this post, I will walk you through a simple example of valuing caps. I want to talk about two specific cases:

Value caps given a constant volatility
Value caps given a cap volatility surface

Caps, as you might know, can be valued as a sum of caplets. The value of each caplet is determined by the Black formula. In practice, each caplet would have a different volatility. Meaning, a caplet that is in the near term can have a different volotility profile compared to the caplet that is far away in tenor. Similarly caplet volatilities differ with the strike as well.

In [1]:

import QuantLib as ql

In [2]:

calc_date = ql.Date(14, 6, 2016)
ql.Settings.instance().evaluationDate = calc_date

Constant Volatility

Let us start by constructing different componets required in valuing the caps. The components that we would need are:

interest rate term structure for discounting
interest rate term structure for the floating leg
construction of the cap
the pricing engine to value caps using the Black formula

For simplicity, we will construct only one interest rate term structure here, and assume that the discounting and the floating leg is referenced by the same. Below the term structure of interest rates is constructed from a set of zero rates.

In [3]:

dates = [ql.Date(14,6,2016), ql.Date(14,9,2016), 
         ql.Date(14,12,2016), ql.Date(14,6,2017),
         ql.Date(14,6,2019), ql.Date(14,6,2021),
         ql.Date(15,6,2026), ql.Date(16,6,2031),
         ql.Date(16,6,2036), ql.Date(14,6,2046)
         ]
yields = [0.000000, 0.006616, 0.007049, 0.007795,
          0.009599, 0.011203, 0.015068, 0.017583,
          0.018998, 0.020080]
day_count = ql.ActualActual()
calendar = ql.UnitedStates()
interpolation = ql.Linear()
compounding = ql.Compounded
compounding_frequency = ql.Annual

term_structure = ql.ZeroCurve(dates, yields, day_count, calendar, 
                       interpolation, compounding, compounding_frequency)
ts_handle = ql.YieldTermStructureHandle(term_structure)

As a next step, lets construct the cap itself. In order to do that, we start by constructing the Schedule object to project the cashflow dates.

In [4]:

start_date = ql.Date(14, 6, 2016)
end_date = ql.Date(14, 6 , 2026)
period = ql.Period(3, ql.Months)
calendar = ql.UnitedStates()
buss_convention = ql.ModifiedFollowing
rule = ql.DateGeneration.Forward
end_of_month = False

schedule = ql.Schedule(start_date, end_date, period,
                       calendar, buss_convention, buss_convention, 
                       rule, end_of_month)

Now that we have the schedule, we construct the USDLibor index. Below, you can see that I use addFixing method to provide a fixing date for June 10, 2016. According the schedule constructed, the start date of the cap is June 14, 2016, and there is a 2 bussiness day settlement lag (meaning June 10 reference date) embedded in the USDLibor definition. So in order to set the rate for the accrual period, the rate is obtained from the fixing data provided. For all future dates, the libor rates are automatically inferred using the forward rates provided by the given interest rate term structure.

In [5]:

ibor_index = ql.USDLibor(ql.Period(3, ql.Months), ts_handle)
ibor_index.addFixing(ql.Date(10,6,2016), 0.0065560)

ibor_leg = ql.IborLeg([1000000], schedule, ibor_index)

Now that we have all the required pieces, the Cap can be constructed by passing the ibor_leg and the strike information. Constructing a floot is done through the Floor class. The BlackCapFloorEngine can be used to price the cap with constant volatility as shown below.

In [6]:

strike = 0.02
cap = ql.Cap(ibor_leg, [strike])

vols = ql.QuoteHandle(ql.SimpleQuote(0.547295))
engine = ql.BlackCapFloorEngine(ts_handle, vols)

cap.setPricingEngine(engine)
print cap.NPV()

54369.8580629

Using Volatility Surfaces

In the above exercise, we used a constant volatility value. In practice, one needs to strip the market quoted capfloor volatilities to infer the volatility of each and every caplet. QuantLib provides excellent tools in order to do that. Let us assume the following dummy data represents the volatility surface quoted by the market. I have the various strikes, expiries, and the volatility quotes in percentage format. I take the raw data and create a Matrix in order to construct the volatility surface.

In [7]:

strikes = [0.01,0.015, 0.02]
expiries = [ql.Period(i, ql.Years) for i in range(1,11)] + [ql.Period(12, ql.Years)]
vols = ql.Matrix(len(expiries), len(strikes))
data = [[47.27, 55.47, 64.07, 70.14, 72.13, 69.41, 72.15, 67.28, 66.08, 68.64, 65.83],
   [46.65,54.15,61.47,65.53,66.28,62.83,64.42,60.05,58.71,60.35,55.91],
   [46.6,52.65,59.32,62.05,62.0,58.09,59.03,55.0,53.59,54.74,49.54]
   ]

for i in range(vols.rows()):
    for j in range(vols.columns()):
        vols[i][j] = data[j][i]/100.0

The CapFloorTermVolSurface offers a way to store the capfloor volatilities. These are however CapFloor volatilities, and not the volatilities of the individual options.

In [8]:

calendar = ql.UnitedStates()
bdc = ql.ModifiedFollowing
daycount = ql.Actual365Fixed()
settlement_days = 2
capfloor_vol = ql.CapFloorTermVolSurface(settlement_days, calendar, bdc, expiries, strikes, vols, daycount)

The OptionletStripper1 class lets you to strip the individual caplet/floorlet volatilities from the capfloor volatilities. We have to 'jump' some hoops here to make it useful for pricing. The OptionletStripper1 class does not allow you to be consumed directly by a pricing engine. The StrippedOptionletAdapter takes the stripped optionlet volatilities, and creates a term structure of optionlet volatilities. We then wrap that into a handle using OptionletVolatilityStructureHandle.

In [9]:

optionlet_surf = ql.OptionletStripper1(capfloor_vol, ibor_index)
ovs_handle = ql.OptionletVolatilityStructureHandle(
    ql.StrippedOptionletAdapter(optionlet_surf)
)

Below, we visulaize the capfloor volatility surface, and the optionlet volatility suface for a fixed strike.

In [10]:

import matplotlib.pyplot as plt
import numpy as np
%matplotlib inline

In [11]:

tenors = np.arange(0,10,0.25)
strike = 0.015
capfloor_vols = [capfloor_vol.volatility(t, strike) for t in tenors]
opionlet_vols = [ovs_handle.volatility(t, strike) for t in tenors]

plt.plot(tenors, capfloor_vols, "--", label="CapFloor Vols")
plt.plot(tenors, opionlet_vols,"-", label="Optionlet Vols")
plt.legend(bbox_to_anchor=(0.5, 0.25))

Out[11]:

<matplotlib.legend.Legend at 0x894efd0>

The BlackCapFloorEngine can accept the optionlet volatility surface in order to price the caps or floors.

In [12]:

engine2 = ql.BlackCapFloorEngine(ts_handle, ovs_handle)
cap.setPricingEngine(engine2)
print cap.NPV()

54384.928315

The QuantLib C++ class allow for one to view the projected cashflows in terms of individual caplets. I just realized that the python extension does not have this feature added to it. Will give a PR one of these days and update this post.

Hope you find this useful.

QuantLib Python Cookbook Announcement

Goutham Balaraman — Wed, 15 Jun 2016 00:00:00 -0700

Hello everybody! Luigi Ballabio and I have published "QuantLib Python Cookbook" on Leanpub. This book is a collection of Luigi's youtube videos and my blog posts tutorials on QuantLib python.

This book gives an introduction to various QuantLib concepts such as interest rate curve construction, calibration and simulation of interest and equity models, and finally valuation of bonds and equities. This book is meant to be a resource for the QuantLib community at large.

Modeling Volatility Smile and Heston Model Calibration Using QuantLib Python

Goutham Balaraman — Thu, 19 May 2016 00:00:00 -0700

European options on an equity underlying such as an index (S&P 500) or a stock (AMZN) trade for different combinations of strikes and maturities. It turns out that the Black-Scholes implied volatility for these options with different maturities and strikes is not the same. The fact that the implied volatility varies with strike is often referred in the market as having a smile.

In [1]:

import QuantLib as ql
import math

First let us define some of the basic data conventions such as the day_count, calendar etc.

In [2]:

day_count = ql.Actual365Fixed()
calendar = ql.UnitedStates()

calculation_date = ql.Date(6, 11, 2015)

spot = 659.37
ql.Settings.instance().evaluationDate = calculation_date

dividend_yield = ql.QuoteHandle(ql.SimpleQuote(0.0))
risk_free_rate = 0.01
dividend_rate = 0.0
flat_ts = ql.YieldTermStructureHandle(
    ql.FlatForward(calculation_date, risk_free_rate, day_count))
dividend_ts = ql.YieldTermStructureHandle(
    ql.FlatForward(calculation_date, dividend_rate, day_count))

Following is a sample matrix of volatility quote by exipiry and strike. The volatilities are log-normal volatilities and can be interpolated to construct the implied volatility surface.

In [3]:

expiration_dates = [ql.Date(6,12,2015), ql.Date(6,1,2016), ql.Date(6,2,2016),
                    ql.Date(6,3,2016), ql.Date(6,4,2016), ql.Date(6,5,2016), 
                    ql.Date(6,6,2016), ql.Date(6,7,2016), ql.Date(6,8,2016),
                    ql.Date(6,9,2016), ql.Date(6,10,2016), ql.Date(6,11,2016), 
                    ql.Date(6,12,2016), ql.Date(6,1,2017), ql.Date(6,2,2017),
                    ql.Date(6,3,2017), ql.Date(6,4,2017), ql.Date(6,5,2017), 
                    ql.Date(6,6,2017), ql.Date(6,7,2017), ql.Date(6,8,2017),
                    ql.Date(6,9,2017), ql.Date(6,10,2017), ql.Date(6,11,2017)]
strikes = [527.50, 560.46, 593.43, 626.40, 659.37, 692.34, 725.31, 758.28]
data = [
[0.37819, 0.34177, 0.30394, 0.27832, 0.26453, 0.25916, 0.25941, 0.26127],
[0.3445, 0.31769, 0.2933, 0.27614, 0.26575, 0.25729, 0.25228, 0.25202],
[0.37419, 0.35372, 0.33729, 0.32492, 0.31601, 0.30883, 0.30036, 0.29568],
[0.37498, 0.35847, 0.34475, 0.33399, 0.32715, 0.31943, 0.31098, 0.30506],
[0.35941, 0.34516, 0.33296, 0.32275, 0.31867, 0.30969, 0.30239, 0.29631],
[0.35521, 0.34242, 0.33154, 0.3219, 0.31948, 0.31096, 0.30424, 0.2984],
[0.35442, 0.34267, 0.33288, 0.32374, 0.32245, 0.31474, 0.30838, 0.30283],
[0.35384, 0.34286, 0.33386, 0.32507, 0.3246, 0.31745, 0.31135, 0.306],
[0.35338, 0.343, 0.33464, 0.32614, 0.3263, 0.31961, 0.31371, 0.30852],
[0.35301, 0.34312, 0.33526, 0.32698, 0.32766, 0.32132, 0.31558, 0.31052],
[0.35272, 0.34322, 0.33574, 0.32765, 0.32873, 0.32267, 0.31705, 0.31209],
[0.35246, 0.3433, 0.33617, 0.32822, 0.32965, 0.32383, 0.31831, 0.31344],
[0.35226, 0.34336, 0.33651, 0.32869, 0.3304, 0.32477, 0.31934, 0.31453],
[0.35207, 0.34342, 0.33681, 0.32911, 0.33106, 0.32561, 0.32025, 0.3155],
[0.35171, 0.34327, 0.33679, 0.32931, 0.3319, 0.32665, 0.32139, 0.31675],
[0.35128, 0.343, 0.33658, 0.32937, 0.33276, 0.32769, 0.32255, 0.31802],
[0.35086, 0.34274, 0.33637, 0.32943, 0.3336, 0.32872, 0.32368, 0.31927],
[0.35049, 0.34252, 0.33618, 0.32948, 0.33432, 0.32959, 0.32465, 0.32034],
[0.35016, 0.34231, 0.33602, 0.32953, 0.33498, 0.3304, 0.32554, 0.32132],
[0.34986, 0.34213, 0.33587, 0.32957, 0.33556, 0.3311, 0.32631, 0.32217],
[0.34959, 0.34196, 0.33573, 0.32961, 0.3361, 0.33176, 0.32704, 0.32296],
[0.34934, 0.34181, 0.33561, 0.32964, 0.33658, 0.33235, 0.32769, 0.32368],
[0.34912, 0.34167, 0.3355, 0.32967, 0.33701, 0.33288, 0.32827, 0.32432],
[0.34891, 0.34154, 0.33539, 0.3297, 0.33742, 0.33337, 0.32881, 0.32492]]

Implied Volatility Surface

Each row in data is a different exipiration time, and each column corresponds to various strikes as given in strikes. We load all this data into the QuantLib Matrix object. This can then be used seamlessly in the various surface construction routines. The variable implied_vols holds the above data in a Matrix format. One unusual bit of info that one needs to pay attention to is the ordering of the rows and columns in the Matrix object. The implied volatilities in the QuantLib context needs to have strikes along the row dimension and expiries in the column dimension. This is transpose of the way the data was constructed above. All of this detail is taken care by swapping the i and j variables below. Pay attention to the line:

implied_vols[i][j] = data[j][i]

in the cell below.

In [4]:

implied_vols = ql.Matrix(len(strikes), len(expiration_dates))
for i in range(implied_vols.rows()):
    for j in range(implied_vols.columns()):
        implied_vols[i][j] = data[j][i]

Now the Black volatility surface can be constructed using the BlackVarianceSurface method.

In [5]:

black_var_surface = ql.BlackVarianceSurface(
    calculation_date, calendar, 
    expiration_dates, strikes, 
    implied_vols, day_count)

The volatilities for any given strike and expiry pair can be easily obtained using black_var_surface shown below.

In [6]:

strike = 600.0
expiry = 1.2 # years
black_var_surface.blackVol(expiry, strike)

Out[6]:

0.3352982638587421

Visualization

In [7]:

import numpy as np
% matplotlib inline
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.pyplot as plt
from matplotlib import cm

Given an expiry, we can visualize the volatility as a function of the strike.

In [8]:

strikes_grid = np.arange(strikes[0], strikes[-1],10)
expiry = 1.0 # years
implied_vols = [black_var_surface.blackVol(expiry, s) 
                for s in strikes_grid] # can interpolate here
actual_data = data[11] # cherry picked the data for given expiry

fig, ax = plt.subplots()
ax.plot(strikes_grid, implied_vols, label="Black Surface")
ax.plot(strikes, actual_data, "o", label="Actual")
ax.set_xlabel("Strikes", size=12)
ax.set_ylabel("Vols", size=12)
legend = ax.legend(loc="upper right")

The whole volatility surface can also be visualised as shown below.

In [10]:

plot_years = np.arange(0, 2, 0.1)
plot_strikes = np.arange(535, 750, 1)
fig = plt.figure()
ax = fig.gca(projection='3d')
X, Y = np.meshgrid(plot_strikes, plot_years)
Z = np.array([black_var_surface.blackVol(y, x) 
              for xr, yr in zip(X, Y) 
                  for x, y in zip(xr,yr) ]
             ).reshape(len(X), len(X[0]))

surf = ax.plot_surface(X,Y,Z, rstride=1, cstride=1, cmap=cm.coolwarm, 
                linewidth=0.1)
fig.colorbar(surf, shrink=0.5, aspect=5)

Out[10]:

<matplotlib.colorbar.Colorbar instance at 0x0963C7D8>

One can also construct a local volatility surface (a la Dupire) using the LocalVolSurface. There are some issues with this as shown below.

In [11]:

local_vol_surface = ql.LocalVolSurface(
    ql.BlackVolTermStructureHandle(black_var_surface), 
    flat_ts, 
    dividend_ts, 
    spot)

In [12]:

plot_years = np.arange(0, 2, 0.1)
plot_strikes = np.arange(535, 750, 1)
fig = plt.figure()
ax = fig.gca(projection='3d')
X, Y = np.meshgrid(plot_strikes, plot_years)
Z = np.array([local_vol_surface.localVol(y, x) 
              for xr, yr in zip(X, Y) 
                  for x, y in zip(xr,yr) ]
             ).reshape(len(X), len(X[0]))

surf = ax.plot_surface(X, Y, Z, rstride=1, cstride=1, cmap=cm.coolwarm, 
                linewidth=0.1)
fig.colorbar(surf, shrink=0.5, aspect=5)

Out[12]:

<matplotlib.colorbar.Colorbar instance at 0x091720F8>

The correct pricing of local volatility surface requires an arbitrage free implied volatility surface. If the input implied volatility surface is not arbitrage free, this can lead to negative transition probabilities and/or negative local volatilities and can give rise to mispricing. Refer to Fengler's arbtirage free smoothing [1] which QuantLib currently lacks.

When you use an arbitrary smoothing, you will notice that the local volatility surface leads to undesired negative volatilities.

In [13]:

black_var_surface.setInterpolation("bicubic")
local_vol_surface = ql.LocalVolSurface(
    ql.BlackVolTermStructureHandle(black_var_surface), 
    flat_ts, 
    dividend_ts, 
    spot)
plot_years = np.arange(0, 2, 0.15)
plot_strikes = np.arange(535, 750, 10)
fig = plt.figure()
ax = fig.gca(projection='3d')
X, Y = np.meshgrid(plot_strikes, plot_years)
Z = np.array([local_vol_surface.localVol(y, x) 
              for xr, yr in zip(X, Y) 
                  for x, y in zip(xr,yr) ]
             ).reshape(len(X), len(X[0]))

surf = ax.plot_surface(Y,X, Z, rstride=1, cstride=1, cmap=cm.coolwarm, 
                linewidth=0.1)
fig.colorbar(surf, shrink=0.5, aspect=5)


---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-13-51cf0ac8418d> in <module>()
     12 Z = np.array([local_vol_surface.localVol(y, x) 
     13               for xr, yr in zip(X, Y)
---> 14                   for x, y in zip(xr,yr) ]
     15              ).reshape(len(X), len(X[0]))
     16 

C:\Users\gbalaram\Documents\Code\env\Lib\site-packages\QuantLib\QuantLib.pyc in localVol(self, *args)
   7467 
   7468     def localVol(self, *args):
-> 7469         return _QuantLib.LocalVolTermStructure_localVol(self, *args)
   7470 
   7471     def enableExtrapolation(self):

RuntimeError: negative local vol^2 at strike 655 and time 0.75; the black vol surface is not smooth enough

Heston Model Calibration

Heston model is defined by the following stochastic differential equations.

\begin{eqnarray} dS(t, S) &=& \mu S dt + \sqrt{v} S dW_1 \\ dv(t, S) &=& \kappa (\theta - v) dt + \sigma \sqrt{v} dW_2 \\ dW_1 dW_2 &=& \rho dt \end{eqnarray}

Here the asset is modeled as a stochastic process that depends on volatility $v$ which is a mean reverting stochastic process with a constant volatility of volatility $\sigma$. The two stochastic processes have a correlation $\rho$.

Let us look at how we can calibrate the Heston model to some market quotes. As an example, let's say we are interested in trading options with 1 year maturity. So we will calibrate the Heston model to fit to market volatility quotes with one year maturity. Before we do that, we need to construct the pricing engine that the calibration routines would need. In order to do that, we start by constructing the Heston model with some dummy starting parameters as shown below.

In [14]:

# dummy parameters
v0 = 0.01; kappa = 0.2; theta = 0.02; rho = -0.75; sigma = 0.5;

process = ql.HestonProcess(flat_ts, dividend_ts, 
                           ql.QuoteHandle(ql.SimpleQuote(spot)), 
                           v0, kappa, theta, sigma, rho)
model = ql.HestonModel(process)
engine = ql.AnalyticHestonEngine(model) 
# engine = ql.FdHestonVanillaEngine(model)

Now that we have the Heston model and a pricing engine, let us pick the quotes with all strikes and 1 year maturity in order to calibrate the Heston model. We build the Heston model helper which will be fed into the calibration routines.

In [15]:

heston_helpers = []
black_var_surface.setInterpolation("bicubic")
one_year_idx = 11 # 12th row in data is for 1 year expiry
date = expiration_dates[one_year_idx]
for j, s in enumerate(strikes):
    t = (date - calculation_date )
    p = ql.Period(t, ql.Days)
    sigma = data[one_year_idx][j]
    #sigma = black_var_surface.blackVol(t/365.25, s)
    helper = ql.HestonModelHelper(p, calendar, spot, s, 
                                  ql.QuoteHandle(ql.SimpleQuote(sigma)),
                                  flat_ts, 
                                  dividend_ts)
    helper.setPricingEngine(engine)
    heston_helpers.append(helper)

In [16]:

lm = ql.LevenbergMarquardt(1e-8, 1e-8, 1e-8)
model.calibrate(heston_helpers, lm, 
                 ql.EndCriteria(500, 50, 1.0e-8,1.0e-8, 1.0e-8))
theta, kappa, sigma, rho, v0 = model.params()

In [17]:

print "theta = %f, kappa = %f, sigma = %f, rho = %f, v0 = %f" % (theta, kappa, sigma, rho, v0)

theta = 0.135147, kappa = 1.878471, sigma = 0.002035, rho = -1.000000, v0 = 0.043485

Let us look at the quality of calibration by pricing the options used in the calibration using the model and lets get an estimate of the relative error.

In [18]:

avg = 0.0

print "%15s %15s %15s %20s" % (
    "Strikes", "Market Value", 
     "Model Value", "Relative Error (%)")
print "="*70
for i, opt in enumerate(heston_helpers):
    err = (opt.modelValue()/opt.marketValue() - 1.0)
    print "%15.2f %14.5f %15.5f %20.7f " % (
        strikes[i], opt.marketValue(), 
        opt.modelValue(), 
        100.0*(opt.modelValue()/opt.marketValue() - 1.0))
    avg += abs(err)
avg = avg*100.0/len(heston_helpers)
print "-"*70
print "Average Abs Error (%%) : %5.3f" % (avg)

        Strikes    Market Value     Model Value   Relative Error (%)
======================================================================
         527.50      184.23863       177.26114           -3.7872037 
         560.46      162.13295       156.75990           -3.3139776 
         593.43      141.96233       138.13403           -2.6967003 
         626.40      123.03554       121.33552           -1.3817289 
         659.37      108.50169       106.27965           -2.0479306 
         692.34       93.29771        92.86049           -0.4686261 
         725.31       79.64951        80.95876            1.6437604 
         758.28       67.62146        70.44848            4.1806550 
----------------------------------------------------------------------
Average Abs Error (%) : 2.440

References

[1] Mathias R. Fengler, Arbitrage Free Smoothing of Implied Volatility Surface, https://core.ac.uk/download/files/153/6978470.pdf

Click here to download the ipython notebook.

Valuing Convertible Bonds Using QuantLib Python

Goutham Balaraman — Wed, 16 Mar 2016 00:00:00 -0700

In [1]:

import QuantLib as ql

In this blog I will work through an example of valuing convertible bonds in QuantLib. Lets start by doing the usual setup of creating a calculation_date and setting it as the evaluationDate.

In [2]:

calculation_date = ql.Date(9,1,2004)
ql.Settings.instance().evaluationDate = calculation_date

One little quirk in the QuantLib convertible bond implementation is that there are places where the redemption amount is hard coded to 100. So if you have conversion ratio evaluated as \begin{eqnarray} Conversion\ Ratio = \frac{Redemption\ Amount}{Conversion\ Price} \end{eqnarray}

you will need to scale to an appropriate value with a redemption amount of 100. For instance, vendors report conversion ratio with a redemption amount of 1000. The conversion ratio obtained this way should be divided by a factor of 10 to get the equivalent conversion ratio for use in the QuantLib calculations. This is a limitation right now (as of version 1.7), which can be fixed in the future.

Following is the details of the convertible bond of interest.

In [3]:

# St. Mary Land & Exploration Company 
# Bloomberg ticker: SM 5.75 03/15/22 

redemption = 100.00
face_amount = 100.0
spot_price = 29.04
conversion_price = 26.0
conversion_ratio = 3.84615  # BBG quotes 38.4615; had to scale by a factor of 10

issue_date = ql.Date(15,3,2002)        
maturity_date = ql.Date(15,3,2022)

settlement_days = 2
calendar = ql.UnitedStates(ql.UnitedStates.GovernmentBond)
coupon = 0.0575
frequency = ql.Semiannual
tenor = ql.Period(frequency)

day_count = ql.Thirty360()
accrual_convention = ql.Unadjusted
payment_convention = ql.Unadjusted

call_dates = [ql.Date(20,3,2007)]
call_price = 100.0
put_dates = [ql.Date(20,3,2007), ql.Date(15,3,2012), ql.Date(15,3,2017)]
put_price = 100.0

# assumptions
dividend_yield = 0.02
credit_spread_rate = 0.03  
risk_free_rate = 0.04
volatility = 0.40

The call and put schedule for this bond is created as shown below. Here for each call date, we create a CallabilityPrice, then use that to form the Callability object. This is appended to the CallabilitySchedule object to form a list of call and put schedules.

In [4]:

callability_schedule = ql.CallabilitySchedule()


for call_date in call_dates:
   callability_price  = ql.CallabilityPrice(call_price, 
                                            ql.CallabilityPrice.Clean)
   callability_schedule.append(ql.Callability(callability_price, 
                                       ql.Callability.Call,
                                       call_date)
                        )
    
for put_date in put_dates:
    puttability_price = ql.CallabilityPrice(put_price, 
                                            ql.CallabilityPrice.Clean)
    callability_schedule.append(ql.Callability(puttability_price,
                                               ql.Callability.Put,
                                               put_date))

Any dividend information for the underlying stock is used to form the DividendSchedule.

In [5]:

dividend_schedule = ql.DividendSchedule() # No dividends
dividend_amount = dividend_yield*spot_price
next_dividend_date = ql.Date(1,12,2004)
dividend_amount = spot_price*dividend_yield
for i in range(4):
    date = calendar.advance(next_dividend_date, 1, ql.Years)
    dividend_schedule.append(
        ql.FixedDividend(dividend_amount, date)
    )

Now we bulid the fixed coupon convertible bond object

In [6]:

schedule = ql.Schedule(issue_date, maturity_date, tenor,
                       calendar, accrual_convention, accrual_convention,
                       ql.DateGeneration.Backward, False)

credit_spread_handle = ql.QuoteHandle(ql.SimpleQuote(credit_spread_rate))
exercise = ql.AmericanExercise(calculation_date, maturity_date)

convertible_bond = ql.ConvertibleFixedCouponBond(exercise,
                                                 conversion_ratio,
                                                 dividend_schedule,
                                                 callability_schedule, 
                                                 credit_spread_handle,
                                                 issue_date,
                                                 settlement_days,
                                                 [coupon],
                                                 day_count,
                                                 schedule,
                                                 redemption)

Build the Black-Scholes-Merton process to model the equity part

In [7]:

spot_price_handle = ql.QuoteHandle(ql.SimpleQuote(spot_price))
yield_ts_handle = ql.YieldTermStructureHandle(
    ql.FlatForward(calculation_date, risk_free_rate, day_count)
)
dividend_ts_handle = ql.YieldTermStructureHandle(
    ql.FlatForward(calculation_date, dividend_yield, day_count)
)
volatility_ts_handle = ql.BlackVolTermStructureHandle(
    ql.BlackConstantVol(calculation_date, calendar,volatility, day_count)
)

bsm_process = ql.BlackScholesMertonProcess(spot_price_handle, 
                                           dividend_ts_handle,
                                           yield_ts_handle,
                                           volatility_ts_handle)

Build the convertible bond pricing engine

In [8]:

time_steps = 1000
engine = ql.BinomialConvertibleEngine(bsm_process, "crr", time_steps)

Price the bond

In [9]:

convertible_bond.setPricingEngine(engine)
print "NPV ", convertible_bond.NPV()

NPV  132.308276818

Click here to download the ipython notebook on convertible bonds.

Understanding Interest Rate Risk

Goutham Balaraman — Wed, 09 Mar 2016 00:00:00 -0800

Interest rate changes affect returns on bonds and other fixed income assets, in which a lot of people nearing retirement would have their money invested. But volatility in interest rates also affect non-investment businesses which then indirectly affect the prices that we pay for products that we consume.

There are different businesses units that care about interest rate risk. Some businesses hurt when rates go down, while others care if the rates go up. The fact that different businesses have different concerns on interest rate movements mean that hedging strategies can be employed. In some sense, hedging is a way of distributing risk from one entity to others. When all is well, this can be a great thing for your business. The concept or interest rate risk management involves understanding, quantifying and hedging interest rate risk.

Who cares about interest rate risk?

There are different industries and businesses that are affected by interest rate fluctuations. Let us look at a few examples.

Fixed Income Investors: Funds that we contribute to our retirement accounts or pension plans are invested in equity and fixed income funds. Fixed income investors purchase bonds which are affected by interest rate fluctuations. Price of a fixed income bond is (generally) inversely propotional to interest rate movements, i.e. the value of the investment drops when interest rates rise and gains when interest rates drop. The sensitivity of fixed income investment to interest rate depends on the asset's maturity. Longer maturity investments are more sensitive to interest rate changes than short maturity investments.
Mortgage Lenders & Servicers: Mortgage lenders allow prospective home buyers to lock in on a mortgage rate for a couple of months. Lets say, that the lender has provided the home buyer a locked mortgage rate for a loan of \$100,000. But in 2 months time, when the home buyer closes on the loan, should the interest rates rise, then mortgage loan is worth less than \$100,000 and the lender stands to lose when he sells in the secondary market. Similarly, on the mortgage servicing side, when interest rates go down, the loans in the servicing portfolio could drop out due to interest rate refinancing. This can lead to decrease in projected cashflows.
Small and Large Businesses: Businesses need to raise capital in order to expand, and very often do so by using loans from financial institutions. The loans given out to small to medium businesses typically carry floating rate payments. Because of a variable interest rate loan, businesses are exposed to the interest rate fluctuations.

Addressing Interest Rate Risk

There are various tools available to market practitioners to mitigate exposure to interest rate risks. Derivative contracts such as interest rate swaps allow two parties to exchange cashflows such as fixed to float, or float on one tenor with float on another tenor. By entering a fixed for float swap, businesses can convert a floating payment on their loans into a net fixed payment.

Futures contracts on interest rate with different maturities on the deliverable allows one to hedge interest rate risk exposure on their books. The Futures contracts are exchange traded and offer liquidity and do not have credit risk associated.

Conclusion

In this blog post, I provided an introduction to interest rate risk and how it can be managed. At a later post, I will drill into some of the specifics of how certain derivatives can help hedge interest rate risk.

An Introduction to the Fundamental Review of the Trading Book (FRTB)

Goutham Balaraman — Fri, 19 Feb 2016 00:00:00 -0800

Basel Committee on Banking Supervision (BCBS) released a consultative document titled, "The Fundamental Review of the Trading Book" (FRTB) in 2012 1. The intent of this document and the subsequent discussions and studies was to address the shortcomings of the existing market risk capital framework, and reduce the variability in the minimum capital standard for market risk across different jurisdictions.

The key changes proposed under the new market risk framework finalized on 14th January 2016:

A revised internal models-approach (IMA). The new approach introduces a more rigorous model approval process that enables supervisors to remove internal modelling permission for individual trading desks, more consistent identification and capitalisation of material risk factors across banks, and constraints on the capital-reducing effects of hedging and diversification.
A revised standardised approach (SA). The revisions fundamentally overhaul the standardised approach to make it sufficiently risk-sensitive to serve as a credible fallback for, as well as a floor to, the IMA, while still providing an appropriate standard for banks that do not require a sophisticated treatment for market risk.
A shift from Value-at-Risk (VaR) to an Expected Shortfall (ES) measure of risk under stress. Use of ES will help to ensure a more prudent capture of “tail risk” and capital adequacy during periods of significant financial market stress.
Incorporation of the risk of market illiquidity. Varying liquidity horizons are incorporated into the revised SA and IMA to mitigate the risk of a sudden and severe impairment of market liquidity across asset markets. These replace the static 10-day horizon assumed for all traded instruments under VaR in the current framework.
A revised boundary between the trading book and banking book. Establishment of a more objective boundary will serve to reduce incentives to arbitrage between the regulatory banking and trading books, while still being aligned with banks' risk management practices.

The BCBS estimates that FRTB will result in 40% increase in total market risk capital requirements as compared to the current Basel 2.5 framework. The changes proposed under FRTB not only have an impact on the capital requirements, but it will also pose significant challenges to a bank's technology infrastructure.

.. to be continued

Valuing Options on Commodity Futures Using QuantLib Python

Gouthaman Balaraman — Thu, 17 Dec 2015 00:00:00 -0800

The Black-Scholes equation is the well known model to price equity European options. In the case of equities, the spot price fluctuates and hence the intuition to model as a stochastic process makes sense. In the case of commodities, however, the spot price does not fluctuate as much, and hence cannot be modeled as a stochastic process to value options on commodities. In the 1976 paper [1], Fischer Black addressed this problem by modeling the futures price, which demonstrates fluctuations, as the lognormal stochastic process.

The futures price at time $t$, $F_t$ with a is modeled as:

\begin{eqnarray} dF_t &=& \sigma_t F_t dW \end{eqnarray}

where $\sigma_t$ is the volatility of the underlying, and $dW$ is the Weiner process.

The value of an option at strike $K$, maturing at $T$, risk free rate $r$ with volatility $\sigma$ of the underlying is given by the closed form expression:

\begin{eqnarray} c &=& e^{-r T} [FN(d_1) - KN(d_2)] \nonumber \\ p &=& e^{-r T} [KN(-d_2) - FN(-d_1)] \end{eqnarray}

where \begin{eqnarray} d_1 &=& \frac{\ln(F/K) + (\sigma^2/2)T}{\sigma\sqrt{T}} \nonumber\\ d_2 &=& \frac{\ln(F/K) - (\sigma^2/2)T}{\sigma\sqrt{T}} = d_1 - \sigma\sqrt{T} \end{eqnarray}

This formula can be easily used to price caps, swaptions, futures options contract. Here we will use QuantLib to price the options on commodity futures.

In [1]:

import QuantLib as ql
import math

In [2]:

calendar = ql.UnitedStates()
bussiness_convention = ql.ModifiedFollowing
settlement_days = 0
day_count = ql.ActualActual()

Option on Treasury Futures Contract

Options on treasury futures (10 Yr Note TYF6C 119) can be valued using the Black formula. Let us value a Call option maturing on December 24, 2015, with a strike of 119. The current futures price is 126.95, and the volatility is 11.567%. The risk free rate as of December 1, 2015 is 0.105%. Let us value this Call option as of December 1, 2015.

In [3]:

interest_rate = 0.00105
calc_date = ql.Date(1,12,2015)
yield_curve = ql.FlatForward(calc_date, 
                             interest_rate,
                             day_count,
                             ql.Compounded,
                             ql.Continuous)

In [4]:

ql.Settings.instance().evaluationDate = calc_date
option_maturity_date = ql.Date(24,12,2015)
strike = 119
spot = 126.953# futures price
volatility = 11.567/100.
flavor = ql.Option.Call

discount = yield_curve.discount(option_maturity_date)
strikepayoff = ql.PlainVanillaPayoff(flavor, strike)
T = yield_curve.dayCounter().yearFraction(calc_date, 
                                          option_maturity_date)
stddev = volatility*math.sqrt(T)

black = ql.BlackCalculator(strikepayoff, 
                           spot, 
                           stddev, 
                           discount)

In [5]:

print "%-20s: %4.4f" %("Option Price", black.value() )
print "%-20s: %4.4f" %("Delta", black.delta(spot) )
print "%-20s: %4.4f" %("Gamma", black.gamma(spot) )
print "%-20s: %4.4f" %("Theta", black.theta(spot, T) )
print "%-20s: %4.4f" %("Vega", black.vega(T) )
print "%-20s: %4.4f" %("Rho", black.rho( T) )

Option Price        : 7.9686
Delta               : 0.9875
Gamma               : 0.0088
Theta               : -0.9356
Vega                : 1.0285
Rho                 : 7.3974

Natural Gas Futures Option

I saw this question on quantlib users group. Thought I will add this example as well.

Call option with a 3.5 strike, spot 2.919, volatility 0.4251. The interest rate is 0.15%.

In [6]:

interest_rate = 0.0015
calc_date = ql.Date(23,9,2015)
yield_curve = ql.FlatForward(calc_date, 
                             interest_rate,
                             day_count,
                             ql.Compounded,
                             ql.Continuous)

In [7]:

ql.Settings.instance().evaluationDate = calc_date
T = 96.12/365.

strike = 3.5
spot = 2.919
volatility = 0.4251
flavor = ql.Option.Call

discount = yield_curve.discount(T)
strikepayoff = ql.PlainVanillaPayoff(flavor, strike)
stddev = volatility*math.sqrt(T)

strikepayoff = ql.PlainVanillaPayoff(flavor, strike)
black = ql.BlackCalculator(strikepayoff, spot, stddev, discount)

In [8]:

print "%-20s: %4.4f" %("Option Price", black.value() )
print "%-20s: %4.4f" %("Delta", black.delta(spot) )
print "%-20s: %4.4f" %("Gamma", black.gamma(spot) )
print "%-20s: %4.4f" %("Theta", black.theta(spot, T) )
print "%-20s: %4.4f" %("Vega", black.vega(T) )
print "%-20s: %4.4f" %("Rho", black.rho( T) )

Option Price        : 0.0789
Delta               : 0.2347
Gamma               : 0.4822
Theta               : -0.3711
Vega                : 0.4600
Rho                 : 0.1597

Conclusion

In this notebook, I demonstrated how Black formula can be used to value options on commodity futures. It is worth pointing out that different vendors usually have different scaling conventions when it comes to reporting greeks. One would needs to take that into account when comparing the values shown by QuantLib with that of other vendors

References

[1] Fischer Black, The pricing of commodity contracts, Journal of Financial Economics, (3) 167-179 (1976)

Click here to download the ipython notebook on treasury futures.

Valuing Treasury Futures Using QuantLib Python

Gouthaman Balaraman — Wed, 02 Dec 2015 00:00:00 -0800

In this post, we will learn how to value treasury futures contract using QuantLib. The treasury futures contract gives the buyer the right to buy the underlying by the time the contract expires. The underlying is usually delivered from a basket of securities. So in order to properly value the futures contract, we would need to find the deliverable. Here we start by doing a naive calculation by constructing a fictional security. We will see what is wrong about this approach. As a next step we will perform the cheapest to deliver calculation and subsequently use that deliverable to value the same contract.

In [1]:

import QuantLib as ql
import math

In [2]:

calc_date = ql.Date(30,11,2015)
ql.Settings.instance().evaluationDate = calc_date
day_count = ql.ActualActual()
calendar = ql.UnitedStates()
bussiness_convention = ql.Following
end_of_month = False
settlement_days = 0
face_amount = 100
coupon_frequency = ql.Period(ql.Semiannual)

Build Yield Curve

As a first step, we build the treasury curve out of the treasury securities such as T-Bills, T-Notes and Treasury bonds.

In [3]:

Expand Code

In [4]:

yield_curve = ql.PiecewiseCubicZero(calc_date, bond_helpers, day_count)
yield_curve_handle = ql.YieldTermStructureHandle(yield_curve)
for d in maturity_dates:
    print "|%20s  %9.6f|"%(d, yield_curve.discount(d))

| December 24th, 2015   0.999935|
| February 25th, 2016   0.999576|
|      May 26th, 2016   0.998119|
| November 10th, 2016   0.995472|
| November 30th, 2017   0.981524|
| November 15th, 2018   0.964278|
| November 30th, 2020   0.920306|
| November 30th, 2022   0.868533|
| November 15th, 2025   0.799447|
| November 15th, 2045   0.384829|

Treasury Futures

Here we want to understand how to model treasury futures contract. Let us look at the TYZ5, the treasury futures on the 10 year note for delivery in December 2015. The notional deliverable is a 10-year 6% coupon note. In reality, the seller of the futures contract can deliver from a basket of securities.

For now, lets assume that the deliverable is actually a 6% coupon 10-year note issued as of the calculation date. Let us construct a 10 year treasury note and value this security. The futures price for the TYZ5 is $127.0625.

In [5]:

Expand Code

In [6]:

bond_issue_date = calc_date 
delivery_date = ql.Date(1,12,2015)

bond_maturity_date = bond_issue_date + ql.Period(10, ql.Years)
day_count = ql.ActualActual()
coupon_frequency = ql.Period(6, ql.Months)
coupon_rate = 6/100.

deliverable = create_tsy_security(bond_issue_date,
                                  bond_maturity_date,
                                  coupon_rate,
                                  coupon_frequency,
                                  day_count,
                                  calendar
                                  )
bond_engine = ql.DiscountingBondEngine(yield_curve_handle)
deliverable.setPricingEngine(bond_engine)

Lets calculate the Z-Spread for this deliverable. The Z-Spread is the static spread added to the yield curve to match the price of the security. This spread is a measure of the risk associated with the security. For a treasury security, you would expect this to be zero.

In [7]:

futures_price = 127.0625
clean_price = futures_price*yield_curve.discount(delivery_date)

zspread = ql.BondFunctions.zSpread(deliverable, clean_price,
                                   yield_curve, day_count, 
                                   ql.Compounded, ql.Semiannual, 
                                   calc_date)*10000
print "Z-Spread =%3.0fbp" % (zspread)

Z-Spread = 71bp

Here we get a spread of 71 basis points. This is unusually high for a treasury futures contract.

Cheapest To Deliver

Above we used a fictional 6% coupon bond as the deliverable. In reality, the deliverable is picked from a basket of securities based on what is the cheapest to deliver. Cheapest to deliver is not the cheapest in price. The seller of the futures contract, has to buy the delivery security from the market and sell it at an adjusted futures price. The adjusted futures price is given as:

Adjusted Futures Price = Futures Price x Conversion Factor

The gain or loss to the seller is given by the basis,

Basis = Cash Price - Adjusted Futures Price

So the cheapest to deliver is expected to be the security with the lowest basis. The conversion factor for a security is the price of the security with a 6% yield. Let us look at a basket of securities that is underlying this futures contract to understand this aspect.

In [8]:

Expand Code

# basket describes the coupon rate, maturity date, price
day_count = ql.ActualActual()
basket = [(1.625, ql.Date(15,8,2022), 97.921875), 
          (1.625, ql.Date(15,11,2022), 97.671875),
          (1.75, ql.Date(30,9,2022), 98.546875),
          (1.75, ql.Date(15,5,2023), 97.984375),
          (1.875, ql.Date(31,8,2022), 99.375),
          (1.875, ql.Date(31,10,2022),99.296875),
          (2.0, ql.Date(31,7,2022), 100.265625), 
          (2.0, ql.Date(15,2,2023), 100.0625),
          (2.0, ql.Date(15,2,2025), 98.296875),
          (2.0, ql.Date(15,8,2025), 98.09375),
          (2.125, ql.Date(30,6,2022), 101.06250),
          (2.125, ql.Date(15,5,2025),99.25),
          (2.25, ql.Date(15,11,2024),100.546875),
          (2.25, ql.Date(15,11,2025),100.375),
          (2.375, ql.Date(15,8,2024),101.671875),
          (2.5, ql.Date(15,8,2023),103.25),
          (2.5,ql.Date(15,5,2024),102.796875),
          (2.75, ql.Date(15,11,2023),105.0625),
          (2.75,ql.Date(15,2,2024),104.875)
          ]
securities = []
min_basis = 100; min_basis_index = -1
for i, b in enumerate(basket):
    coupon, maturity, price = b
    issue = maturity - ql.Period(10,ql.Years)
    s = create_tsy_security(issue,maturity, coupon/100.)
    bond_engine = ql.DiscountingBondEngine(yield_curve_handle)
    s.setPricingEngine(bond_engine)
    cf = ql.BondFunctions.cleanPrice(s,0.06,
                                     day_count, ql.Compounded,
                                     ql.Semiannual, calc_date)/100.
    adjusted_futures_price = futures_price * cf
    basis = price-adjusted_futures_price
    if basis< min_basis:
        min_basis = basis 
        min_basis_index = i
    securities.append((s,cf))
    
ctd_info = basket[min_basis_index]
ctd_bond,ctd_cf = securities[min_basis_index]
ctd_price = ctd_info[2]
print "%-30s = %lf" % ("Minimum Basis", min_basis)
print "%-30s = %lf" % ("Conversion Factor", ctd_cf)
print "%-30s = %lf" % ("Coupon", ctd_info[0])
print "%-30s = %s" % ("Maturity", ctd_info[1])
print "%-30s = %lf" % ("Price", ctd_info[2])

Minimum Basis                  = 0.450601
Conversion Factor              = 0.791830
Coupon                         = 2.125000
Maturity                       = June 30th, 2022
Price                          = 101.062500

The basis is the loss for a notional of \$100 that the seller accrues to close this contract. For a single futures contract (which has a \$100000 notional), there is a loss of \$450.60.

NOTE: You will need my pull request to execute the FixedRateBondForward class since it is not exposed in SWIG at the moment.

In [9]:

futures_maturity_date = ql.Date(21,12,2015)
futures = ql.FixedRateBondForward(calc_date, futures_maturity_date, 
                                  ql.Position.Long, 0.0, settlement_days,
                                  day_count, calendar, bussiness_convention,
                                  ctd_bond, yield_curve_handle, yield_curve_handle)

The valuation of the futures contract and the underlying is shown below:

In [10]:

Expand Code

Model Futures Price            = 127.610365
Market Futures Price           = 127.062500
Model Adjustment               = 0.547865
Implied Yield                  = -7.473%
Forward Z-Spread               = 1.6bps
Forward YTM                    = 1.952%

References

[1] Understanding Treasury Futures, CME Group PDF

Conclusion

In this notebook, we looked into understanding and valuing treasury futures contracts. We used the QuantLib FixedRateBondForward class in order to model the futures contract. But, we also made a cheapest to deliver calculation to figure out the deliverable.

Click here to download the ipython notebook on treasury futures.

QuantLib Python Tutorials With Examples

Gouthaman Balaraman — Fri, 30 Oct 2015 00:00:00 -0700

I have written a lot of little tutorials on using QuantLib python bindings. In these posts I explain some of the QuantLib concepts using minimal examples. Following are the links to these posts:

Introduction to QuantLib Python: This post will walk through some of the basics of QuantLib Python library.
Modeling Fixed Rate Bonds in QuantLib Python: This post will walk through an example of modeling fixed rate bonds using QuantLib Python.
An Introduction to Interest Rate Term Structure in QuantLib Python: This post will walk through the basics of bootstrapping yield curve in QuantLib Python.
Hull White Term Structure Simulations with QuantLib Python: Discusses simulation of the Hull White interest rate term structure model in QuantLib Python
Option Model Handbook, Part III: European Option Pricing With QuantLib Python: Demonstrates how to price European options using QuantLib Python. Methods using Black-Scholes-Merton formula and binomial tree will be discussed.
On the Convergence of Hull White Monte Carlo Simulations: Discusses the convergence of the Monte-Carlo simulations of the Hull-White model
Valuing European Option Using the Heston Model in QuantLib Python: Introduces an example on how to value European options using Heston model in Quantlib Python
Modeling Vanilla Interest Rate Swaps Using QuantLib Python: Provides a basic introduction to valuing interest rate swaps using QuantLib Python.
Short Interest Rate Model Calibration in QuantLib Python: Provides examples of short interest rate model calibration to swaption volatilities in QuantLib Python
Valuing Treasury Futures Using QuantLib Python: Provides an introduction to valuation of treasury futures contract in QuantLib Python.
Valuing Options on Commodity Futures Using QuantLib Python: Describes how to value options on commodity futures contract using the Black formula in QuantLib Python
Valuing Convertible Bonds Using QuantLib Python: Provides an introduction to valuation of convertible bonds using QuantLib Python with a minimal example.
Modeling Volatility Smile and Heston Model Calibration Using QuantLib Python: Provides an introduction to constructing implied volatility surface consistend with the smile observed in the market and calibrating Heston model using QuantLib Python.
QuantLib Python Cookbook Announcement: Announcement of the "QuantLib Python Cookbook"
Valuing Interest Rate Caps and Floors Using QuantLib Python: A tutorial on valuing caps and floors using QuantLib Python.
QuantLib Python Notebooks On Docker: Running QuantLib python notebooks on Docker
Announcing qtk for QuantLib Python: Announcing qtk, a new interface to interact with QuantLib Python
Valuing Bonds with Credit Spreads in QuantLib Python: Provides an example of valuing bonds with credit spreads using QuantLib Python. This post walks through an example of shifting the yield term structure.
Heston Model Calibration Using QuantLib Python and Scipy Optimize: In this post we do a deep dive on calibration of Heston model using QuantLib Python and Scipy's Optimize package.
Valuing Callable Bonds Using QuantLib Python: In this post we look at valuing callable bonds using QuantLib Python
Variance Reduction in Hull-White Monte Carlo Simulation Using Moment Matching: This post explains how to use moment matching to reduce variance in Monte Carlo simulation of the Hull-White term structure model.
American Option Pricing with QuantLib and Python: This post explains valuing American Options using QuantLib and Python

Short Interest Rate Model Calibration in QuantLib Python

Gouthaman Balaraman — Tue, 27 Oct 2015 00:00:00 -0700

I have talked about Hull-White model in my earlier blog posts. The focus of those posts was to see how to use the model classes. The model parameters were assumed to be given. However in practice, the model parameters need to calibrated from market data. Typically instruments such as swaptions, caps or floors and their market prices / volatilities are taken as inputs. Then the model parameters are fit in such a way that the model prices these options close enough. The goodness of fit depends, apart from the choice of the numerical methods, on the type of model itself. This is because models such as Hull-White 1 factor cannot fit some of the humped volatility term structures observed in the market. Never the less, Hull-White is usually a good starting point to understand calibration process.

Here we will discuss Hull-White model in detail. Then we will also show how the same procedure can be applied to calibrate other short rate models.

In [1]:

import QuantLib as ql
from collections import namedtuple
import math

Hull-White 1 Factor Model

Hull-White model was one of the first practical exogenous models that attempted to fit to the market interest rate term structures. The model is described as:

\begin{equation} dr_t = (\theta(t) - a r_t) dt + \sigma dW_t \end{equation}

where $a$ is the mean reversion constant, $\sigma$ is the volatility parameter. The parameter $\theta(t)$ is chosen in order to fit the input term structure of interest rates.

What is the "right" value for parameters $a$ and $\sigma$? This is the question that we address by calibrating to market instruments.

In [2]:

today = ql.Date(15, ql.February, 2002);
settlement= ql.Date(19, ql.February, 2002);
ql.Settings.instance().evaluationDate = today;
term_structure = ql.YieldTermStructureHandle(
    ql.FlatForward(settlement,0.04875825,ql.Actual365Fixed())
    )
index = ql.Euribor1Y(term_structure)

In this example we are going to calibrate to the swaption volatilities as shown below.

In [3]:

CalibrationData = namedtuple("CalibrationData", 
                             "start, length, volatility")
data = [CalibrationData(1, 5, 0.1148),
        CalibrationData(2, 4, 0.1108),
        CalibrationData(3, 3, 0.1070),
        CalibrationData(4, 2, 0.1021),
        CalibrationData(5, 1, 0.1000 )]

In [4]:

Expand Code

Calibrating Reversion and Volaitility

Here we use the JamshidianSwaptionEngine to value the swaptions as part of calibration. The JamshidianSwaptionEngine requires one-factor affine models as input. For other interest rate models, we need a pricing engine that is more suited to those models.

In [5]:

model = ql.HullWhite(term_structure);
engine = ql.JamshidianSwaptionEngine(model)
swaptions = create_swaption_helpers(data, index, term_structure, engine)

optimization_method = ql.LevenbergMarquardt(1.0e-8,1.0e-8,1.0e-8)
end_criteria = ql.EndCriteria(10000, 100, 1e-6, 1e-8, 1e-8)
model.calibrate(swaptions, optimization_method, end_criteria)

a, sigma = model.params()
print "a = %6.5f, sigma = %6.5f" % (a, sigma)

a = 0.04642, sigma = 0.00580

In [6]:

calibration_report(swaptions, data)

----------------------------------------------------------------------------------
    Model Price    Market Price     Implied Vol      Market Vol       Rel Error
----------------------------------------------------------------------------------
        0.00878         0.00949         0.10620         0.11480        -0.07485
        0.00967         0.01008         0.10629         0.11080        -0.04061
        0.00866         0.00872         0.10634         0.10700        -0.00614
        0.00649         0.00623         0.10644         0.10210         0.04237
        0.00354         0.00332         0.10661         0.10000         0.06582
----------------------------------------------------------------------------------
Cumulative Error :         0.11614

In [7]:

model = ql.HullWhite(term_structure);
engine = ql.JamshidianSwaptionEngine(model)
swaptions = create_swaption_helpers(data, index, term_structure, engine)

optimization_method = ql.LevenbergMarquardt(1.0e-8,1.0e-8,1.0e-8)
end_criteria = ql.EndCriteria(10000, 100, 1e-6, 1e-8, 1e-8)
model.calibrate(swaptions, optimization_method, end_criteria)

a, sigma = model.params()
print "a = %6.5f, sigma = %6.5f" % (a, sigma)
calibration_report(swaptions, data)

a = 0.04642, sigma = 0.00580
----------------------------------------------------------------------------------
    Model Price    Market Price     Implied Vol      Market Vol       Rel Error
----------------------------------------------------------------------------------
        0.00878         0.00949         0.10620         0.11480        -0.07485
        0.00967         0.01008         0.10629         0.11080        -0.04061
        0.00866         0.00872         0.10634         0.10700        -0.00614
        0.00649         0.00623         0.10644         0.10210         0.04237
        0.00354         0.00332         0.10661         0.10000         0.06582
----------------------------------------------------------------------------------
Cumulative Error :         0.11614

Calibrating Volatility With Fixed Reversion

Some times we need to calibrate with one parameter held fixed. This can be done in the QuantLib libraries. However, this ability is not exposed in the SWIG wrappers as of version 1.6. I have created a github issue and provided a patch to address this issue. You will need this patch to execute the following cells. Below, the model is calibrated with a fixed reversion value of 5%.

In [8]:

constrained_model = ql.HullWhite(term_structure, 0.05, 0.001);
engine = ql.JamshidianSwaptionEngine(constrained_model)
swaptions = create_swaption_helpers(data, index, term_structure, engine)

optimization_method = ql.LevenbergMarquardt(1.0e-8,1.0e-8,1.0e-8)
end_criteria = ql.EndCriteria(10000, 100, 1e-6, 1e-8, 1e-8)
constrained_model.calibrate(swaptions, optimization_method, end_criteria, ql.NoConstraint(), [], [True, False])

a, sigma = constrained_model.params()
print "a = %6.5f, sigma = %6.5f" % (a, sigma)

a = 0.05000, sigma = 0.00586

In [9]:

calibration_report(swaptions, data)

----------------------------------------------------------------------------------
    Model Price    Market Price     Implied Vol      Market Vol       Rel Error
----------------------------------------------------------------------------------
        0.00878         0.00949         0.10621         0.11480        -0.07474
        0.00967         0.01008         0.10628         0.11080        -0.04068
        0.00866         0.00872         0.10633         0.10700        -0.00626
        0.00649         0.00623         0.10644         0.10210         0.04231
        0.00354         0.00332         0.10663         0.10000         0.06595
----------------------------------------------------------------------------------
Cumulative Error :         0.11615

Black Karasinski Model

The Black Karasinski model is described as:

\begin{equation} d\ln(r_t) = (\theta_t - a \ln(r_t)) dt + \sigma dW_t \end{equation}

In order to calibrate, we use the TreeSwaptionEngine which will work with all short rate models. The calibration is shown below.

In [10]:

model = ql.BlackKarasinski(term_structure);
engine = ql.TreeSwaptionEngine(model, 100)
swaptions = create_swaption_helpers(data, index, term_structure, engine)

optimization_method = ql.LevenbergMarquardt(1.0e-8,1.0e-8,1.0e-8)
end_criteria = ql.EndCriteria(10000, 100, 1e-6, 1e-8, 1e-8)
model.calibrate(swaptions, optimization_method, end_criteria)

a, sigma =  model.params()
print "a = %6.5f, sigma = %6.5f" % (a, sigma)

a = 0.03902, sigma = 0.11695

In [11]:

calibration_report(swaptions, data)

----------------------------------------------------------------------------------
    Model Price    Market Price     Implied Vol      Market Vol       Rel Error
----------------------------------------------------------------------------------
        0.00872         0.00949         0.10550         0.11480        -0.08095
        0.00967         0.01008         0.10631         0.11080        -0.04045
        0.00868         0.00872         0.10654         0.10700        -0.00430
        0.00650         0.00623         0.10666         0.10210         0.04446
        0.00355         0.00332         0.10676         0.10000         0.06733
----------------------------------------------------------------------------------
Cumulative Error :         0.12163

G2++ Model

As a final example, let us look at a calibration example of the 2-factor G2++ model. \begin{equation} dr_t = \varphi(t) + x_t + y_t \end{equation}

where $ x_t $ and $ y_t $ are defined by

\begin{eqnarray} dx_t &=& -a x_t dt + \sigma dW^1_t\nonumber \\ dy_t &=& -b y_t dt + \eta dW^2_t \nonumber \\ \left<dW^1_t dW^2_t\right> &=& \rho dt \end{eqnarray}

Once again, we use the TreeSwaptionEngine to value the swaptions in the calibration step.

In [12]:

model = ql.G2(term_structure);
engine = ql.TreeSwaptionEngine(model, 25)
# engine = ql.G2SwaptionEngine(model, 10, 400)
# engine = ql.FdG2SwaptionEngine(model)
swaptions = create_swaption_helpers(data, index, term_structure, engine)

optimization_method = ql.LevenbergMarquardt(1.0e-8,1.0e-8,1.0e-8)
end_criteria = ql.EndCriteria(1000, 100, 1e-6, 1e-8, 1e-8)
model.calibrate(swaptions, optimization_method, end_criteria)

a, sigma, b, eta, rho = model.params()
print "a = %6.5f, sigma = %6.5f, b = %6.5f, eta = %6.5f, rho = %6.5f " % (a, sigma, b, eta, rho)

a = 0.04050, sigma = 0.00474, b = 0.04438, eta = 0.00301, rho = 0.03541

In [13]:

calibration_report(swaptions, data)

----------------------------------------------------------------------------------
    Model Price    Market Price     Implied Vol      Market Vol       Rel Error
----------------------------------------------------------------------------------
        0.00870         0.00949         0.10535         0.11480        -0.08228
        0.00967         0.01008         0.10633         0.11080        -0.04029
        0.00868         0.00872         0.10651         0.10700        -0.00456
        0.00650         0.00623         0.10665         0.10210         0.04438
        0.00355         0.00332         0.10680         0.10000         0.06770
----------------------------------------------------------------------------------
Cumulative Error :         0.12265

Conclusion

In this post, we saw some simple examples of calibrating the model to the swaption volatilities.

Click here to download the ipython notebook

Modeling Vanilla Interest Rate Swaps Using QuantLib Python

Gouthaman Balaraman — Tue, 20 Oct 2015 00:00:00 -0700

An Interest Rate Swap is a financial derivative instrument in which two parties agree to exchange interest rate cash flows based on a notional amount from a fixed rate to a floating rate or from one floating rate to another floating rate.

Here we will consider an example of a plain vanilla USD swap with 10 million notional and 10 year maturity. Let the fixed leg pay 2.5% coupon semiannually, and the floating leg pay Libor 3m quarterly.

Sample Code

In [1]:

Expand Code

Here we construct the yield curve objects. For simplicity, we will use flat curves for discounting and Libor 3M. This will help us focus on the Swap construction part. Please refer to curve construction example for some details. Once we construct the

In [2]:

# construct discount curve and libor curve

risk_free_rate = 0.01
libor_rate = 0.02
day_count = ql.Actual365Fixed()

discount_curve = ql.YieldTermStructureHandle(
    ql.FlatForward(calculation_date, risk_free_rate, day_count)
)

libor_curve = ql.YieldTermStructureHandle(
    ql.FlatForward(calculation_date, libor_rate, day_count)
)
#libor3M_index = ql.Euribor3M(libor_curve)  
libor3M_index = ql.USDLibor(ql.Period(3, ql.Months), libor_curve)

To construct the Swap instrument, we have to specify the fixed rate leg and floating rate leg. We construct the fixed rate and floating rate leg schedules below.

In [3]:

calendar = ql.UnitedStates()
settle_date = calendar.advance(calculation_date, 5, ql.Days)
maturity_date = calendar.advance(settle_date, 10, ql.Years)

fixed_leg_tenor = ql.Period(6, ql.Months)
fixed_schedule = ql.Schedule(settle_date, maturity_date, 
                             fixed_leg_tenor, calendar,
                             ql.ModifiedFollowing, ql.ModifiedFollowing,
                             ql.DateGeneration.Forward, False)

float_leg_tenor = ql.Period(3, ql.Months)
float_schedule = ql.Schedule (settle_date, maturity_date, 
                              float_leg_tenor, calendar,
                              ql.ModifiedFollowing, ql.ModifiedFollowing,
                              ql.DateGeneration.Forward, False)

Below, we construct a VanillaSwap object by including the fixed and float leg schedules created above.

In [4]:

notional = 10000000
fixed_rate = 0.025
fixed_leg_daycount = ql.Actual360()
float_spread = 0.004
float_leg_daycount = ql.Actual360()

ir_swap = ql.VanillaSwap(ql.VanillaSwap.Payer, notional, fixed_schedule, 
               fixed_rate, fixed_leg_daycount, float_schedule,
               libor3M_index, float_spread, float_leg_daycount )

We evaluate the swap using a discounting engine.

In [5]:

swap_engine = ql.DiscountingSwapEngine(discount_curve)
ir_swap.setPricingEngine(swap_engine)

Result Analysis

The cashflows for the fixed and floating leg can be extracted from the ir_swap object. The fixed leg cashflows are shown below:

In [6]:

Expand Code

 1    April 27th, 2016     127083.33
 2    October 27th, 2016   127083.33
 3    April 27th, 2017     126388.89
 4    October 27th, 2017   127083.33
 5    April 27th, 2018     126388.89
 6    October 29th, 2018   128472.22
 7    April 29th, 2019     126388.89
 8    October 28th, 2019   126388.89
 9    April 27th, 2020     126388.89
10    October 27th, 2020   127083.33
11    April 27th, 2021     126388.89
12    October 27th, 2021   127083.33
13    April 27th, 2022     126388.89
14    October 27th, 2022   127083.33
15    April 27th, 2023     126388.89
16    October 27th, 2023   127083.33
17    April 29th, 2024     128472.22
18    October 28th, 2024   126388.89
19    April 28th, 2025     126388.89
20    October 27th, 2025   126388.89

The floating leg cashflows are shown below:

In [7]:

Expand Code

 1    January 27th, 2016    60760.46
 2    April 27th, 2016      60098.65
 3    July 27th, 2016       60098.65
 4    October 27th, 2016    60760.46
 5    January 27th, 2017    60760.46
 6    April 27th, 2017      59436.87
 7    July 27th, 2017       60098.65
 8    October 27th, 2017    60760.46
 9    January 29th, 2018    62084.17
10    April 27th, 2018      58113.40
11    July 27th, 2018       60098.65
12    October 29th, 2018    62084.17
13    January 28th, 2019    60098.65
14    April 29th, 2019      60098.65
15    July 29th, 2019       60098.65
16    October 28th, 2019    60098.65
17    January 27th, 2020    60098.65
18    April 27th, 2020      60098.65
19    July 27th, 2020       60098.65
20    October 27th, 2020    60760.46
21    January 27th, 2021    60760.46
22    April 27th, 2021      59436.87
23    July 27th, 2021       60098.65
24    October 27th, 2021    60760.46
25    January 27th, 2022    60760.46
26    April 27th, 2022      59436.87
27    July 27th, 2022       60098.65
28    October 27th, 2022    60760.46
29    January 27th, 2023    60760.46
30    April 27th, 2023      59436.87
31    July 27th, 2023       60098.65
32    October 27th, 2023    60760.46
33    January 29th, 2024    62084.17
34    April 29th, 2024      60098.65
35    July 29th, 2024       60098.65
36    October 28th, 2024    60098.65
37    January 27th, 2025    60098.65
38    April 28th, 2025      60098.65
39    July 28th, 2025       60098.65
40    October 27th, 2025    60098.65

Some other analytics such as the fair value, fair spread etc can be extracted as shown below.

In [8]:

Expand Code

Net Present Value   :          -115054.034
Fair Spread         :                0.005
Fair Rate           :                0.024
Fixed Leg BPS       :            -9629.981
Floating Leg BPS    :             9642.042

Conclusion

Here we saw a simple example on how to construct a swap and value them. We evaluated the fixed and floating legs and then valued the VanillaSwap using the DiscountingSwapEngine.

Click here to download the ipython notebook on interest rate swaps.

Valuing European Option Using the Heston Model in QuantLib Python

Gouthaman Balaraman — Tue, 08 Sep 2015 00:00:00 -0700

Heston model can be used to value options by modeling the underlying asset such as the stock of a company. The one major feature of the Heston model is that it inocrporates a stochastic volatility term.

\begin{eqnarray} dS_t &=& \mu S_tdt + \sqrt{V_t} S_t dW_t^1 \\ dV_t &=& \kappa(\theta-V_t) + \sigma \sqrt{V_t} dW_t^2 \end{eqnarray}

Here :

$S_t$ is the asset's value at time $t$
$\mu$ is the expected growth rate of the log normal stock value
$V_t$ is the variance of the asset $S_t$
$W_t^1$ is the stochastic process governing the $S_t$ process
$\theta$ is the value of mean reversion for the variance $V_t$
$\kappa$ is the strengh of mean reversion
$\sigma$ is the volatility of volatility
$W_t^2$ is the stochastic process governing the $V_t$ process
The correlation between $W_t^1$ and $W_t^2$ is $\rho$

In contrast, the Black-Scholes-Merton process assumes that the volatility is constant.

In [1]:

Expand Code

Let us consider a European call option for AAPL with a strike price of \$130 maturing on 15th Jan, 2016. Let the spot price be \$127.62. The volatility of the underlying stock is know to be 20%, and has a dividend yield of 1.63%. We assume a short term risk free rate of 0.1%. Lets value this option as of 8th May, 2015.

In [2]:

Expand Code

Using the above inputs, we construct the European option as shown below.

In [3]:

# construct the European Option
payoff = ql.PlainVanillaPayoff(option_type, strike_price)
exercise = ql.EuropeanExercise(maturity_date)
european_option = ql.VanillaOption(payoff, exercise)

In order to price the option using the Heston model, we first create the Heston process. In order to create the Heston process, we use the parameter values: mean reversion strength kappa = 0.1, the spot variance v0 = volatility*volatility = 0.04, the mean reversion variance theta=v0, volatility of volatility sigma = 0.1 and the correlation between the asset price and its variance is rho = -0.75.

In [4]:

Expand Code

On valuing the option using the Heston model, we get the net present value as:

In [5]:

engine = ql.AnalyticHestonEngine(ql.HestonModel(heston_process),0.01, 1000)
european_option.setPricingEngine(engine)
h_price = european_option.NPV()
print "The Heston model price is",h_price

The Heston model price is 6.5308651372

Performing the same calculation using the Black-Scholes-Merton process, we get:

In [6]:

Expand Code

The Black-Scholes-Merton model price is  6.74927181246

The difference in the price between the two models is: bs_price - h_price = 0.21840667525992163. This difference is due to the stochastic modeling of the volatility as a CIR-process.

Conclusion

This post provided a minimal example of valuing European options using the Heston model. Comparison with the Black-Scholes-Merton model is shown for instructional purpose.

Adding Caching to Python Flask-Blogging Engine

Goutham Balaraman — Sat, 25 Jul 2015 00:00:00 -0700

I deployed my blog (using Flask-Blogging extension) to a site I am building. I am currently hosting the test version on the free instance of Openshift. So these are not very powerful servers, and I wasn't expecting any great performance. Here is what I saw while using Version 0.3.2 of Flask-Blogging:

$> ab -kc 30 -t 10 https://ucarpool.org/blog/

Benchmarking ucarpool.org (be patient)
Finished 267 requests

Document Path:          /blog/
Document Length:        4882 bytes

Concurrency Level:      30
Time taken for tests:   10.061 seconds
Complete requests:      267
Failed requests:        0
Keep-Alive requests:    267
Total transferred:      1428480 bytes
HTML transferred:       1303494 bytes
Requests per second:    26.54 [#/sec] (mean)
Time per request:       1130.451 [ms] (mean)
Time per request:       37.682 [ms] (mean, across all concurrent requests)
Transfer rate:          138.65 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   34  95.5      0     353
Processing:   272 1040 374.4    936    2166
Waiting:      267 1037 374.5    933    2163
Total:        546 1074 349.5    968    2166

Percentage of the requests served within a certain time (ms)
  50%    964
  66%   1039
  75%   1133
  80%   1248
  90%   1812
  95%   1910
  98%   2030
  99%   2070
 100%   2166 (longest request)

Thats a measely 26 requests/sec!

One of the features I have wanted to add to this extension is the ability to cache the pages. Blogs are typically heavy on reads, and light on writes. This makes an excellent case for caching.

In order to enable caching support, I am using Flask-Cache the caching extension for Flask. The version 0.4.0 of Flask-Blogging is released with caching support built in. Flask-Cache makes the caching backend configurable. It has support for various backends such as filesystem, redis, and memcache.

Here is resulting performance after using a filesystem based cache:

$> ab -kc 30 -t 10 https://ucarpool.org/blog/
This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking ucarpool.org (be patient)
Finished 1815 requests

Document Path:          /blog/
Document Length:        4882 bytes

Concurrency Level:      30
Time taken for tests:   10.004 seconds
Complete requests:      1815
Failed requests:        0
Keep-Alive requests:    1815
Total transferred:      9710280 bytes
HTML transferred:       8860830 bytes
Requests per second:    181.43 [#/sec] (mean)
Time per request:       165.351 [ms] (mean)
Time per request:       5.512 [ms] (mean, across all concurrent requests)
Transfer rate:          947.92 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    6  44.3      0     400
Processing:    80  158  46.1    150     411
Waiting:       79  156  46.1    147     410
Total:         83  164  58.9    152     518

Percentage of the requests served within a certain time (ms)
  50%    152
  66%    173
  75%    183
  80%    190
  90%    217
  95%    277
  98%    357
  99%    420
 100%    518 (longest request)

Now we are able to serve 180 requests/second. A good 7X performance gain for using a filesystem cache. Using in memory like Redis, or SSD filesystem should be even better.

Conclusion

Flask-Blogging, the Flask extension to add Markdown based blog support to Flask sites, incorporates caching support which has greatly improved its performance.

Flask-Blogging: A Python Flask Blog Engine as an Extension

Goutham Balaraman — Sat, 04 Jul 2015 00:00:00 -0700

I wanted to add a blog to my flask site. I decided to write it as an extension. This makes the blog code easy to plug into any site, and fairly reusable.

Flask-Blogging is a blog engine as a Flask extension based on Markdown. Please see:

Flask-Blogging documentation
Flask-Blogging github project page

Out of the box Flask-Blogging has support for the following:

Bootstrap based site
Markdown based blog editor
Models to store blog
Authentication of User’s choice
Sitemap, ATOM support
Disqus support for comments
Google analytics for usage tracking
Well documented, tested, and extensible design

Minimal Example

Here is a minimal example for getting a blog up and running. There is no security in the authentication here. But if you have authentication setup using either Flask-Login or Flask-Security, it should be straight forward to configure authentication. This example uses version 0.3.2.

from flask import Flask, render_template_string, redirect
from sqlalchemy import create_engine, MetaData
from flask.ext.login import UserMixin, LoginManager, \
    login_user, logout_user
from flask.ext.blogging import SQLAStorage, BloggingEngine

app = Flask(__name__)
app.config["SECRET_KEY"] = "secret"  # for WTF-forms and login
app.config["BLOGGING_URL_PREFIX"] = "/blog"
app.config["BLOGGING_DISQUS_SITENAME"] = "test"
app.config["BLOGGING_SITEURL"] = "http://localhost:8000"

# extensions
engine = create_engine('sqlite:////tmp/blog.db')
meta = MetaData()
sql_storage = SQLAStorage(engine, metadata=meta)
blog_engine = BloggingEngine(app, sql_storage)
login_manager = LoginManager(app)
meta.create_all(bind=engine)

# user class for providing authentication
class User(UserMixin):
    def __init__(self, user_id):
        self.id = user_id

    def get_name(self):
        return "Paul Dirac"  # typically the user's name

@login_manager.user_loader
@blog_engine.user_loader
def load_user(user_id):
    return User(user_id)

index_template = """
<!DOCTYPE html>
<html>
    <head> </head>
    <body>
        {% if current_user.is_authenticated() %}
            <a href="/logout/">Logout</a>
        {% else %}
            <a href="/login/">Login</a>
        {% endif %}
        &nbsp&nbsp<a href="/blog/">Blog</a>
        &nbsp&nbsp<a href="/blog/sitemap.xml">Sitemap</a>
        &nbsp&nbsp<a href="/blog/feeds/all.atom.xml">ATOM</a>
    </body>
</html>
"""

@app.route("/")
def index():
    return render_template_string(index_template)

@app.route("/login/")
def login():
    user = User("testuser")
    login_user(user)
    return redirect("/blog")

@app.route("/logout/")
def logout():
    logout_user()
    return redirect("/")


if __name__ == "__main__":
    app.run(debug=True, port=8000, use_reloader=True)

Screen Shots

Blog Editor

Here is a screenshot of the blog editor page. This editor uses the awesome Bootstrap-Markdown editor.

Blog Editor

Blog Page

The blog page when rendered looks as shown here. The page is configured out of the box to include LaTeX support. The math is rendered using MathJax. I have thought about using KaTex, but decided to wait till they have support for eqnarray. As you can see, it uses Disqus for comments.

Blog Page

Installation

Install the extension with the following commands:

$ easy_install flask-blogging

or alternatively if you have pip installed:

$ pip install flask-blogging

Conclusion

Here I introduced the Flask-Blogging extension that can be used as a flask blog engine to your flask site.

How to Build a Fundamental Factor Model

Gouthaman Balaraman — Mon, 22 Jun 2015 00:00:00 -0700

In a multi-factor model, the return of a stock can be broken out into multiple factors.

$$ r_i = \alpha_i + \sum_{j=1}^{K} \beta_{ij} f_j + \epsilon_i $$

The term $\epsilon_i$ is the error term which is assumed to have a distribution with zero mean. Now we can write the average return as

$$ E(r_i) = E(\alpha_i) + \sum_{j=1}^{K} \beta_{ij} E(f_j) $$

This can be expressed concisely using the matrix representation:

$$ r_i = \mathbf{\beta}^T_i \mathbf{f} + \epsilon_i $$

where:

\begin{eqarray} \mathbf{\beta}i &=& (\alpha_i, \beta, \beta_{i2}, ..., \beta_{iK})^T \ \mathbf(f) &=& (1, f_1, f_2, ..., f_K)^T \end{eqarray}

Using this notation, the average return can then be written as:

$$ E(r_i) = \mathbf{\beta}_i^T E(\mathbf{f}) $$

We see that the average return on the stock is the product of factor exposures ($\mathbf{\beta}_i$) and factor premium ($\mathbf{f}$). In a fundamental factor model, the factor exposures are characteristics of an investment (such as a stock) that is known, such as the Price to Earnings ratio, or momentum of the stock, or market capitalization. The factor premium is the unknown that we wish to determine empirically.

One of the tenets of quantitative portfolio management is: the average return of an investment is the payoff for taking risk. The risk of a stock has two components:

Total Risk = Non-Diversifiable Risk + Diversifiable Risk

Risk can be measured using the variance of the returns:

$$ V(r_i) = V(\alpha + \beta_{i1}f_1 + ... + \beta_{iK}f_K) + V(\epsilon_i) + V(\epsilon_i)$$.

This can be expressed in the matrix notation as:

$$ V(r_i) = \mathbf{\beta}_i^T V(\mathbf{f}) \mathbf{\beta}_i + V(\epsilon_i) $$

where $V(\mathbf{f})$ is a $(K+1) \times (K+1)$-dimensional variance-covariance matrix. The diversifiable component which an investor can mitigate by diversification is the term $V(\epsilon_i)$. The non-diversifiable componenent which the market rewards is obtained from factor exposures $\mathbf{\beta}_i$ and factor premium risk $V(\mathbf{f})$.

Preparatory Work

There are a series of steps that one has to do before we can run a multi-variable regression to determine the factor premiums.

Choosing the factors

We need to choose the factors that we wish to include in the model. This step is usually driven by the intuition an investor has on the market, and characteristics of the stock that are important. For instance, some fundamental factors that would make sense for equity investments include:

Valuation Factors: Market capitalization (size), price-to-book ratio (P/B), price-to-earnings ratio (P/E), price-to-sales ratio (P/S).
Solvency Factors: Debt-to-equity ratio (D/E), current ratio (CUR, a metric of short term solvency), interest coverage ratio.
Growth Potential Factors: Capital-expenditure-to-sales ratio, R&D-expenditure-to-sales ratio, advertising-expenditure-to-sales ratio.
Technical factors: Momentum, trading volume, short interest shares shorted
Analyst factors: Analyst rating changes, earnings revision etc.

Similarly some fundamental factors that would make sense for fixed income investments include:

Credit Factors: Credit quality, different vendor ratings
Interest Rate Sensitivity Factors: Effective duration, key rate duration at various tenors, convexity
Asset Type Factors: Some factors can be asset specific, such as:
1. Municipals:
  1. State or issuer
  2. Purpose such as revenue or general obligation
2. Mortgages:
  1. Coupon
  2. Amortization type
  3. Issuer

Treatment of the Risk-Free Rate

Since we are building a model to measure the payoff for undertaking risk, it is conventional to subtract the risk-free rate from the returns of the investment. The rational is that the risk-free rate is the return that can be obtained on any investment without any risk. In order to do this, we need to define what is considered risk-free. In the U.S. markets, the U.S. treasury bills are used as a proxy for the risk free rate. In other markets, one has to exercise care in identifying a risk free rate. All sovereign bond/bill rates do not immediately translate into risk-free rates because a lot of countries (developing and developed) have risk of default as well.

Once the risk-free rate $r_{ft}$ has been identified, the excess return can be defined as

$$ r_{it}' = r_{it} - r_{ft} $$

Building the Data Sample

One of the crucial components in the factor model building process is to create a data sample that can then be used to deduce the model parameters. For example, to model the U.S. equity market one can use the top 2000 to 3000 stocks by market cap. Depending on where we cutoff, this will include stocks in the large-cap and mid-cap range. This would mean that the model may not precisely model the effects in the small-cap stocks. On the other hand, limiting to stocks in the mid-cap or large-cap range can provide more stability to the model, and that could be a desirable thing.

Once we have a filter for deciding how we want to build the basket of instruments for model building, we also have to decide the frequency and period for which we need to collect the data. What do we mean by that? The regression step involved in the model building will fit the stock returns $r_{it}$ for a period (such as one day returns or monthly returns) as a function of various factors $f$. The return data is collected over multiple periods, such as 3 years or 5 years. For instance, our data sample could be top 3000 stocks by market cap with monthly returns collected for 3 years. A rule of thumb is that the frequency of the investor's re-balancing strategy should determine the frequency of the returns used in the model building. If the re-balances monthly, then one should use monthly returns in the model fitting. There should be sufficient time periods included in order to avoid spurious fitting effects and to improve robustness of fit. It is conventional to use 3 to 5 years worth of data while using monthly returns, and 2 to 3 years while using weekly returns.

Another factor that one should keep in mind is to include companies that are not only alive at the time of model building but also to include companies that may have gotten delisted or bankrupt. Failure to do so will introduce what is called as survivorship bias in your model.

Fitting the Model

Once we have the data sample assembled, one can use regression utilities in statistical packages such as R, Python's Statsmodels package, Matlab, SAS to perform a panel regression. The goal of the panel regression is to estimate factor premium $\mathbf{f}$ given by the equation

$$ r_i = \mathbf{\beta}^T_i \mathbf{f} + \epsilon_i $$

where the returns $r_i$ and the factor exposures $\mathbf{\beta}_i$ are known. A simple way to estimate factor premiums is by using Ordinary Least Squares (OLS). This will also give us an estimate of the sample variance-covariance matrix $V(\mathbf{\hat{f}})$.

\begin{eqnarray} \mathbf{f} &=& \left( \sum_{t=1}^T \sum_{i=1}^N (\mathbf{\beta}_{it} - \bar{\mathbf{\beta}} ) ( \mathbf{\beta}_{it} - \bar{\mathbf{\beta}} )^T \right)^{-1} \sum_{t=1}^{T} \sum_{i=1}^{N} (\mathbf{\beta}_{it} - \bar{\mathbf{\beta}} ) (r_{it} - \bar{r}) \end{eqnarray} where

$$ \bar{\mathbf{\beta}} = \frac{1}{NT} \sum_{t=1}^{T} \sum_{i=1}^{N} \mathbf{\beta}_{it} $$ and $$ \bar{r} = \frac{1}{NT} \sum_{t=1}^{T} \sum_{i=1}^{N} r_{it} $$

The variance-covariance matrix of the factor premium estimate is given as:

\begin{eqnarray} \hat{V}(\hat{f}) &=& \hat{\sigma}^2 \left( \sum_{t=1}^T \sum_{i=1}^N
(\mathbf{\beta}_{it} - \bar{\mathbf{\beta}} ) ( \mathbf{\beta}_{it} - \bar{\mathbf{\beta}} )^T \right) \end{eqnarray}

where $\hat{\sigma}^2$ is the variance of the error term $\epsilon_{it}$ $$ \hat{\sigma}^2 = \frac{1}{NT} \sum_{t=1}^T \sum_{i=1}^N
(r_{it} - \mathbf{\beta}_{it}\hat{f}) $$

Robustness Considerations

It is important that the estimates of the factor premium and the variance of factor premiums are robust. One way to accomplish that is by using generalized least squares (GLS). Corrections for heteroskedasticity and autocorrelation can be incorporated by using the Newey-West Estimator to correct for heteroskedacity and auto correlation. The statistical package R has an implementation of Newey-West in the sandwich package. Covariance Shrinkage was suggested by Ledoit and Wolf as a way to reduce estimation error in the Covariance Matrix. This technique helps reduce the extremes in the covariance matrix coefficients towards a central value, and is necessary in the portfolio optimization process.

Conclusions

Here we gave an introduction to the process of building a factor model. This text is informative for some one to get started. There are lots of nuances that one needs to consider to incorporate the intuition of investment managers and follow the wind as the market currents change.

On the Convergence of Hull White Monte Carlo Simulations

Gouthaman Balaraman — Fri, 22 May 2015 00:00:00 -0700

I had recently written an introductory post on simulating short rates in the Hull-White Model. This question on the QuantLib forum raised some interesting questions on the convergence of the Hull-White model simulations. In this post, I discuss the convergence of Monte-Carlo simulations using the Hull-White model.

The Hull-White Short Rate Model is defined as:

$$ dr_t = (\theta(t) - a r_t)dt + \sigma dW_t $$,

where $a$ and $\sigma $ are constants, and $\theta(t)$ is chosen in order to fit the input term structure of interest rates. Here we use QuantLib to show how to simulate the Hull-White model and investigate some of the properties.

The variables used in this post are described below:

timestep is the number of steps used to discretize the time grid
hw_process the object that defines the Hull-White process,
length is the time span of the simulation in years
low_discrepancy is a boolean variable that is used to chose Sobol low discrepancy random or not
brownnian_bridge is a boolean that choses brownian bridge for path generation
num_paths is the number of paths in the simulation
a is the constant parameter in the Hull-White model
sigma is the constant parameter $\sigma$ in the Hull-White model that describes volatility

In [1]:

Expand Code

The get_path_generator function creates the a path generator. This function takes various inputs such as

In [2]:

Expand Code

The generate_paths function uses the generic path generator produced by the get_path_generator function to return a tuple of the array of the points in the time grid and a matrix of the short rates generated.

In [3]:

Expand Code

The generate_paths_zero_price essentially is a wrapper around generate_path_generator and generate_paths taking all the required raw inputs. This function returns the average of zero prices from all the paths for different points in time. I wrote this out so that I can conveniently change all the required inputs and easily plot the results.

In [4]:

Expand Code

def generate_paths_zero_price(spot_curve_handle, a, sigma, timestep, length, 
                              num_paths, avg_grid_array, low_discrepancy=False, 
                              brownian_bridge=True):
    """
    This function returns a tuple (T_array, F_array), where T_array is the array 
    of points in the time grid, and F_array is the array of the average of zero 
    prices observed from the simulation.
    """
    hw_process = ql.HullWhiteProcess(spot_curve_handle, a, sigma)
    seq = get_path_generator(
            timestep, hw_process, length, low_discrepancy, brownian_bridge
    )
    time, paths = generate_paths(num_paths, timestep, seq)
    avgs = [(time[j],(np.mean([math.exp(-simps(paths[i][0:j], time[0:j])) 
                               for i in range(num_paths)]))) 
            for j in avg_grid_array
            ]
    return zip(*avgs)

def generate_paths_discount_factors(spot_curve_handle, a, sigma, timestep, length, 
                              num_paths, avg_grid_array, low_discrepancy=False, 
                              brownian_bridge=True):
    """
    This function returns a tuple (T_array, S_matrix), where T_array is the array 
    of points in the time grid, and S_matrix is the matrix of the spot rates for 
    each path in the different points in the time grid.
    """
    hw_process = ql.HullWhiteProcess(spot_curve_handle, a, sigma)
    seq = get_path_generator(
            timestep, hw_process, length, low_discrepancy, brownian_bridge
    )
    time, paths = generate_paths(num_paths, timestep, seq)
    arr = np.zeros((num_paths, len(avg_grid_array)))
    for i in range(num_paths):
        arr[i, :] =  [np.exp(-simps(paths[i][0:j], time[0:j])) for j in avg_grid_array ]
    t_array = [time[j] for j in avg_grid_array]
    return t_array, arr

def V(t,T, a, sigma):
    """ Variance of the integral of short rates, used below"""
    return sigma*sigma/a/a*(T-t + 2.0/a*math.exp(-a*(T-t)) - 
                            1.0/(2.0*a)*math.exp(-2.0*a*(T-t)) - 3.0/(2.0*a) )

Factors affecting the convergence

In order to understand the convergence of Monte-Carlo for the Hull-White model, let us compare the market discount factor,

$$ P^M(t, T) = \exp\left(-\int_t^Tf^{M} (t,u)du\right)$$

with the expectation of the discount factors from the sample of Monte-Carlo paths,

$$P^{MC}(t,T) = E_t\left\{ e^{-\int_t^T r^{MC}(u) du} \right\}$$.

Here $f^{M}(t, T)$ is the instantaneous forward rate implied by the market, and $r^{MC}(s)$ is the instantaneous short rate from the Monte-Carlo simulations. The error in the Monte-Carlo simulation can be defined as:

$$ \epsilon(T) = P^M(0,T) - P^{MC}(0,T) $$

As a first step, let us look at the plots of $\epsilon(t)$ for different values of $a$ and $\sigma$.

In [5]:

Expand Code

The above plot illustrates that for $\sigma=0.01$, the Monte-Carlo model shows good convergence, and the convergence gradually deteriorates as $\sigma$ increases and approaches $a$.

In [6]:

Expand Code

The above plot illustrates that for $a=0.1$ the convergence of Monte-Carlo is poor, and it gradually improves as $a$ increases more than $\sigma$.

From the plots above, we observe that the convergence is good if the ratio $\sigma/a < 1$, and the convergence detiorates as the ratio $\sigma/a$ increases above unity. Now, let us try to formalize this observation from the theoretical footing of the Hull-White model.

Distribution of Discount Factors

The Monte-Carlo approach estimates the market discount factor as the expectation of discount factors from each Monte-Carlo path. If distribution of discount factors has a standard deviation $\sigma_D$, then the error in our estimate of $P^{MC}(t,T)$ on using $N$ paths will be of the order of: $$\epsilon(t,T) \approx \frac{\sigma_D}{\sqrt(N)}. $$

In other words, there are two factors at play in our Monte-Carlo estimate, the number of Monte-Carlo paths $N$ and the standard deviation of the distribution of discount factors $\sigma$. Using more Monte-Carlo paths will lead to improved convergence. But at the same time, the $\sigma_D$ has to be relatively small for us to get a good estimate.

The integral of short rates can be shown to be normally distributed (refer Brigo-Mercurio, second edition page 75), and is given as $$ \int_t^T r(u) du | \mathcal{F}_t \sim \mathcal{N}\left(B(t,T)[r(t)-\alpha(t)] + \ln\frac{P^M(0,t)}{P^M(0,T)} + \frac{1}{2}[V(0,T) - V(0,t)], V(t,T)\right)$$

where, \begin{eqnarray} B(t,T) &=& \frac{1}{a} \left[ 1 - e^{-a(T-t)}\right] \\ V(t,T) &=& \frac{\sigma^2}{a^2}\left[ T - t + \frac{2}{a}e^{-a(T-t)} - \frac{1}{2a}e^{-2a(T-t)} - \frac{3}{2a}\right] \end{eqnarray}

Based on this result, the discount factor from the Monte-Carlo simulation of short rates

$$ P^{MC}(t, T) = \exp\left(- \int_t^T r(u) du | \mathcal{F}_t \right)$$

will have a log-normal distribution with a standard deviation

$$ \sigma_D(t,T) = P^M(t,T)\sqrt{e^{V(t,T)} -1 }$$

This result follows from the fact that if $X$ is a random process with a normal distribution having mean $\mu$ and standard deviation $\sigma$, then log-normal distribution $Y=e^X$ will satisfy: \begin{eqnarray} E(Y) &=& e^{\mu+\sigma^2/2}\\ Var(Y) &=& (e^{\sigma^2} -1 )E(Y)^2 \end{eqnarray}

In [7]:

Expand Code

The plot above compares the standard deviation of the discount factors $\sigma_D$ from the closed form expression with a Monte-Carlo estimate. The empirical estimate is in agreement with the theoretical expectation. We can estimate the value of $\sigma_D$ for the asymptotic limit of $T\rightarrow\infty$:

$$ \sigma_D(0, T) \approx P^M(0,T)e^{f^M(0,T)/a-\sigma^2/(4a^3)}\sqrt{e^{\sigma^2T/a^2} - 1}$$

The exponential term, $e^{\sigma^2T/a^2}$, can become very large when $\sigma^2T/a^2$ grows above 1. Thus we can expect good convergence when $\sigma^2T/a^2$ remains small or close to zero for the time $T$ of interest to us.

The above result suggests that if the parameters $\sigma$ and $a$ are not chosen carefully, (i.e. $\sigma/a >1$) then the convergence of the simulation would be poor and the results untrustworthy.

Conclusion

This post shows how convergence of Monte Carlo simulations of Hull-White short rate model is dependent on the ratio $\sigma /a$. You can download the ipython notebook on convergence of Monte Carlo simulations of Hull-White model.

Option Model Handbook, Part III: European Option Pricing With QuantLib Python

Gouthaman Balaraman — Fri, 08 May 2015 00:00:00 -0700

I have written about option pricing earlier. The introduction to option pricing gave an overview of the theory behind option pricing. The post on introduction to binomial trees outlined the binomial tree method to price options.

In this post, we will use QuantLib and the Python extension to illustrate a very simple example. Here we are going to price a European option using the Black-Scholes-Merton formula. We will price them again using the Binomial tree and understand the agreement between the two.

In [1]:

import QuantLib as ql # version 1.5
import matplotlib.pyplot as plt
%matplotlib inline

In [2]:

# option data
maturity_date = ql.Date(15, 1, 2016)
spot_price = 127.62
strike_price = 130
volatility = 0.20 # the historical vols for a year
dividend_rate =  0.0163
option_type = ql.Option.Call

risk_free_rate = 0.001
day_count = ql.Actual365Fixed()
calendar = ql.UnitedStates()

calculation_date = ql.Date(8, 5, 2015)
ql.Settings.instance().evaluationDate = calculation_date

We construct the European option here.

In [3]:

# construct the European Option
payoff = ql.PlainVanillaPayoff(option_type, strike_price)
exercise = ql.EuropeanExercise(maturity_date)
european_option = ql.VanillaOption(payoff, exercise)

The Black-Scholes-Merto process is constructed here.

In [4]:

spot_handle = ql.QuoteHandle(
    ql.SimpleQuote(spot_price)
)
flat_ts = ql.YieldTermStructureHandle(
    ql.FlatForward(calculation_date, risk_free_rate, day_count)
)
dividend_yield = ql.YieldTermStructureHandle(
    ql.FlatForward(calculation_date, dividend_rate, day_count)
)
flat_vol_ts = ql.BlackVolTermStructureHandle(
    ql.BlackConstantVol(calculation_date, calendar, volatility, day_count)
)
bsm_process = ql.BlackScholesMertonProcess(spot_handle, 
                                           dividend_yield, 
                                           flat_ts, 
                                           flat_vol_ts)

Lets compute the theoretical price using the AnalyticEuropeanEngine.

In [5]:

european_option.setPricingEngine(ql.AnalyticEuropeanEngine(bsm_process))
bs_price = european_option.NPV()
print "The theoretical price is ", bs_price

The theoretical price is  6.74927181246

Lets compute the price using the binomial-tree approach.

In [6]:

def binomial_price(bsm_process, steps):
    binomial_engine = ql.BinomialVanillaEngine(bsm_process, "crr", steps)
    european_option.setPricingEngine(binomial_engine)
    return european_option.NPV()

steps = range(2, 100, 1)
prices = [binomial_price(bsm_process, step) for step in steps]

In the plot below, we show the convergence of binomial-tree approach by comparing its price with the BSM price.

In [8]:

plt.plot(steps, prices, label="Binomial Tree Price", lw=2, alpha=0.6)
plt.plot([0,100],[bs_price, bs_price], "r--", label="BSM Price", lw=2, alpha=0.6)
plt.xlabel("Steps")
plt.ylabel("Price")
plt.title("Binomial Tree Price For Varying Steps")
plt.legend()

Out[8]:

<matplotlib.legend.Legend at 0x7f0b85fa7510>

Conclusion

This post shows how to price European Options using the theoretical and binomial-tree methods in QuantLib Python. You can download the ipython notebook on European option pricing with QuantLib.

Hull White Term Structure Simulations with QuantLib Python

Gouthaman Balaraman — Fri, 01 May 2015 00:00:00 -0700

The Hull-White Short Rate Model is defined as:

$$ dr_t = (\theta(t) - a r_t)dt + \sigma dW_t $$

where $a$ and $ \sigma $ are constants, and $\theta(t)$ is chosen in order to fit the input term structure of interest rates. Here we use QuantLib to show how to simulate the Hull-White model and investigate some of the properties.

We import the libraries and set things up as shown below:

In [1]:

import QuantLib as ql
import matplotlib.pyplot as plt
import numpy as np
% matplotlib inline

The constants that we use for this example is all defined as shown below. Variables sigma and a are the constants that define the Hull-White model. In the simulation, we discretize the time span of length 30 years into 360 intervals (one per month) as defined by the timestep variable. For simplicity we will use a constant forward rate term structure as an input. It is straight forward to swap with another term structure here.

In [2]:

sigma = 0.1
a = 0.1
timestep = 360
length = 30 # in years
forward_rate = 0.05
day_count = ql.Thirty360()
todays_date = ql.Date(15, 1, 2015)

In [3]:

ql.Settings.instance().evaluationDate = todays_date

spot_curve = ql.FlatForward(todays_date, ql.QuoteHandle(ql.SimpleQuote(forward_rate)), day_count)
spot_curve_handle = ql.YieldTermStructureHandle(spot_curve)

In [4]:

hw_process = ql.HullWhiteProcess(spot_curve_handle, a, sigma)
rng = ql.GaussianRandomSequenceGenerator(ql.UniformRandomSequenceGenerator(timestep, ql.UniformRandomGenerator()))
seq = ql.GaussianPathGenerator(hw_process, length, timestep, rng, False)

The Hull-White process is constructed by passing the term-structure, a and sigma. To create the path generator, one has to provide a random sequence generator along with other simulation inputs such as timestep and `length.

A function to generate paths can be written as shown below:

In [5]:

def generate_paths(num_paths, timestep):
    arr = np.zeros((num_paths, timestep+1))
    for i in range(num_paths):
        sample_path = seq.next()
        path = sample_path.value()
        time = [path.time(j) for j in range(len(path))]
        value = [path[j] for j in range(len(path))]
        arr[i, :] = np.array(value)
    return np.array(time), arr

The simulation of the short rates look as shown below:

In [6]:

num_paths = 10
time, paths = generate_paths(num_paths, timestep)
for i in range(num_paths):
    plt.plot(time, paths[i, :], lw=0.8, alpha=0.6)
plt.title("Hull-White Short Rate Simulation")
plt.show()

The short rate $r(t)$ is given a distribution with the properties:

$$ E\{r(t) | F_s\} = r(s)e^{-a(t-s)} + \alpha(t) - \alpha(s)e^{-a(t-s)} $$ $$ Var\{ r(t) | F_s \} = \frac{\sigma^2}{2a} [1 - e^{-2a(t-s)}] $$ where $$ \alpha(t) = f^M(0, t) + \frac{\sigma^2} {2a^2}(1-e^{-at})^2$$

as shown in Brigo & Mercurio's book on Interest Rate Models.

In [7]:

num_paths = 1000
time, paths = generate_paths(num_paths, timestep)

The mean and variance compared between the simulation (red dotted line) and theory (blue line).

In [8]:

vol = [np.var(paths[:, i]) for i in range(timestep+1)]
plt.plot(time, vol, "r-.", lw=3, alpha=0.6)
plt.plot(time,sigma*sigma/(2*a)*(1.0-np.exp(-2.0*a*np.array(time))), "b-", lw=2, alpha=0.5)
plt.title("Variance of Short Rates")

Out[8]:

<matplotlib.text.Text at 0x7f555d561ad0>

In [9]:

def alpha(forward, sigma, a, t):
    return forward + 0.5* np.power(sigma/a*(1.0 - np.exp(-a*t)), 2)

avg = [np.mean(paths[:, i]) for i in range(timestep+1)]
plt.plot(time, avg, "r-.", lw=3, alpha=0.6)
plt.plot(time,alpha(forward_rate, sigma, a, time), "b-", lw=2, alpha=0.6)
plt.title("Mean of Short Rates")

Out[9]:

<matplotlib.text.Text at 0x7f555d6a9e10>

Conclusion

This post shows how to simulate Hull-White short rate model using QuantLib Python. You can download the ipython notebook on Hull-White simulations.

An Introduction to Interest Rate Term Structure in QuantLib Python

Goutham Balaraman — Wed, 08 Apr 2015 00:00:00 -0700

Term structure is pivotal to pricing securities. One would need a YieldTermStructure object created in QuantLib to use with pricing engines. In an earlier post on modeling bonds using QuantLib we discussed how to use spot rates directly with bond pricing engine. Here in this post we will show how to bootstrap yield curve using QuantLib.

As usual lets import QuantLib and do some initialization.

import QuantLib as ql

def print_curve(xlist, ylist, precision=3):
    """
    Method to print curve in a nice format
    """
    print "----------------------"
    print "Maturities\tCurve"
    print "----------------------"
    for x,y in zip(xlist, ylist):
        print x,"\t\t", round(y, precision)
    print "----------------------"

The deposit rates and fixed rate bond rates are provided below. This example is based on Exhibit 5-5 given in Frank Fabozzi's Bond Markets, Analysis and Strategies, Sixth Edition.

# Deposit rates
depo_maturities = [ql.Period(6,ql.Months), ql.Period(12, ql.Months)]
depo_rates = [5.25, 5.5]

# Bond rates
bond_maturities = [ql.Period(6*i, ql.Months) for i in range(3,21)]
bond_rates = [5.75, 6.0, 6.25, 6.5, 6.75, 6.80, 7.00, 7.1, 7.15,
              7.2, 7.3, 7.35, 7.4, 7.5, 7.6, 7.6, 7.7, 7.8]

print_curve(depo_maturities+bond_maturities, depo_rates+bond_rates)

Lets define some of the constants required for the rest of the objects needed below.

# some constants and conventions
# here we just assume for the sake of example
# that some of the constants are the same for
# depo rates and bond rates

calc_date = ql.Date(15, 1, 2015)
ql.Settings.instance().evaluationDate = calc_date

calendar = ql.UnitedStates()
bussiness_convention = ql.Unadjusted
day_count = ql.Thirty360()
end_of_month = True
settlement_days = 0
face_amount = 100
coupon_frequency = ql.Period(ql.Semiannual)
settlement_days = 0

The basic idea of bootstrapping using QuantLib is to use the deposit rates and bond rates to create individual helpers. Then use the combination of the two helpers to construct the yield curve.

# create deposit rate helpers from depo_rates
depo_helpers = [ql.DepositRateHelper(ql.QuoteHandle(ql.SimpleQuote(r/100.0)),
                                     m,
                                     settlement_days,
                                     calendar,
                                     bussiness_convention,
                                     end_of_month,
                                     day_count )
                for r, m in zip(depo_rates, depo_maturities)]

The rest of the points are coupon bonds. We assume that the YTM given for the bonds are all par rates. So we have bonds with coupon rate same as the YTM.

# create fixed rate bond helpers from fixed rate bonds
bond_helpers = []
for r, m in zip(bond_rates, bond_maturities):
    termination_date = calc_date + m
    schedule = ql.Schedule(calc_date,
                   termination_date,
                   coupon_frequency,
                   calendar,
                   bussiness_convention,
                   bussiness_convention,
                   ql.DateGeneration.Backward,
                   end_of_month)

    helper = ql.FixedRateBondHelper(ql.QuoteHandle(ql.SimpleQuote(face_amount)),
                                        settlement_days,
                                        face_amount,
                                        schedule,
                                        [r/100.0],
                                        day_count,
                                        bussiness_convention,
                                        )
    bond_helpers.append(helper)

The yield curve is constructed by putting the two helpers together.

rate_helpers = depo_helpers + bond_helpers
yieldcurve = ql.PiecewiseLogCubicDiscount(calc_date,
                             rate_helpers,
                             day_count)

The spot rates is obtined from yieldcurve object using the zeroRate method.

# get spot rates
spots = []
tenors = []
for d in yieldcurve.dates():
    yrs = day_count.yearFraction(calc_date, d)
    compounding = ql.Compounded
    freq = ql.Semiannual
    zero_rate = yieldcurve.zeroRate(yrs, compounding, freq)
    tenors.append(yrs)
    eq_rate = zero_rate.equivalentRate(day_count,
                                       compounding,
                                       freq,
                                       calc_date,
                                       d).rate()
    spots.append(100*eq_rate)

The bootstrap curve looks as shown below:

Maturity	Spots
0.0	0.0
0.5	5.25
1.0	5.426
1.5	5.761
2.0	6.02
2.5	6.283
3.0	6.55
3.5	6.822
4.0	6.87
4.5	7.095
5.0	7.205
5.5	7.257
6.0	7.31
6.5	7.429
7.0	7.485
7.5	7.543
8.0	7.671
8.5	7.802
9.0	7.791
9.5	7.929
10.0	8.072

Once we have the spots, the zero coupon curve can be directly constructed the next time as show in the bond pricing example. The yieldcurve.dates() and yieldcuve.zeroRate(...) methods would provide for the necessary rates as shown above.

Conclusion

In this post we showed how to bootstrap yield curve to get spot rates.

Download the bootstrap yield curve ipython notebook.

Modeling Fixed Rate Bonds in QuantLib Python

Goutham Balaraman — Mon, 30 Mar 2015 00:00:00 -0700

Let's consider a hypothetical bond with a par value of 100, that pays 6% coupon semi-annually issued on January 15th, 2015 and set to mature on January 15th, 2016. The bond will pay a coupon on July 15th, 2015 and January 15th, 2016. The par amount of 100 will also be paid on the January 15th, 2016.

To make things simpler, lets assume that we know the spot rates of the treasury as of January 15th, 2015. The annualized spot rates are 0.5% for 6 months and 0.7% for 1 year point. Lets calculate the fair value of this bond.

>>> 3/pow(1+0.005, 0.5) + (100 + 3)/(1+0.007)
105.27653992490681

Lets calculate the same thing using QuantLib.

>>> import QuantLib as ql
>>> todaysDate = ql.Date(15, 1, 2015)
>>> ql.Settings.instance().evaluationDate = todaysDate
>>> spotDates = [ql.Date(15, 1, 2015), ql.Date(15, 7, 2015), ql.Date(15, 1, 2016)]
>>> spotRates = [0.0, 0.005, 0.007]
>>> dayCount = ql.Thirty360()
>>> calendar = ql.UnitedStates()
>>> interpolation = ql.Linear()
>>> compounding = ql.Compounded
>>> compoundingFrequency = ql.Annual
>>> spotCurve = ql.ZeroCurve(spotDates, spotRates, dayCount, calendar, interpolation,
                             compounding, compoundingFrequency)
>>> spotCurveHandle = ql.YieldTermStructureHandle(spotCurve)

So far we have created the term structure and the variables are rather self explanatory. Now lets construct the fixed rate bond.

>>> issueDate = ql.Date(15, 1, 2015)
>>> maturityDate = ql.Date(15, 1, 2016)
>>> tenor = ql.Period(ql.Semiannual)
>>> calendar = ql.UnitedStates()
>>> bussinessConvention = ql.Unadjusted
>>> dateGeneration = ql.DateGeneration.Backward
>>> monthEnd = False
>>> schedule = ql.Schedule (issueDate, maturityDate, tenor, calendar, bussinessConvention,
                            bussinessConvention , dateGeneration, monthEnd)
>>> list(schedule)
[Date(15, 1, 12015), Date(15,7,2015), Date(15,1,2016)]


# Now lets build the coupon
>>> dayCount = ql.Thirty360()
>>> couponRate = .06
>>> coupons = [couponRate]

# Now lets construct the FixedRateBond
>>> settlementDays = 0
>>> faceValue = 100
>>> fixedRateBond = ql.FixedRateBond(settlementDays, faceValue, schedule, coupons, dayCount)

# create a bond engine with the term structure as input;
# set the bond to use this bond engine
>>> bondEngine = ql.DiscountingBondEngine(spotCurveHandle)
>>> fixedRateBond.setPricingEngine(bondEngine)

# Finally the price
>>> fixedRateBond.NPV()
105.27653992490683

Voila!

Download the modeling bonds ipython notebook.

Introduction to QuantLib Python

Goutham Balaraman — Tue, 24 Mar 2015 00:00:00 -0700

I installed the latest version of QuantLib (V1.5) and the python wrapper to QuantLib. My experiments lately have been to get a feel for the QuantLib API. The library itself is so extensive, that it is rather hard for a new comer to get going. In this post we will look into some of the basic classes and functionality in QuantLib.

Let us import QuantLib as:

import QuantLib as ql

Time SubModule

The ql/time sub-folder implements various time related classes. Lets take a look at the Date object which can be constructed as Date(date, month, year).

>>> date = ql.Date(31, 3, 2015) # 31 March, 2015
>>> print date
March 31st, 2015
>>> date.dayOfMonth()
31
>>> date.month()
3
>>> date.year()
2015
>>> date.weekday() == ql.Tuesday
True

# arithmetic with dates
>>> date + 1  # add a day
Date(1,4,2015)
>>> date - 1  # subtract a day
Date(30,3,2015)
>>> date + ql.Period(1, ql.Months)
Date(30,4,2015)
>>> date + ql.Period(1, ql.Weeks)
Date(7,4,2015)
>>> date + ql.Period(1, ql.Years)
Date(31,3,2016)

# logical operations
>>> ql.Date(31, 3, 2015) > ql.Date(1, 3, 2015)
True

The Schedule object can be used to construct a list of dates such as coupon payments. Lets look at some examples.

>>> date1 = ql.Date(1, 1, 2015)
>>> date2 = ql.Date(1, 1, 2016)
>>> tenor = ql.Period(ql.Monthly)
>>> calendar = ql.UnitedStates()
>>>
>>> schedule = ql.Schedule(date1, date2, tenor, calendar, ql.Following,
                           ql.Following, ql.DateGeneration.Forward, False)
>>> list(schedule)
[Date(2,1,2015),
 Date(2,2,2015),
 Date(2,3,2015),
 Date(1,4,2015),
 Date(1,5,2015),
 Date(1,6,2015),
 Date(1,7,2015),
 Date(3,8,2015),
 Date(1,9,2015),
 Date(1,10,2015),
 Date(2,11,2015),
 Date(1,12,2015),
 Date(4,1,2016)]

Here we have generated a Schedule object that will contain dates between date1 and date2 with the tenor specifying the Period to be every Month. The calendar object is used for determining holidays. The two arguments following the calendar in the Schedule constructor are the BussinessDayConvention. Here we chose the convention to be the day following holidays. That is why we see that holidays are excluded in the list of dates.

Interest Rate

The InterestRate class can be used to store the interest rate with the compounding type, day count and the frequency of compounding. Below we show how to create an interest rate of 5.0% compounded annually, using Actual/Actual day count convention.

>>> annualRate = 0.05
>>> dayCount = ql.ActualActual()
>>> compoundType = ql.Compounded
>>> frequency = ql.Annual
>>> interestRate = ql.InterestRate(annualRate, dayCount, compoundType, frequency)

Lets say if you invest a dollar at the interest rate described by interestRate, the compoundFactor method gives you how much your investment will be worth after t years. Below we show that the value returned by compoundFactor for 2 years agrees with the expected compounding formula.

>>> interestRate.compoundFactor(2.0)
1.1025
>>> (1.0 + annualRate)*(1.0 + annualRate)  # Check the above calculation
1.1025

The discountFactor method returns the reciprocal of the compoundFactor method. The discount factor is useful while calculating the present value of future cashflows.

>>> interestRate.discountFactor(2.0)
0.9070294784580498
>>> 1.0 / interestRate.compoundFactor(2.0)
0.9070294784580498

A given interest rate can be converted into other types using the equivalentRate method as :

>>> newFrequency = ql.Semiannual
>>> effectiveRate = interestRate.equivalentRate(compoundType, newFrequency, 1)
>>> effectiveRate.rate()
0.04939015319191986

The InterestRate class also has an impliedRate method. The impliedRate method takes compound factor to return the implied rate. The impliedRate method is a static method in the InterestRate class and can be used without an instance of InterestRate. Internally the equivalentRate method invokes the impliedRate method in its calculations.

Here we have converted into a semi-annual compounding type. A 4.939% of semi-annual compounding is equivalent to 5.0% annual compounding. This should mean, that both should give identical discount factors. Lets check that:

>>> interestRate.discountFactor(1.0)
0.9523809523809523
>>> effectiveRate.discountFactor(1.0)
0.9523809523809521

So this means that pricing bonds using either interest rate convention should give the same net present value (barring some precision).

Conclusion

In this post we looked at the basics of QuantLib:

We learnt how to use Date and Schedule classes from the time sub-module
we learnt how to use the InterestRate class

Using SQLAlchemy in Luigi Workflow Pipeline

Gouthaman Balaraman — Thu, 05 Feb 2015 00:00:00 -0800

The luigi.contrib.sqla provides support for SQLAlchmey through the SQLAlchemyTarget for storing in databases supported by SQLAlchemy. The user would be responsible for installing the required database driver to connect using SQLAlchemy.

Minimal example of a job to copy data to database using SQLAlchemy is as shown below:

from sqlalchemy import String
import luigi
from luigi.contrib import sqla

class SQLATask(sqla.CopyToTable):
    # columns defines the table schema, with each element corresponding
    # to a column in the format (args, kwargs) which will be sent to
    # the sqlalchemy.Column(*args, **kwargs)
    columns = [
        (["item", String(64)], {"primary_key": True}),
        (["property", String(64)], {})
    ]
    connection_string = "sqlite://"  # in memory SQLite database
    table = "item_property"  # name of the table to store data

    def rows(self):
        for row in [("item1" "property1"), ("item2", "property2")]:
            yield row

if __name__ == '__main__':
    task = SQLATask()
    luigi.build([task], local_scheduler=True)

If the target table where the data needs to be copied already exists, then the column schema definition can be skipped and instead the reflect flag can be set as True. Here is a modified version of the above example:

from sqlalchemy import String
import luigi
from luigi.contrib import sqla

class SQLATask(sqla.CopyToTable):
    # If database table is already created, then the schema can be loaded
    # by setting the reflect flag to True
    reflect = True
    connection_string = "sqlite://"  # in memory SQLite database
    table = "item_property"  # name of the table to store data

    def rows(self):
        for row in [("item1" "property1"), ("item2", "property2")]:
            yield row

if __name__ == '__main__':
    task = SQLATask()
    luigi.build([task], local_scheduler=True)

In the above examples, the data that needs to be copied was directly provided by overriding the rows method. Alternately, if the data comes from another task, the modified example would look as shown below:

from sqlalchemy import String
import luigi
from luigi.contrib import sqla
from luigi.mock import MockFile

class BaseTask(luigi.Task):
    def output(self):
        return MockFile("BaseTask")

    def run(self):
        out = self.output().open("w")
        TASK_LIST = ["item%d\tproperty%d\n" % (i, i) for i in range(10)]
        for task in TASK_LIST:
            out.write(task)
        out.close()

class SQLATask(sqla.CopyToTable):
    # columns defines the table schema, with each element corresponding
    # to a column in the format (args, kwargs) which will be sent to
    # the sqlalchemy.Column(*args, **kwargs)
    columns = [
        (["item", String(64)], {"primary_key": True}),
        (["property", String(64)], {})
    ]
    connection_string = "sqlite://"  # in memory SQLite database
    table = "item_property"  # name of the table to store data

    def requires(self):
        return BaseTask()

if __name__ == '__main__':
    task1, task2 = SQLATask(), BaseTask()
    luigi.build([task1, task2], local_scheduler=True)

In the above example, the output from BaseTask is copied into the database. Here we did not have to implement the rows method because by default rows implementation assumes every line is a row with column values separated by a tab. One can define column_separator option for the task if the values are say comma separated instead of tab separated.

The other option to sqla.CopyToTable that can be of help with performance aspect is the chunk_size. The default is 5000. This is the number of rows that will be inserted in a transaction at a time. Depending on the size of the inserts, this value can be tuned for performance.

Building Task Pipelines Using Luigi

Gouthaman Balaraman — Wed, 31 Dec 2014 00:00:00 -0800

It is incredibly easy to write a script to process some data in python. But if you have a lot of tasks that depend on each other, and you need to create a robust work flow, then thinking in terms of a data pipeline is useful.

Luigi is a framework for building data pipelines, and managing workflows. The onus of setting up each unit of work is on you as the developer. Luigi taks care of resolving dependencies, manages the overall workflow, and most importantly handles failures. As a bonus Luigi provides a rather nice visualization tool and a command line interface.

Luigi Basics

In Luigi, a data pipeline is built by defining Task instances. For every Task, you can define its dependency by specifying the requires method for the Task. Every Task can define an output method to specify the Target where the results of the Task should go. Lets look at a simple example to get our feet wet, and gradually build complex cases.

Hello World!

This example is rather self-explanatory. I use the MockFile class as the Target just so that I can print to console. One can instead use luigi.LocalFileTarget(filename) to use the file system as the target. The main_task_cls specifies SimpleTask as the task to run. The actual processing part of the task is encapsulated in the run method of the SimpleTask class.

When the script is executed, you should see an output that looks like this:

DEBUG: Checking if SimpleTask() is complete
INFO: Scheduled SimpleTask() (PENDING)
INFO: Done scheduling tasks
INFO: Running Worker with 1 processes
DEBUG: Asking scheduler for work...
DEBUG: Pending tasks: 1
INFO: [pid 30338] Worker Worker(salt=329921834, host=G-ubuntu, username=myuser, pid=30338) running   SimpleTask()
SimpleTask: Hello World!
INFO: [pid 30338] Worker Worker(salt=329921834, host=G-ubuntu, username=myuser, pid=30338) done      SimpleTask()
DEBUG: 1 running tasks, waiting for next task to finish
DEBUG: Asking scheduler for work...
INFO: Done
INFO: There are no more tasks to run at this time
INFO: Worker Worker(salt=329921834, host=G-ubuntu, username=myuser, pid=30338) was stopped. Shutting down Keep-Alive thread

There you go! You have learnt a basic example.

Linked Task Example

The above example was a good starter example. Though we did not really do much in terms of building a pipeline. Lets modify the above code a little bit, so we can build a pipeline.

This example is built on top of the "Hello World" example from above. The SimpleTask outputs the text "Hello World!". The DecoratedTask takes this output from SimpleTask and prefixes with the word "Decorated".

I use the MockFile here just as a way to see the results on the console. This is usually a good testing tool, though I am not sure if it is production ready approach.

The output for the above example would look like:

DEBUG: Checking if DecoratedTask() is complete
INFO: Scheduled DecoratedTask() (PENDING)
DEBUG: Checking if SimpleTask() is complete
INFO: Scheduled SimpleTask() (PENDING)
INFO: Done scheduling tasks
INFO: Running Worker with 1 processes
DEBUG: Asking scheduler for work...
DEBUG: Pending tasks: 2
INFO: [pid 30648] Worker Worker(salt=907129510, host=G-ubuntu, username=myuser, pid=30648) running   SimpleTask()
SimpleTask: Hello World!
INFO: [pid 30648] Worker Worker(salt=907129510, host=G-ubuntu, username=v, pid=30648) done      SimpleTask()
DEBUG: 1 running tasks, waiting for next task to finish
DEBUG: Asking scheduler for work...
DEBUG: Pending tasks: 1
INFO: [pid 30648] Worker Worker(salt=907129510, host=G-ubuntu, username=myuser, pid=30648) running   DecoratedTask()
DecoratedTask: Decorated Hello World!

INFO: [pid 30648] Worker Worker(salt=907129510, host=G-ubuntu, username=myuser, pid=30648) done      DecoratedTask()
DEBUG: 1 running tasks, waiting for next task to finish
DEBUG: Asking scheduler for work...
INFO: Done
INFO: There are no more tasks to run at this time
INFO: Worker Worker(salt=907129510, host=G-ubuntu, username=myuser, pid=30648) was stopped. Shutting down Keep-Alive thread

Conclusion

Here we learnt a really basic example that should give you some sense of building Luigi based task pipelines.

Flask App Directory Structure

Gouthaman Balaraman — Mon, 29 Dec 2014 00:00:00 -0800

This post is a summary of what I learnt perusing a lot of websites. My goal here is to:

summarize the directory structure for a flask app
setup SQLAlchemy with migrations
outline the setup for transition to production

Directory Structure

If you want to build a website based on the Flask framework for python. I recommend you organize you app as:

~/AppHome
        |-- run.py
        |-- config.py
        |__ /myapp
                |-- __init__.py
                |__ /subapp1
                        |-- __init__.py
                        |-- models.py
                        |-- views.py
                        |-- viewmodel.py
                |__ /templates
                        |-- base.html
                        |-- index.html
                        |__ /subapp1
                                |-- subapp1_index.html
                                |-- otherpage.html
                |__ /static
                        |__ /js
                        |__ /css
        |-- setup.py
        |-- MANIFEST.in
        |-- Requirements.txt
        |__ /env
        |
        |-- manage.py
        |__ /migrations
                |__ ...
        |__ /etl
                |__ ...
        |
        |-- fabfile.py

What I have shown above is the overall structure of a Flask application that will transition over to production smoothly. Let me explain the structure briefly before we dive into any code.

run.py is the actual python code that will import the app and execute
config.py is the file for storing configurations for the app
myapp is the folder containing the python code and its dependencies such as templates and static files
myapp/__init__.py is typically where I create the Flask app instance and do all the configuration
myapp/subapp1 is a sub-package under the myapp packages. I recommend using sub-packages to organize your web application with the help of Blueprints.
- myapp/subapp1/__init__.py is the place to create the Blueprint definition and other initializations corresponding to your sub-package.
- myapp/subapp1/models.py is the place to define the SQLAlchemy models.
- myapp/subapp1/views.py is the place to define the routes that the submodule will provide
- myapp/subapp1/viewmodel.py is the bridge between model and view. Use this file to create functions to wrap your database queries using models defined in models.py that ``views.py can call. In the future, should you want to add caching support, you can directly add here and have the benefit of caching.
- templates folder with its sub structure contains the Flask html templates. By mirroring the template sub-folders similar to the app sub-modules, it would be easier to maintain.
- static folder contains all the static resources. By isolating this folder, we could use other servers such as nginx to serve static content.
setup.py script can contain the setup script to bundle your app.
MANIFEST.in typically contains resources about your package, other resources that you wish to package.
Requirements.txt is the place to put all your package dependencies. You can easily install all dependencies using pip.
env can contain your virtual environment for development.
manage.py is the place to put any kinds of Flask related scripts, such as database migrations.
migrations folder containing configuration and scripts associated with database migration.
etl is a folder to put scripts related to ETL tasks. This is something that you could move to a different location. Though sometimes it is convenient to group them if you do need to share code with your app.
fabfile.py to automate local and server side tasks that need to be run often.

... to be continued.

MySQL Cheat Sheet

Gouthaman Balaraman — Tue, 09 Dec 2014 00:00:00 -0800

Here are some notes and commands on MySQL administration

Create a database:
```
CREATE DATABASE mydatabase;
```

Creating a user for the database:

CREATE USER 'testuser'@localhost IDENTIFIED BY 'password';

Granting testuser all privileges on a table:

GRANT ALL PRIVILEGES ON mydatabase.* TO 'testuser'@localhost;

Show a list of databases:
```
SHOW DATABASES;
```
Pick a database to use:
```
USE mydatabase;
```
Show a list of tables after selecting a database:
```
SHOW TABLES;
```

Updating a column with a constant value:

UPDATE table1 SET column_a='value_a', column_b='value_b' WHERE query_id='1';

Deleting rows that satisfy a criterion:
```
DELETE FROM table1 WHERE query_id='1';
```
Resetting the primary key value after any deletes that were performed:
```
ALTER TABLE table1 AUTO_INCREMENT=1;
```

Disabling safe update mode, and turning back on:

SET SQL_SAFE_UPDATES = 0;
DELETE FROM table1 WHERE query_id='1';
SET SQL_SAFE_UPDATES = 1;

Use the above with caution.

Counting duplicates in a column:
```
SELECT query_id, COUNT(*) c FROM table1 GROUP BY query_id HAVING c > 1;
```
This would count all duplicates in query_id and list the count for each. You can drop the HAVING c > 1 part if you just want to get a count on a certain column.

Getting size occupied by a database named mydatabase listed for each table:

SELECT table_name AS "Tables",
round(((data_length + index_length) / 1024 / 1024), 2) "Size in MB"
FROM information_schema.TABLES
WHERE table_schema = "mydatabase"
ORDER BY (data_length + index_length) DESC;

Securing Authentication Tokens

Gouthaman Balaraman — Tue, 02 Dec 2014 00:00:00 -0800

The article Minimal Flask Login Example provided an introduction to token based authentication using the Flask-Login extension for the Flask web framework. The focus of that article was to highlight the crux of authentication logic. One glaring omission was that the token itself was nothing but the username and password passed as clear text. This clearly will not work!

The high level control flow involving token based authentication is as follows:

client (browser) makes a request for the web token in exchange for authentication credential
the authentication credentials are serialized to create a token, and server responds with this token
every time the client needs to request a secured page, the client would provide the authentication token to the server part of the request
the server deserializes the token, fetches the authentication credentials and validates the token
if the token is valid, then the server responds with an access to the secured page; should the token be invalid, then the access to secured page is denied.

In the Minimal Flask Login Example, we skipped the serialization part for simplicity. A more rigorous way of doing this would be to use JSONWebSignatureSerializer in the itsdangerous package to serialize the authentication credentials.

from itsdangerous import JSONWebSignatureSerializer
s = JSONWebSignatureSerializer('secret-key')
token = s.dumps({'username': JaneDoe, 'password' : 'secret'})

The token in the above code can be used to pass from the server side. Validating a token is simple as well.

from itsdangerous import JSONWebSignatureSerializer
s = JSONWebSignatureSerializer('secret-key')
credential = s.loads(token)

The above code will get the credential corresponding to the user which can then be checked against what is stored in the database.

Multi-Processing With Pandas

Goutham Balaraman — Wed, 19 Nov 2014 00:00:00 -0800

NOTE: There are better ways to do distributed processing with pandas now (2017). Please refer to an updated post using dask.

If you have used pandas, you must be familiar with the awesome functionality and tools that it brings to data processing. I have used pandas as a tool to read data files and transform them into various summaries of interest. My usual process pipeline would start with a text file with data in a CSV format. I would read data into a pandas DataFrame and run various transformations of interest.

Recently I stumbled into a problem with this approach. My file was big, in the 100's of MBs. I was running a 32-bit version of python, and I started getting MemoryError. This happens because pandas and numpy would need to allocate contiguous memory blocks, and 32-bit system would have a cap at 2GB. Additionally processing a huge file took some time (more than my impatience could tolerate). The approach I took to solve this problem is:

Read the large input file in smaller chunks so it wouldn't run into MemoryError
Use multi-processing to process the input file in parallel to speed up processing

Sample Code

Pandas read_table method can take chunksize as an argument and return an iterator while reading a file. This means that you can process individual DataFrames consisting of chunksize rows at a time. You can then put the individual results together.

import pandas as pd

LARGE_FILE = "D:\\my_large_file.txt"
CHUNKSIZE = 100000 # processing 100,000 rows at a time

def process_frame(df):
        # process data frame
        return len(df)

if __name__ == '__main__':
        reader = pd.read_table(LARGE_FILE, chunksize=CHUNKSIZE)

        result = 0
        for df in reader:
                # process each data frame
                result += process_frame(df)

        print "There are %d rows of data"%(result)

The code chunk above shows you how to read file in smaller chunks and process each chunk at a time. You can also add a multiprocessing twist to it to get performance boost. Here is a multiprocessing version of the same snippet from above.

import pandas as pd
import multiprocessing as mp

LARGE_FILE = "D:\\my_large_file.txt"
CHUNKSIZE = 100000 # processing 100,000 rows at a time

def process_frame(df):
        # process data frame
        return len(df)

if __name__ == '__main__':
        reader = pd.read_table(LARGE_FILE, chunksize=CHUNKSIZE)
        pool = mp.Pool(4) # use 4 processes

        funclist = []
        for df in reader:
                # process each data frame
                f = pool.apply_async(process_frame,[df])
                funclist.append(f)

        result = 0
        for f in funclist:
                result += f.get(timeout=10) # timeout in 10 seconds

        print "There are %d rows of data"%(result)

The code snippet above should be fairly self explanatory. The idea here is to asynchronously process chunk of data by pushing it into a multiprocessing pool queue. Each process in pool will work on the task, and return the result.

Note, it is important to create the Pool inside the __main__ block. That is because, only one main process should create the pool and distribute asynchronously amongst different processes.

Setting Up Linode For The Impatient

Gouthaman Balaraman — Thu, 13 Nov 2014 00:00:00 -0800

When you get a linode instance, you really get a barren (sort of) linux machine on the cloud. That means you have to take care of customizing the server based on your needs. Here are some notes based on what I wanted to setup.

My requirements were the following:

Python Setup: Setup python to perform installations using pip.
Secure Server: Enable firewall etc.

Linode Basics

Once I setup the linode account, I used the ssh instructions on the "Remote Access" in the linode dashboard to get the IP as:
```
ssh root@<ip-address>
```
Setup hostname:
```
echo "ServerName" > /etc/hostname
hostname -F /etc/hostname
```
Try hostname on the command line to see the hostname printed correctly.
Add to /etc/hosts. In your desktop, associate the linode IP with a custom servername of your liking. Edit /etc/hosts to look like:
```
127.0.0.1   localhost
127.0.1.1   G-ubuntu
111.11.11.1 ServerName
```
Also on you linode add an entry to associate 127.0.0.1 with the server hostname. If not you will a error message that the <ServerName is not recognized when you sudo. Add the following to /etc/hostname in your linode:
```
127.0.0.1  ServerName
```

Securing Linode

Setting up firewall: When I type sudo iptables -L I get:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Now create a file to store the firewall rules using sudo nano /etc/iptables.firewall.rules and paste the following:

*filter

#  Allow all loopback (lo0) traffic and drop all traffic to 127/8 that doesn't use lo0
-A INPUT -i lo -j ACCEPT
-A INPUT -d 127.0.0.0/8 -j REJECT

#  Accept all established inbound connections
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

#  Allow all outbound traffic - you can modify this to only allow certain traffic
-A OUTPUT -j ACCEPT

#  Allow HTTP and HTTPS connections from anywhere (the normal ports for websites and SSL).
-A INPUT -p tcp --dport 80 -j ACCEPT
-A INPUT -p tcp --dport 443 -j ACCEPT

#  Allow SSH connections
#
#  The -dport number should be the same port number you set in sshd_config
#
-A INPUT -p tcp -m state --state NEW --dport 22 -j ACCEPT

#  Allow ping
-A INPUT -p icmp --icmp-type echo-request -j ACCEPT

#  Log iptables denied calls
-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7

#  Drop all other inbound - default deny unless explicitly allowed policy
-A INPUT -j DROP
-A FORWARD -j DROP

COMMIT

Now the firewall rules can be activated using:

sudo iptables-restore < /etc/iptables.firewall.rules

Now when you type sudo iptables -L:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere
REJECT     all  --  anywhere             127.0.0.0/8          reject-with icmp-port-unreachable
ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:http
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:https
ACCEPT     tcp  --  anywhere             anywhere             state NEW tcp dpt:ssh
ACCEPT     icmp --  anywhere             anywhere
LOG        all  --  anywhere             anywhere             limit: avg 5/min burst 5 LOG level debug prefix "iptables denied: "
DROP       all  --  anywhere             anywhere

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere

Now to ensure that the firewall is going to be up and running every time you reboot your Linode, edit /etc/network/if-pre-up.d/firewall as sudo to add:

#!/bin/sh
/sbin/iptables-restore < /etc/iptables.firewall.rules

Once you have saved these edits, make this file executable using:

sudo chmod +x /etc/network/if-pre-up.d/firewall

This should secure your server. If you want to make sure your firewall is up and running, reboot the server and check what you get when you type sudo iptables -L.

Installing and Configuring Fail2Ban: Fail2Ban is an application that prevents dictionary attacks on your server. When Fail2Ban detects multiple failed login attempts from the same IP address, it creates temporary firewall rules that block traffic from the attacker’s IP address. Attempted logins can be monitored on a variety of protocols, including SSH, HTTP, and SMTP. By default, Fail2Ban monitors SSH only.

Here’s how to install and configure Fail2Ban:
- Install Fail2Ban by entering the following command:
```
sudo apt-get install fail2ban
```
- Optionally, you can override the default Fail2Ban configuration by creating a new jail.local file. Enter the following command to create the file:
```
sudo nano /etc/fail2ban/jail.local
```
- To learn more about Fail2Ban configuration options, see this article on the Fail2Ban website. Fail2Ban is now installed and running on your Linode. It will monitor your log files for failed login attempts. After an IP address has exceeded the maximum number of authentication attempts, it will be blocked at the network level and the event will be logged in /var/log/fail2ban.log.

Python Setup

An Elegy

Gouthaman Balaraman — Sun, 26 Oct 2014 00:00:00 -0700

On this day, we mourn the loss of Jayam
Mother to our dads,
Grandmother to us,
Great grandmother to our offspring
And an affectionate human being to all.

May we also remember the spirit that represents the person she was
And what she has left in us - in our little genes and in  our memories.

Lets us remember how her partial blindness
Did not impede her ability to fulfil her parental duties.

Let us remember her love for music
And how she strived to impart the same in us.

Let us remember that in her own quiet ways -
Folding clothes in a corner,
Sipping Narasus coffee,
Telling bedtime stories,
She made a resonant impact in all our lives.


Perhaps this is the day for us to pause from all the rigmarole
And remember what it means to be remembered.

Perhaps this is the day for us to believe
That in the face of impossibilities it is possible to carry on.

Perhaps this day should be a reminder for us
To live the gift that is the joy of living.

May her soul rest in peace!

Understanding the Econometric Factor Model

Gouthaman Balaraman — Sun, 26 Oct 2014 00:00:00 -0700

The post explains the Econometric Factor Model widget that I have built. The factors used in the econometric factor model are: unemployment, consumer sentiment, market, size and value/growth. A brief explanation of the factors are as follows.

The unemployment factor is a proxy for the state of economy, and exposure to this factor measures the impact the economy can have on the performance of this stock.
The consumer sentiment is a voice of consumers behaviour and sentiment, and this factor can help understand the impact of consumer confidence on the stock's performance.
The exposure to market factor captures how the stock's performance is correlated with that of the market.
The size factor (or SML - Small Minus Large ) is a measure of the excess returns that the small cap has over the large cap stocks. The exposure to this factor is meant to capture the impact size has on the performance of this stock. Small (large) cap stock will typically have positive (negative) exposures to this factor.
The value/growth (or HML - High book-to-value Minus Low) is meant to identify the value or growth aspect of a stock.

As a use case, let us try a few examples and see if it makes sense. The following table is based on the factor model run using 5 years of historical returns as of 09-30-2014.

Ticker	Unemployment	Consumer Sentiment	Market	Size	Value
{{item.Ticker}}	{{item.Unemployment}}	{{item.ConsumerSentiment}}	{{item.Market}}	{{item.Size}}	{{item.Value}}

United States Steel Corporation (X), Delta Airlines (DAL), and MicroSoft (MSFT) have negative exposure to unemployment, meaning increase in unemployment (or a detiorating economy) hurts the returns of this company. Mc Donalds (MCD) has a positive exposure to unemployment, in a hurting economy MCD has a better business.

Delta Airlines has the most significant exposure to the consumer sentiment factor. This makes sense because travel industry benefits from a positive consumer sentiment.

The companies on the higher end of the large cap spectrum have a negative exposure to the size factor. The strong positive exposure of DAL to size is a little puzzling, and its history of mergers with Northwest might explain. This is also highlights that factor models do need more than intuition sometimes to make sense and make correct use of.

Econometric Factor Model

Gouthaman Balaraman — Mon, 20 Oct 2014 00:00:00 -0700

Please refer to Understanding Econometric Factor Model for details.

Factor Exposures - {{tickerData.Ticker}}

Factor	Exposure	Standard Deviation
{{item}}	{{tickerData[item].exposure}}	{{tickerData[item].std}}

Risk - {{tickerData.Ticker}}

Total Risk (%)	Diversifiable Risk (%)	Non-Diversifiable Risk (%)
{{tickerData.TotalRisk}}	{{tickerData.DiversifiableRisk}}	{{tickerData.NonDiversifiableRisk}}

As of date: {{tickerData.Date}}

Option Model Handbook, Part II: Introduction to Binomial Trees

Gouthaman Balaraman — Tue, 30 Sep 2014 00:00:00 -0700

This is a series of articles pertaining to option model pricing. Here I will explain the various concepts such as risk-neutral valuation, Black-Scholes formula, and binomial tree option pricing models.

Introduction to Binomial Trees

Let us construct a tree whose pricing is given as shown in the figure above, restricting to time $t_0$ and $t_1$. Initial price of the stock is $S_0$ at $t_0$ and has the possibility of moving to $S_0u$ or $S_0d$ at time $t_1$. Let $p$ be the probability of the price to rise from $S_0$ to $S_0u$. Calculating the expected return from the stock at $t_1$ and making use of risk-neutral valuation

$$ E(S_{t_1}) = pS_0u + (1-p)S_0d = S_0 e^{r(t_1-t_0)} $$

we get,

\begin{equation} \label{eq:btp} p = \frac{e^{r(t_1-t_0)} - d}{u - d} \end{equation}

We need to chose appropriate values for the parameters $u$ and $d$ which can be obtained from equating the variance of the return to $\sigma^2 \Delta t$. The variance of the stock price return on the binomial tree is:

$$ pu^2 + (1−p)d^2 − [ pu+ (1−p)d]^2 = \sigma^2 \Delta t $$

On ignoring terms of order higher than $\Delta t^2$ and making use of $ud = 1$, we get

\begin{equation} u = e^{\sigma \sqrt{\Delta t}} \nonumber \end{equation} \begin{equation} d = e^{-\sigma\sqrt{\Delta t}} \end{equation}

One can make use of the above construction to value an option. Let $f$ be the current value of an option on a stock. Let the payoff of the option after one step (time T) on a binomial tree be $f_u$ and $f_d$ for up and down movement of the stock respectively. The value of the option in this case is given as:

$$ f = e^{−rT}[ pf_u + (1−p)f_d] $$.

Modern Portfolio Theory Statistics

Gouthaman Balaraman — Sun, 21 Sep 2014 00:00:00 -0700

This widget shows Modern Portfolio Theory (MPT) statistics for a selected list of stocks. The calculations were made using Quandl data in the WIKI dataset. The ETF SPY was used as a benchmark using a 5-year time horizon. If the time series is not long enough, then an N/A is shown. You can also download the CSV_File containing all the metrics.

Ticker	{{tickerData.Ticker}}
Alpha (%)	{{tickerData.Alpha}}
Beta	{{tickerData.Beta}}
R-Squared	{{tickerData.RSquared}}
Momentum (%)	{{tickerData.Momentum}}
Annualized Return (%)	{{tickerData.AnnualizedReturn}}
Standard Deviation (%)	{{tickerData.StandardDeviation}}

Sharpe Ratio	{{tickerData.SharpeRatio}}
Sortino Ratio	{{tickerData.SortinoRatio}}
Information Ratio	{{tickerData.InformationRatio}}
Treynor Ratio	{{tickerData.TreynorRatio}}
Tracking Error (%)	{{tickerData.TrackingError}}
Upside Capture (%)	{{tickerData.UpsideCapture}}
Downside Capture (%)	{{tickerData.DownsideCapture}}

As of date: {{tickerData.Date}}

Option Model Handbook, Part I: Introduction to Option Models

Gouthaman Balaraman — Fri, 08 Aug 2014 00:00:00 -0700

Introduction to Option Models

The bedrock of options pricing is the risk-neutral valuation principle, which relates the expected value of a ﬁnancial product at a future time to its current price. This is consistent with no-arbitrage hypothesis. Vanilla options have a theoretical price given by the Black-Scholes formula. In this article we will briefly touch upon these two concepts.

Risk-Neutral Valuation

An important general principle in options pricing is the risk-neutral valuation. According to this principle, the expected return from a stock at time $T$, $E(S_T)$, is the risk free value of the current stock price:

$$ E(S_T) = S_0 e^{rT} $$

The continuous compounding risk-free rate is $r$ and the current stock price is $S_0$. The principal of risk-neutral valuation can be used to create a binomial model for price movement, and subsequently a method to value options.

Black-Scholes Formula

Value of a vanilla European call option, struck at $K$ with time $T$ to maturity, can be derived using the above mentioned risk-neutral valuation principle. The payoﬀ of the call at maturity on an underlying with price $V$ (at maturity) is $max(V − K, 0)$. The expected value of this payoﬀ can be found, assuming a geometric brownian motion price movement for the underlying as,

\begin{equation} E[max(V-K,0)] = \int_K^\infty (V-K)g(V)dV \end{equation}

where $g(V)$ is the probability density function of $V$ such that $log(V)$ is normally distributed with standard deviation $w$. Thus the current value of the call is

$$ C = e^{-rT} E[max(V-K,0)] $$

which after a lengthy calculation comes out to be

\begin{equation} \label{eq:bs} C = S_0 N(d_1) - K e^{-rT} N(d_2) \end{equation}

where $S_0$ is the current price of the underlying, the risk free rate $r$, the volatility of the underlying is $\sigma$,

$$ d_1 = \frac{ \ln(S_0/K) + (r+\sigma^2/2)T } { \sigma \sqrt{T} } $$

$$ d_2 = \frac{ \ln(S_0/K) + (r-\sigma^2/2)T } { \sigma \sqrt{T} } = d_1 - \sigma\sqrt{T}$$

and $N(x)$ is the cumulative distribution function. The put-call parity relates the price of the put and call prices, and is given as:

$$ C - P = S_0 - e^{-rT}K $$

Put-Call parity and Eq. $\ref{eq:bs}$ can be used to arrive at put prices.

For the case where the underlying has a continuous payout (dividend), the Black-Scholes formula can be extended to yield

$$ C = S_0 e^{-qT} N(d_1^\prime) - K e^{-rT} N(d_2^\prime) $$

where

$$ d_1^\prime = \frac{ \ln(S_0/K) + (r - q + \sigma^2/2)T} {\sigma\sqrt{T}} $$

$$ d_2^\prime = d_1^\prime - \sigma\sqrt{T} $$

Conclusion

In this article I explained the fundamentals of option model pricing. We looked at the risk neutral valuation, and how it can be used to derive the Black-Scholes model.

Minimal Example of Calling Lua Functions from C++

Goutham Balaraman — Tue, 05 Aug 2014 00:00:00 -0700

If you want to extend and customize the capabilities of a C++ application without requiring a full recompilation, then using a embedded scripting language is the way to go. Lua is one such embeddable scripting language, and is very popular among game developers. The main advantage of Lua, in my opinion, is that the core API is very minimal, has very small memory footprint. The availability of LuaJIT makes it a very performant alternative as well.

This article is a continuation of the earlier introductory article Minimal C++ Example <|filename|minimal-lua.rst>. In this article we will discuss how to call lua functions from C++.

Example Code

extern "C" {
      #include "lua.h"
      #include "lualib.h"
      #include "lauxlib.h"
}
#include <string.h>

int main(int argc, char* argv[])
{
      // initialization
      lua_State * L = lua_open();
      luaL_openlibs(L);

      // execute script
      const char lua_script[] = "function sum(a, b) return a+b; end"; // a function that returns sum of two
      int load_stat = luaL_loadbuffer(L,lua_script,strlen(lua_script),lua_script);
      lua_pcall(L, 0, 0, 0);

      // load the function from global
      lua_getglobal(L,"sum");
      if(lua_isfunction(L, -1) )
      {
        // push function arguments into stack
        lua_pushnumber(L, 5.0);
        lua_pushnumber(L, 6.0);
        lua_pcall(L,2,1,0);
        double sumval = 0.0;
        if (!lua_isnil(L, -1))
        {
          sumval = lua_tonumber(L,-1);
              lua_pop(L,1);
        }
        printf("sum=%lf\n",sumval);
      }

      // cleanup
      lua_close(L);
      return 0;
}

Code Explained

The initial part of the code initialises the lua_State loads and executes the script with lua_pcall(L,0,0,0). Once the script is loaded, the functions are available in the global namespace. Here the script basically is a function that takes two numbers and returns the sum of the two. Using the lua_getglobal(L, "sum") call, we load the function into the stack. We can check if the function was loaded correctly using the lua_isfunction(L, -1). Then we pass the two arguments of the function by pushing them into the stack. Then the lua_pcall method executes the function and loads the result onto the stack. The successful execution of the function can be checked by checking that the stack is not nil using !lua_isnil(L,-1). The returned value can then be accessed by casting the result in the top of the stack using lua_tonumber(L,-1). We use lua_pop to clear the result from the stack.

Running this example should print:

sum=11.000000

on the screen.

Conclusion

This article gave a very minimal example explaining how to call a lua function from C++.

Running ZEO as a Windows Service

Gouthaman Balaraman — Mon, 04 Aug 2014 00:00:00 -0700

The other day, I wanted to run ZEO as a Windows service. The runzeo.py part of ZEO will let you run the client, but it doesn't work well for deployment on a windows machine. So I used pywin32 to wrap the runzeo.py into a Windows Service.

Code

You can fetch the source from my github repo. This script requires pywin32 and ZEO to run. I do intend to put it on PyPi when I have the time

You can run from cmd as Administrator:

> python zeo_winservice.py

you will be given the service options, as shown below:

Usage: 'zeo_winservice.py [options] install|update|remove|start [...]|stop|restart [...]|debug [...]'
Options for 'install' and 'update' commands only:
 --username domain\username : The Username the service is to run under
 --password password : The password for the username
 --startup [manual|auto|disabled|delayed] : How the service starts, default = manual
 --interactive : Allow the service to interact with the desktop.
 --perfmonini file: .ini file to use for registering performance monitor data
 --perfmondll file: .dll file to use when querying the service for
   performance data, default = perfmondata.dll
Options for 'start' and 'stop' commands only:
 --wait seconds: Wait for the service to actually start or stop.
                 If you specify --wait with the 'stop' option, the service
                 and all dependent services will be stopped, each waiting
                 the specified period.

Installing the Service

Before you try to install, make sure you are running cmd as Administrator. I like to install such that it will start up automatically, as shown below:

>python zeo_winservice.py --startup=auto install

which gives you the following screen with ZEO options:

Installing service ZEO WinService
Start the ZEO storage server.

Usage: %s [-C URL] [-a ADDRESS] [-f FILENAME] [-h]

Options:
-C/--configuration URL -- configuration file or URL
-a/--address ADDRESS -- server address of the form PORT, HOST:PORT, or PATH
                        (a PATH must contain at least one "/")
-f/--filename FILENAME -- filename for FileStorage
-t/--timeout TIMEOUT -- transaction timeout in seconds (default no timeout)
-h/--help -- print this usage message and exit
-m/--monitor ADDRESS -- address of monitor server ([HOST:]PORT or PATH)
--pid-file PATH -- relative path to output file containing this process's pid;
                   default $(INSTANCE_HOME)/var/ZEO.pid but only if envar
                   INSTANCE_HOME is defined

Unless -C is specified, -a and -f are required.

Enter command line arguments for ZEO Service:

Now you are prompted with the different configurations for the ZEO Service that you can pass. One thing to note here is that the filename option has to absolute path, and not a relative path.

An example command line argument is:

Enter command line arguments for ZEO Service: -f D:\path\to\data\file.fs -a localhost:9999

Here I am specifying that ZEO be run with the file.fs on localhost port 9999. After installing the script, you need to start it by:

>python zeo_winservice.py start

You will also be able to access the service from task manager or the Windows Services app.

Logging

The logs from the service are sent to the windows Event Log which can be accessed by opening the Event Viewer. Once you open the Event Viewer, the logs can be found under:

Event Viewer->Windows Logs->Application

The logs from this script can be found under ZEO WinService in the Source column.

Minimal Flask-Login Example

Gouthaman Balaraman — Thu, 24 Jul 2014 00:00:00 -0700

The goal of this post is to give a very basic introduction to token based authentication using Flask-Login. Usually the user credentials are stored in a database, with passwords hashed. However the authentication mechanism can be understood without having to worry about database, and various token generation algorithms. As a first step lets focus on just understanding the authentication mechanism. Then in a subsequent post we will handle other important parts.

To run this example, you will need flask and flask-login with their dependencies installed. This can be done using pip as shown below:

>pip install flask
>pip install flask-login

Example Code

Here is the full source code that we are about to discuss:

Code Explained

Lets delve deeper into this example, and I will explain each part of the code in greater detail here.

from flask import Flask, Response
from flask.ext.login import LoginManager, UserMixin, login_required

app = Flask(__name__)
login_manager = LoginManager()
login_manager.init_app(app)

The first five lines of the code import the required modules, and initializes the Flask app. Then the LoginManager instance is created and then is configure for login. Now lets try to understand the User class.

class User(UserMixin):
    # proxy for a database of users
    user_database = {"JohnDoe": ("JohnDoe", "John"),
               "JaneDoe": ("JaneDoe", "Jane")}

    def __init__(self, username, password):
        self.id = username
        self.password = password

    @classmethod
    def get(cls,id):
        return cls.user_database.get(id)

Here I have created the User class by overloading the UserMixin class. The UserMixin class implements some of the default methods, and hence is a convenient starting point. The dict user_database is a proxy for all the database code one would need. I am abstracting this away as a dict for simplicity. The get class method returns the user data from user_database.

For the LoginManager to handle authentication, we have to provide a method for it to load user. Here I use the generic @login_manager.request_loader decorator to decorate the load_user function. The expected behavior of a request_loader is to return a User instance if the provided credentials are valid, and return None otherwise.

@login_manager.request_loader
def load_user(request):
    token = request.headers.get('Authorization')
    if token is None:
        token = request.args.get('token')

    if token is not None:
        username,password = token.split(":") # naive token
        user_entry = User.get(username)
        if (user_entry is not None):
            user = User(user_entry[0],user_entry[1])
            if (user.password == password):
                return user
    return None

The load_user looks for a token in Authorization header, or the request arguments. If a token is provided, then I return an instance of User if the token is valid, and return None otherwise. Here I assume that a valid token would be of the form <username>:<password>. This is a naive token, and should not be used in practice. Using serializers from itsdangerous package can come handy. We will touch upon these issues in another post.

Once this setup is done, in order to require authentication for a route, use the @login_required decorator.

Run the above script, and if you visit the LocalHostUnAuthenticated route without a token you will get a 401 Unauthorized message. If you pass a token to LocalHostAuthenticated, then you will be allowed access to the protected page.

Conclusion

This article explained how to write token based authentication using Flask-Login extension. The focus of this article was explaining the basic workings of flask-login without having to setup database or even the token generation. Once the basic plumbing is setup, one can extend this example in two ways:

Securing Authentication Tokens
have a database to store and retrive user credentials.

Minimal Lua C++ Example

Goutham Balaraman — Thu, 17 Jul 2014 00:00:00 -0700

Example Code

Here is a minimal Lua(V5.1) - C++ example to get one started.

// luaexample.cpp
extern "C" {
  #include "lua.h"
  #include "lualib.h"
  #include "lauxlib.h"
}
#include <string.h>

int main(int argc, char* argv[])
{
      // initialization
      lua_State * L = lua_open();
      luaL_openlibs(L);

      // execute script
      const char lua_script[] = "print('Hello World!')";
      int load_stat = luaL_loadbuffer(L,lua_script,strlen(lua_script),lua_script);
      lua_pcall(L, 0, 0, 0);

      // cleanup
      lua_close(L);
      return 0;
}

Code Explained

Here the lua_open and luaL_openlibs are initialization step in order to prepare the lua_State * L. We store the script that we need to execute in the variable lua_script. The luaL_loadbuffer is used to load the script. The loaded script can be executed by calling lua_pcall(L, 0, 0, 0). The arguments passed to lua_pcall are the lua_State pointer, number of arguments to the script (which is none here), number of values returned (which is none here). The last argument to lua_pcall is the error handler which we will not discuss here.

This needs to be compiled using a C++ compiler and linked to Lua library. When you run the executable, you should see Hello World! printed on the screen.

Conclusion

We looked at a rather simple introductory example of lua interpreter embeded into C++ code.

Flask Track Usage Primer

Gouthaman Balaraman — Wed, 26 Mar 2014 00:00:00 -0700

Recently I had a requirement to add light weight tracking for a Flask app. Tools like Google analytics was not an option because, I wanted raw number and data handy rather than just figures, this was a website page in the intranet, and I also wanted to easily track all the REST apis. That is when I stumbled upon Steve Milner's Flask-Track-Usage package.

The API and usage is incredibly simple, and as of version 0.0.7 supported Mongo DB storage. Since Mongo was not an option for me, I added SQLAlchemy based storage, thus opening doors for a wide array of databases. These additions will be released as part of version 1.0.0.

Here is a quick-start version on using SQLStorage with Flask-Track-Usage:

# mainapp.py
# Create the Flask 'app'
from flask import Flask
app = Flask(__name__)

# Set the configuration items manually for the example
app.config['TRACK_USAGE_USE_FREEGEOIP'] = False
app.config['TRACK_USAGE_INCLUDE_OR_EXCLUDE_VIEWS'] = 'include'

# We will just print out the data for the example
from flask.ext.track_usage import TrackUsage
from flask.ext.track_usage.storage.sql import SQLStorage

# Make an instance of the extension
t = TrackUsage(app, SQLStorage(conn_str="sqlite:///D:\\usage_tracking.db"))

# Make an instance of the extension
t = TrackUsage(app, storage)

# Include the view in the metrics
@t.include
@app.route('/')
def index():
    return "Hello"

# Run the application!
app.run(debug=True)

In the code above, the SQLStorage object is created with a conn_str argument which can be any of the engine configurations supported by SQLAlchemy itself.

You can use Flask-Track-Usage with blueprints as well. This is how you include blueprints views:

# Here app is the Flask app created in mainapp.py
app.config['TRACK_USAGE_INCLUDE_OR_EXCLUDE_VIEWS'] = 'include'

# Make an instance of the extension
t = TrackUsage(app, SQLStorage(conn_str="sqlite:///D:\\usage_tracking.db"))

# exclude just the a_blueprint out
from my_blueprints import a_blueprint
t.exclude_blueprint(a_blueprint)

What we have done here is to include all views, except for the ones in the blueprint a_blueprint. You can instead exclude all by default, and include just the specific ones as shown below.

# Here app is the Flask app created in mainapp.py
app.config['TRACK_USAGE_INCLUDE_OR_EXCLUDE_VIEWS'] = 'exclude'

# Make an instance of the extension
t = TrackUsage(app, SQLStorage(conn_str="sqlite:///D:\\usage_tracking.db"))

from my_blueprints import a_blueprint
t.include_blueprint(a_blueprint)

Once you have setup the extension to track usage, you can even use the _get_usage method part of the Storage object to view the usage hits. This method returns a list of JSON dict and has the following schema:

[
    {
            'url': str,
            'user_agent': {
                'browser': str,
                'language': str,
                'platform': str,
                'version': str,
            },
            'blueprint': str,
            'view_args': dict or None
            'status': int,
            'remote_addr': str,
            'authorization': bool,
            'ip_info': str or None,
            'path': str,
            'speed': float,
            'date': datetime,
    },
    {
        ....
    }
]

Git For Dummies

Goutham Balaraman — Thu, 20 Mar 2014 00:00:00 -0700

Git is a version control with a ton of powerful features. This article is intended to be an introduction for dummies (like me). I have had to do a lot of googling lately to get familiar with git. My understanding of git is a work in progress. Here I summarize some of the basic commands and features that I learnt. I plan to keep this article updated as and when I learn something new.

Git Basics

One of the basic concepts in git is the notion of upstream and origin. The term upstream refers to the main repository lets say the pandas on github hosted at https://github.com/pydata/pandas. However you have forked the to work on an issue, then the origin would refer to the user's fork of the upstream repository.

Initialize repository: in an empty folder:
```
git init
```
You should see a message akin to:
```
Initialized empty Git repository in /home/username/test/.git/
```
Add remote repository: To hook up to your local repository to a remote repository, such as the pandas repository on github:
```
git remote add upstream https://github.com/pydata/pandas.git
```
Note that instead of adding upstream, I could have added my fork as origin as shown below:
```
git remote add origin https://github.com/gouthambs/pandas.git
```
Remove remote repository: You have made a mistake and added a wrong remote repository. To remove a remote repo:
```
git remote remove upstream
```
You can do the same to remove a remote origin:
```
git remote remove origin
```
View remote repositories: Check what the remote repositories are attached to your local repo using:
```
git remote -v
```
This after adding the above pandas repo would show something like:
```
upstream    https://github.com/pydata/pandas.git (fetch)
upstream    https://github.com/pydata/pandas.git (push)
```
Fetching code from remote: Now that you have hooked up to a remote repository, how do you fetch the code from there? Lets say we have added remote upstream, and we want to fetch the code, then do the following:
```
git fetch upstream
```
What this does is it fetches the whole repository, with all its branches and tags etc. Now to actually start using a particular branch you have to checkout.
Checkout code: To checkout the master try:
```
git checkout master
```
Syncing origin with upstream: If you have forked a project and have been working on your fork for a while, what will happen is that the project that you originally forked from has advanced. Then you would need to sync your fork with upstream to ensure that your changes are still valid. This can be done by:
```
git merge upstream/master
```
Here I am syncing the upstream/master with the local checked out project.

Committing Code Changes

Now when you have checked out code and made some modifications. Now you want to share to the world. You have to go through three steps:

Staging your changes: This is a way to specify what you want to commit and what you want to ignore. You stage your commit by:
```
git add <filename>
```
What this does is adds your file for commit. If you type git status you will see that your new file has been staged for commit. This step gives you control on what files you want to commit and what you don't want to commit.
Commit your staged changes: When you commit your changes, then your changes are registered in your local repository:
```
git commit -m <message>
```
This pushes your code changes to the local repository. Your changes will not be sent to the remote repository such as github yet.
Push your commits: You have to push your changes to send it up to the remote repo:
```
git push origin master
```
Here we are assuming that you want to push your changes to the master.

Branching and Merging

Say you have checked out a repository, and you want to work on a feature. Its a good idea to create a feature branch, to work on your modifications.

To create a branch:

git checkout -b myAwesomeFeature
Switched to a new branch "myAwesomeFeature"

Deleting branches: To delete a local branch:
```
git branch -d myAwesomeFeature
```
List branches: To list local branches:
```
git branch
```
To list remote branches only:
```
git branch -r
```
To list all branches:
```
git branch -a
```

Tagging

List all the tags:
```
git tag
```

Creating tags:

git tag -a v0.5.0 -m "Tagging version 0.5.0"

Asynchronous Python Logging Using MSMQ

Goutham Balaraman — Fri, 21 Feb 2014 00:00:00 -0800

If you have a web application running in Python, there can be a need for the logging to not interfere with the performance. Default file based loggers can lead to a slow down because of the constand disk writes. An alternate solution that can be quite handy is logging into a message based logger, such as MSMQ. This post is built on my earlier post Sending MSMQ Messages Using Python, where I discuss how to setup, send and receive messages using MSMQ.

Here I will show how one can use the MSMQ to build a custom handler, that can be used with the logging module in python. Here is the MSMQHandler class:

    # customhandler.py
import logging

class MSMQHandler(logging.Handler):
    def __init__(self,queue_name,label_name,dest_computer=None):
        logging.Handler.__init__(self)
        import os
        import win32com.client
        self.queue_name = queue_name
        self.label_name = label_name
        self.computer_name = dest_computer if not dest_computer is  None\
            else os.getenv('COMPUTERNAME')
        qinfo=win32com.client.Dispatch("MSMQ.MSMQQueueInfo")
        qinfo.FormatName="direct=os:"+self.computer_name+\
            "\\PRIVATE$\\"+self.queue_name
        try:
            self.queue = qinfo.Open(2,0)
        except Exception as e:
            self.queue = None
            raise RuntimeError(str(e))

    def emit(self,record):
        import win32com.client
        if self.queue :
            msg=win32com.client.Dispatch("MSMQ.MSMQMessage")
            msg.Label=self.label_name
            msg.Body = self.format(record)
            msg.Send(self.queue)

    def close(self)        :
        self.acquire()
        try:
            if self.queue:
                self.queue.close()
        finally:
            self.release()

Once you have the handler in place, and setup a private MSMQ queue, say KaruthQueue, then you can incorporate it into your workflow as shown below:

# example.py
from customhandler import MSMQHandler
import logging

lgr = logging.getLogger("Test")
hnd = MSMQHandler("KaruthQueue","QPyLog") # here KaruthQueue is the private queue name
lgr.addHandler(hnd)
lgr.setLevel(logging.INFO)
lgr.info("Test Message")

And that completes the MSMQ logger using python. Now all your logs will be pushed to the KaruthQueue that we created. You can use the example shown here to read the messages in a seperate application and store any way you chose, files, database etc.

Sending MSMQ Messages Using Python

Goutham Balaraman — Fri, 21 Feb 2014 00:00:00 -0800

MSMQ is a message queue in the Windows environment, very much like RabbitMQ. MSMQ can be used as a task queue or for communicating between processes. One advantage of using MSMQ rather than inter-process communication (IPC) is that messages have persistence. Here I show how to send and receive MSMQ messages using python.

Setting Up MSMQ on Windows 7

Before we can proceed with the code, here is a quick note on set-up. To setup MSMQ (disabled by default) go to Control Panel -> Programs and Features -> Turn Windows Features on or off (on the left panel). Now check Microsoft Message Queue(MSMQ) Server if it is not checked. Hit OK. Now we have enabled MSMQ server.

Now that we have the MSMQ server up and running, lets create a private queue so we can use in our script. To do that, go to Control Panel -> Administrative Tools -> Computer Management. Expand Services and Applications -> Message Queuing. Select Private Queues under Message Queuing and right-click and click New. Now in the dialog box put a Queue Name, for this example purpose, say KaruthQueue and click OK.

Installing PyWin32

To send and receive MSMQ messages you need the awesome Python for Windows Extension module, PyWin32 by Mark Hammond. Once you have completed installation, you can check if your installation is fine by trying import win32com. If you don't get any errors, then you are good to go.

Sending MSMQ Messages

Here is a small code snippet that shows you how to send messages using MSMQ.

import win32com.client
import os

qinfo=win32com.client.Dispatch("MSMQ.MSMQQueueInfo")
computer_name = os.getenv('COMPUTERNAME')
qinfo.FormatName="direct=os:"+computer_name+"\\PRIVATE$\\KaruthQueue"
queue=qinfo.Open(2,0)   # Open a ref to queue
msg=win32com.client.Dispatch("MSMQ.MSMQMessage")
msg.Label="TestMsg"
msg.Body = "The quick brown fox jumps over the lazy dog"
msg.Send(queue)

queue.Close()

Voila! We have sent our message. You can check if the message was sent properly by checking under Control Panel -> Administrative Tools -> Computer Management -> Services and Applications -> Message Queuing -> Private Queues -> KaruthQueue -> Queue Messages. Right click and refresh after you have run the script, if you don't see any. You should see an entry there that corresponds to the message that was sent

Receiving MSMQ Messages

import win32com.client
import os

qinfo=win32com.client.Dispatch("MSMQ.MSMQQueueInfo")
computer_name = os.getenv('COMPUTERNAME')
qinfo.FormatName="direct=os:"+computer_name+"\\PRIVATE$\\KaruthQueue"
queue=qinfo.Open(1,0)   # Open a ref to queue to read(1)
msg=queue.Receive()
print "Label:",msg.Label
print "Body :",msg.Body
queue.Close()

Run this code after you have sent the message using the above script. Now you should see your earlier message printed out.

Have fun messaging!

A Short Story of Wall Street

Gouthaman Balaraman — Thu, 23 Jan 2014 00:00:00 -0800

In the last couple of decades, the US market has witnessed a technology bubble, a housing bubble, recession, and market crashes. I wanted to understand the human sentiment, the fear when market crashes, the euphoria while riding the upside of the bubble. Here I narrate the story of wall-street from the view of short-term volatility.

Volatility

Volatility is a finance jargon for quantifying uncertainty over a window of time. If the window is small (like a week), then one can get a view of the market's reaction to day-to-day occurrences. On the other hand if we measure volatility using a larger window (such as a year), then a lot of the day-to-day noise dies out, and you get a macroscopic view of the market.

I wanted to understand how the market reacts to the crashes, and recessions. Do these things just happen out of the blue? Is there a lot of uncertainty, fear or excitement leading up to a market collapse? Is there a way to compare the latest market crash with those that have come before?

The Voice of Uncertainty

The picture above shows the S&P 500 index at the top, and its short-term volatility over the last 33 years. The last 33 years have gone through 4 recessions, the regions shaded in green. Here are some interesting observations:

The markets prior to 1995 were less volatile, by about 24%, than post 1995. The average volatility was 14% for the time period of 1980 to 1995, in contrast with a 17.3% for the period 1995 to 2013.
On pure percentage terms, the market crash on Black Monday (S&P 500 fell by 25.7% on Oct 19, 1987) was larger than the market drop in 2008 (21% drop in the week ending October 10, 2008). But the volatility paints a totally different picture. The uncertainty during 2008 market crash was roughly 20% more than that during the 1987 market crash.
In August of 2011, the S&P downgraded US credit from AAA to AA+. This made the market volatile, and that is the little blip you see in 2011. The irony is, prior to 2008 crash, S&P had rated a some of the mortgage backed securities as high quality debt, even though a lot of them defaulted.

Methodology

I use S&P 500 to be representative of the stock market. The short term volatility is calculated by computing volatility using daily returns over a 45 day window.

Calculating Stock Beta, Volatility, and More

Gouthaman Balaraman — Thu, 09 Jan 2014 00:00:00 -0800

Have you ever wondered how to calculate the Beta value that is shown in GoogleFinance or YahooFinance and what does it mean from an investment perspective? This article will give you an introduction to the concept and demonstrate how you can calculate various time series measures for a stock using python. The Modern Portfolio Theory Statistics page shows calculated betas, alpha, etc for a few thousand stocks.

Beta, Alpha and R-squared

Beta of a stock is a measure of relative risk of the stock with respect to the market. The convention (though not a rule) is to use S&P 500 index as the proxy for market. A beta value of greater than 1 means that the stock returns amplify the market returns on both the upside and downside. On the contrary, a beta value of less than 1 means that the stock returns are subdued in comparison to the market returns. It is very important to understand that beta is a relative measure of risk, and not an absolute measure of risk. That means that we are only saying how risky the stock is vis-a-vis the market. If the stock market itself is overheated and volatile, then a beta of 1 means that the stock is equally volatile, and equally risky.

The beta is calculated using regression analysis. In simple terms if you plot the returns of stock as a function of the returns of the market benchmark (such as S&P 500) and fit it with a straight line, then beta is nothing but the slope of the fitted line. The regression equation is given as shown below.

$$ R^{stock}_i = \alpha + \beta \times R^{market}_i + \epsilon_i $$

In Capital Asset Pricing Model, the returns of the stock $R^{stock}$ and that of the market $R^{market}$ are adjusted for the risk-free rate. Here for simplicity I leave that out. Given the current low interest rate environment, this adjustment will add little value.

Alpha of a stock gives you a measure of the excess return with respect to the benchmark. A positive alpha for a stock or portfolio gives you sense of how well your asset outperformed a benchmark.

The $R^2$ is a measure of how well the the returns of a stock is explained by the returns of the benchmark. If your investment goal is to track a particular benchmark, then you should chose stocks that show a high $R^2$ with respect to the benchmark. The $R^2$ value of $1$ means that the benchmark completely explains the stock returns, while a value of $0$ means that the benchmark does not explain the stock returns.

R-squared is defined as:

$$ R^2 = 1 - SS_{res}/SS_{tot} $$

where

$$ SS_{res} = \sum_i (R^{stock}_i - f^{stock}_i)^2 $$

$$ SS_{tot} = \sum_i (R^{stock}_i - <R^{stock}>)^2 $$

Volatility and Momentum

The measures discussed in the earlier section are what I would call relative measures, i.e., they are with respect to a proxy that is a representation of market. Time series measures such as volatility and momentum are what I would call innate measures.

Volatility is nothing but the standard deviation of the returns of the stock. It gives us a sense of how much the stock returns fluctuate and how risky it is. High volatile stocks have high risk, and also have the potential to offer higher returns. A 3-year history of 1-month returns can be a good sample to calculate volatility.

Momentum is a measure of the past returns over a certain period. The 1-year momentum will be the 1-year return of the stock, where as a 3-year momentum will be the 3-year return of the stock.

Python Code

One can use data from yahoo finance to calculate the stock beta as shown:

from pandas.io.data import DataReader
from datetime import date
import numpy as np
import pandas as pd

# Grab time series data for 5-year history for the stock (here AAPL)
# and for S&P-500 Index
sdate = date(2008,12,31)
edate = date(2013,12,31)
df = DataReader('WFM','yahoo',sdate,edate)
dfb = DataReader('^GSPC','yahoo',sdate,edate)

# create a time-series of monthly data points
rts = df.resample('M',how='last')
rbts = dfb.resample('M',how='last')
dfsm = pd.DataFrame({'s_adjclose' : rts['Adj Close'],
                        'b_adjclose' : rbts['Adj Close']},
                        index=rts.index)

# compute returns
dfsm[['s_returns','b_returns']] = dfsm[['s_adjclose','b_adjclose']]/\
    dfsm[['s_adjclose','b_adjclose']].shift(1) -1
dfsm = dfsm.dropna()
covmat = np.cov(dfsm["s_returns"],dfsm["b_returns"])

# calculate measures now
beta = covmat[0,1]/covmat[1,1]
alpha= np.mean(dfsm["s_returns"])-beta*np.mean(dfsm["b_returns"])

# r_squared     = 1. - SS_res/SS_tot
ypred = alpha + beta * dfsm["b_returns"]
SS_res = np.sum(np.power(ypred-dfsm["s_returns"],2))
SS_tot = covmat[0,0]*(len(dfsm)-1) # SS_tot is sample_variance*(n-1)
r_squared = 1. - SS_res/SS_tot
# 5- year volatiity and 1-year momentum
volatility = np.sqrt(covmat[0,0])
momentum = np.prod(1+dfsm["s_returns"].tail(12).values) -1

# annualize the numbers
prd = 12. # used monthly returns; 12 periods to annualize
alpha = alpha*prd
volatility = volatility*np.sqrt(prd)

print beta,alpha, r_squared, volatility, momentum

Some caveats about the sample code. The returns are calculated using the adjusted close from Yahoo finance data. This is because the adjusted close accounts for dividends and splits etc. In my personal experience I have found the returns calculated this way to be a reasonably close estimate but not always accurate. The volatility is calculated here as a simple standard deviation of the returns. From an option-pricing model perspective volatility is calculated assuming a log-normal distribution for the returns.

The alpha shown above is annualized by scaling by a factor of 12, the periodicity of returns. The same goes for volatility, which is scaled by $\sqrt{12}$ in order to annualize.

Python Multiprocessing as a Task Queue

Goutham Balaraman — Tue, 07 Jan 2014 00:00:00 -0800

When you have computationally intensive tasks in your website (or scripts), it is conventional to use a task queue such as Celery. Using Celery requires some amount of setup and if you want to avoid, try using the following task queue based on the multiprocessing. Depending on the application at hand, Celery might be an overkill. An alternate approach is to use multiprocessing as a task queue.

Here is a simple introduction to multiprocessing:

from multiprocessing import Pool
def expensive_function(x):
        # do your expensive time consuming process
        return x*x
if __name__ == '__main__':
        # start 4 worker processes
        pool = Pool(processes=4)
        # evaluate "f(10)" asynchronously
        result = pool.apply_async(expensive_function, [10])
        print result.get(timeout=1)

The above snippet is copied from the multiprocessing documentation, and is fairly self explanatory. In the main block we start a pool of 4 processes. Then we asynchronously evaluate the expensive_function.

One can use the same idea for a website as shown below in the Flask app example:

from multiprocessing import Pool
from flask import Flask

app = Flask(__name__)
_pool = None

def expensive_function(x):
        # import packages that is used in this function
        # do your expensive time consuming process
        return x*x

@app.route('/expensive_calc/<int:x>')
def route_expcalc(x):
        f = _pool.apply_async(expensive_function,[x])
        r = f.get(timeout=2)
        return 'Result is %d'%r

if __name__=='__main__':
        _pool = Pool(processes=4)
        try:
                # insert production server deployment code
                app.run()
        except KeyboardInterrupt:
                _pool.close()
                _pool.join()