Tessellation is a process that divides a surface into a smoother mesh of triangles. The new OpenGL 4.0 supports Tessellation Shaders. Tessellation Shaders sole purpose is to interpolate geometry to create additional geometry that can perform adaptive subdivision based on criteria such as size or curvatures.
Overall, Tessellation Shaders increase the quality of your final image, but it can do this dynamically. For example, if you want to add more geometric detail to your model, you would usually add detail through your modeling software such as Blender. With Tessellation Shaders, you can add geometric details dynamically.
You may have also heard about Geometry Shaders. Both Tessellation and Geometry shaders are capable of creating new geometry from existing geometry. However, There is a big difference between the two. Tessellation Shaders can create new geometry, but the new geometry is of the same sort as what you started with. For example, you get more lines from a segment, or more triangles from a triangle patch. Geometry Shaders can give you a new different geometry topology.
So where is the Tessellation Shader in the Graphics Pipeline? The Tessellation shader follows the Vertex Shader and comes before the Geometry Shader. This is illustrated below:
The tessellation shader is composed of two stages: The Tessellation Control Shader (TCS) and the Tessellation Evaluation Shader (TES).
Between the TCS and the TES there is an additional operation called the Tessellation Primitive Generator (TPG).
The TPG produces the primitives based on a set of standard tessellation algorithms.
The TCS is responsible for setting up the TPG. i.e., it defines how the primitives should be generated by the TPG and what algorithm to use.
The TES is responsible for determining the position of each vertex of the primitives that are produced by the TPG.
If you are interested in learning more about tessellation, I would recommend these book: Graphics Shaders: Theory and Practice (A bit expensive, but worth every penny)
This book has a very good chapter on Tessellation OpenGL 4.0 Shading Language Cookbook
Note: This is a summary from a Tessellation chapter found in this amazing book Graphic Shaders
PS. Sign up to my newsletter and get OpenGL development tips.
