TypeScript extends JavaScript by adding types, thereby enhancing code quality and understandability through static type checking which enables developers to catch errors at compile-time rather than runtime.

The TypeScript team has built a compiler tsc to process TypeScript type annotations and emit JavaScript code, however, the compiler is not limited to just compiling TypeScript code to JavaScript, it can also be used to build tools and utilities around TypeScript.

In this article, you’ll explore the TypeScript Compiler API, which is an integral part of the TypeScript compiler that exposes various functionalities, enabling you to interact with the compiler programmatically.

The article is organized into different use cases each of which will introduce you to a new aspect of the Compiler API, and by the end of the article, you’ll have a thorough understanding of how the Compiler API works and how to use it to build your own tools. Keep in mind the use cases are not complete, they’re just to demonstrate the concept.

Table of Content

• What is A Compiler?
  • What is The TypeScript Compiler?
  • What is The TypeScript Compiler API?
  • Why would you use the Typescript Compiler API?
• Use Case: Enforce One Class Per File
  • TypeScript Program
  • Source File
  • Abstract Syntax Tree (AST)
  • Node
  • Declaration
  • Statement
  • Expression
• Use Case: No Function Expression
  • Transformer
  • Visitor
  • Context
• Use Case: Replace Function Expression With Function Declaration
  • Factory
  • Lexical Environment
• Use Case: Detect Third-Party Classes Used as Superclasses
  • HeritageClauses
  • Type
  • Type Checker
  • Symbol
• Diagnostic
• Printer
• How The Compiler Works
  • Lexical Analysis
  • Syntax Analysis
  • Semantic Analysis
  • Emitting
• ESLint
• Conclusion
• Next Steps
• References
• Related Projects
• Credits

What is A Compiler?

A compiler is a specialized software program that translates source code written in one programming language into another language, usually machine code or an intermediate form. Compilers perform several tasks including lexical analysis, syntax analysis, semantic analysis, code generation, and more.

Compilers come in various forms, serving different needs, to my understanding TypeScript is a Source-to-source compiler, which means it takes TypeScript code and compiles it into JavaScript code.

What is The TypeScript Compiler?

The TypeScript Compiler (tsc) takes TypeScript code (JavaScript and type information) and compiles it into plain JavaScript as the result while in the process it performs type checking to catch errors at compile time rather than at runtime.

For the compilation to happen you need to feed the compiler a TypeScript file/code and the configuration file (tsconfig.json) to guide the compiler on how to behave.

function IAmAwesome() {}

{
  "compilerOptions": {
    "target": "es5",
    "module": "commonjs",
    "outDir": "./dist",
    "strict": true
  }
}

Run the following command to compile the code:

tsc file.ts --project tsconfig.json

Depending on the configuration, the compiler will generate the following files:

dist
├── file.js
├── file.js.map
├── file.d.ts

• JavaScript code: The compiler will output JavaScript code that can be executed later on.
• Source map: A source map is a file that maps the code within a compressed file back to its original position in a source file to aid debugging. It is mostly used by the browser to map the code it executes back to its original location in the source file.
• Declaration file: A file that provides type information about existing JavaScript code to enable other programs to use the values (functions, variables, …) defined in the file without having to guess what they are.

The TypeScript compiler generates declaration files for all the code it compiles if enabled in tsconfig.json.

What is The TypeScript Compiler API?

The TypeScript Compiler API is an integral part of the TypeScript compiler that exposes various functionalities, enabling you to interact with the compiler programmatically to do stuff like

1. Manual type checking.
2. Code generation.
3. Transform TypeScript code at a granular level.

and more.

TypeScript Compiler API is a lot of interfaces, functions, and classes.

Why would you use the Typescript Compiler API?

Using the TypeScript Compiler API has several benefits, particularly for those interested in building tools around TypeScript. You could utilize the API

1. Write a Language Service Plugin
2. To do Static Code Analysis
3. Or even to build a DSL (Domain Specific Language).
4. Custom Pre/Post build scripts.
5. Code Modification/Migration.
6. Use it as a Front-End for other low-level languages.

Angular recently introduced the Standalone Components, which is a new way to write Angular components without the need to create a module. Angular team created a migration script that does this automatically, and it’s using the Typescript Compiler API.

There are a few interesting projects that utilize the Typescript Compiler API, such as:

• Compile JSONSchema to TypeScript type declarations
• TypeScript AST Viewer
• TypeScript API generator via Swagger scheme

Use Case: Enforce One Class Per File

You’re going to use the Typescript Compiler API to enforce one class per file. This is a common rule that is used in many codebases.

It might be a bit confusing at first, but don’t worry, It’ll get simpler as you go.

Hint: I strongly recommend you check the code again before going to the next use case.

const tsconfigPath = "./tsconfig.json"; // path to your tsconfig.json
const tsConfigParseResult = parseTsConfig(tsconfigPath);
const program = ts.createProgram({
  options: tsConfigParseResult.options,
  rootNames: tsConfigParseResult.fileNames,
  projectReferences: tsConfigParseResult.projectReferences,
  configFileParsingDiagnostics: tsConfigParseResult.errors,
});

/**
 * Apply class per file rule
 */
function classPerFile(file: ts.SourceFile) {
  const classList: ts.ClassDeclaration[] = [];

  // file.forEachChild is a function that takes a callback and
  // calls it for each direct child of the node

  // Loops over all nodes in the file and push classes to classList
  file.forEachChild(node => {
    if (ts.isClassDeclaration(node)) {
      classList.push(node);
    }
  });

  // If there is more than one class in the file, throw an error
  if (classList.length > 1) {
    throw new Error(`
      Only one class per file is allowed.
      Found ${classList.length} classes in ${file.fileName}
      File: ${file.fileName}
  `);
  }
}

const files = program.getSourceFiles();

// Loops over all files in the program and apply classPerFile rule
files
  .filter(file => !file.isDeclarationFile)
  .forEach(file => classPerFile(file));

In this code you’re doing the following:

1. Create a program from tsconfig so you’ve access to all files in the project.
2. Loop over all files in the program and apply the rule.
3. The rule is simple, if there is more than one class in the file, throw an error.

Demo for this use case: To run this code, open the terminal at the bottom and run “npx ts-node ./use-cases/enforce-one-class-per-file/”

Let’s break it down, there are a few key terms that you need to know:

• Program
• Source File
• Node
• Declaration

TypeScript Program

When working with the TypeScript Compiler, one of the central elements you’ll encounter is the Program object. This object serves as the starting point for many of the operations you might want to perform, like type checking, emitting output files, or transforming the source code.

The Program is created using the ts.createProgram function, which can accept a variety of configuration options, such as

• options: These are the compiler options that guide how the TypeScript Compiler will behave. This could include settings like the target ECMAScript version, module resolution strategy, and whether to include type-checking errors, among others.
• rootNames: This property specifies the entry files for the program. It is an array of filenames (.ts) that act as the roots from which the TypeScript Compiler will begin its operations.
• projectReferences: If your TypeScript project consists of multiple sub-projects that reference each other, this property is used to manage those relationships.
• configFileParsingDiagnostics: This property is an array that will capture any diagnostic information or errors that arise when parsing the tsconfig.json file. More on that later.

const tsconfigPath = "./tsconfig.json"; // path to your tsconfig.json
const tsConfigParseResult = parseTsConfig(tsconfigPath);

const program = ts.createProgram({
  options: tsConfigParseResult.options,
  rootNames: tsConfigParseResult.fileNames,
  projectReferences: tsConfigParseResult.projectReferences,
  configFileParsingDiagnostics: tsConfigParseResult.errors,
});

In this sample, a TypeScript program is created from tsconfig parsing results.

Source File

Writing code is actually writing text, you understand it because you know the language, the semantics, the syntax, etc. But the computer doesn’t understand it, it’s just a text.

The compiler will take this text and transform it into something that can be utilised. This transformation is called parsing and its output is called Abstract Syntax Tree (AST).

A source file is a representation of a file in your project, it contains information about the file, such as its name, path, and contents.

The AST is a tree-like data structure and as any tree, it has a root node. The root node is Source File.

Abstract Syntax Tree (AST)

The code you write is essentially a text that isn’t useful unless it can be parsed. That parsing process produces a tree data structure called AST, it contains a lot of information like the name, kind, and position of the node in the source code.

The AST is used by the compiler to understand the code and perform various operations on it. For example, the compiler uses the AST to perform type-checking.

The following code:

function IAmAwesome() {}

will be transformed into the following AST:

{ // -> Source File Node
  "kind": 308,
  "statements": [ // -> Node[]
    { // -> Function Declaration Node
      "kind": 259,
      "name": { // -> Identifier Node
        "kind": 79,
        "escapedText": "IAmAwesome"
      },
      "parameters": [], // Node[]
      "body": { // Block Node
        "kind": 238,
        "statements": []
      }
    }
  ]

Node

In AST, the fundamental unit is called a Node.

Kind: A numeric value that represents the specific type or category of that node. For instance:

• FunctionDeclaration has kind 259
• Block has kind 238

These numbers are exported in an enum called SyntaxKind

The node object has more than just these properties but right now we’re only interested in a few, nonetheless, two additional important properties you might want to know about are:

Parent: This property points to the node that is the parent of the current node in the AST.

Flags: These are binary attributes stored as flags in the node. They can tell you various properties of the node, such as whether it’s a read-only field or it has certain modifiers.

Declaration

Remember the use case you’re trying to solve? enforce one class per file. To do that, you need to check if more than one class is declared in the file.

A declaration is a node that declares something, it could be a variable, function, class, etc.

class Test {
  runner: string = "jest";
}

In this example, we have two declarations:

• Test -> ClassDeclaration
• runner -> PropertyDeclaration

The key difference between a node and a declaration is that

• A “Node” is a generic term that refers to any point in the AST, irrespective of what it represents in the source code.

• A “Declaration” is a specific type of node that has a semantic role in the program: it introduces a new identifier and provides information about it.

Creating a variable const a = 1; is like saying “Hey compiler, I’m creating VariableDeclaration node with name a and value 1”

Statement

A statement is a node that represents a statement in the source code. A statement is a piece of code that performs some action, for example, a variable declaration is a statement that declares a variable.

let a = 1;

In this example the variable declaration let a = 1; is a statement.

Expression

An expression is a node in the code that evaluates to a value. For example

let a = 1 + 2;

The part 1 + 2 is an expression, specifically a binary expression

Another example

const add = function addFn(a: number, b: number) {};

The function ... is an expression, specifically a function expression.

More advanced example

const first = 1,
  second = 2 + 3,
  third = whatIsThird();

• The whole code is a VariableStatement node.
• first = 1, second = 2 + 3 and third = whatIsThird() are a VariableDeclaration nodes.
• first, second, and third are Identifier nodes.
• 1, 2 + 3, and whatIsThird() are NumericLiteral, BinaryExpression, and CallExpression nodes respectively.

Use Case: No Function Expression

Let’s take another example to recap what you’ve learned so far. You’re going to use the Typescript Compiler API to enforce no function expression.

• Function Declaration:

function addFn(a: number, b: number) {
  return a + b;
}

• Function Expression

let addFnExpression = function addFn(a: number, b: number) {
  return a + b;
};

You need to ensure only the first one is allowed.

type Transformer = (
  file: ts.SourceFile
) => ts.TransformerFactory<ts.SourceFile>;
const transformer: Transformer = file => {
  return function (context) {
    const visit: ts.Visitor = node => {
      if (ts.isVariableDeclaration(node)) {
        if (node.initializer && ts.isFunctionExpression(node.initializer)) {
          throw new Error(`
          No function expression allowed.
          Found function expression: ${node.name.getText(file)}
          File: ${file.fileName}
        `);
        }
      }

      // visit each child in this node (look at the visitor node parameter)
      return ts.visitEachChild(node, visit, context);
    };

    // visit each node in the file
    return node => ts.visitEachChild(node, visit, context);
  };
};

const files = program.getSourceFiles();

files
  .filter(file => !file.isDeclarationFile)
  .forEach(file =>
    ts.transform(file, [transformer(file)], program.getCompilerOptions())
  );

I know this isn’t like the first example, but it’s similar.

In the first use case, it was enough to loop over the first level nodes in the file, but in this use case, you need to loop over all nodes in the file, the nested ones as well.

The main logic is within the visit function, it checks if the node is a VariableDeclaration and whether its initializer is a FunctionExpression, and if so, then throws an error.

The looping over the files is the same but with a slight difference, you’re using ts.transform API.

A few key terms that you need to know:

• Transformer
• Visitor
• Context

Demo for this use case: To run this code, open the terminal at the bottom and run “npx ts-node ./use-cases/no-function-expression”

Transformer

As the name implies, the transformer function can transform the AST (the code) in any way you want. In this example, you’re using it to enforce a rule, however, instead of throwing an error, you can transform the code to fix the error (more in the next use case)

Visitor

The visit function is a simpler version of what is called the Visitor Pattern, an essential part of how the TypeScript Compiler API works. Actually, you’ll see that design pattern whenever you work with AST, Hey at least I did!

A “visitor” is basically a function you define to be invoked for each node in the AST during the traversal. The function is called with the current node and has few return choices.

• Return the node as is (no changes).
• Return a new node of the same kind (otherwise might disrupt the AST) to replace it.
• Return undefined to remove the node entirely.
• Return a visitor ts.visitEachChild(node, visit, context) which will visit the node children if have.

Context

The context -TransformationContext- is an object that contains information about the current transformation and has a few methods that you can use to perform various operations like hoistFunctionDeclaration and startLexicalEnvironment. You’ll learn more about it in the next use case.

Use Case: Replace Function Expression With Function Declaration

Same as the previous use case, but instead of throwing an error, you’re going to transform function expression to function declaration.

See how the AST for the function expression looks like

const transformer: Transformer = function (file) {
  return function (context) {
    const visit: ts.Visitor = node => {
      if (ts.isVariableStatement(node)) {
        const varList: ts.VariableDeclaration[] = [];
        const functionList: ts.FunctionExpression[] = [];

        // collect function expression and variable declaration
        for (const declaration of node.declarationList.declarations) {
          // the initializer is expression after assignment operator
          if (declaration.initializer) {
            if (ts.isFunctionExpression(declaration.initializer)) {
              functionList.push(declaration.initializer);
            } else {
              varList.push(declaration);
            }
          }
        }

        for (const functionExpression of functionList) {
          // create function declaration out of function expression
          const functionDeclaration = ts.factory.createFunctionDeclaration(
            functionExpression.modifiers,
            functionExpression.asteriskToken,
            functionExpression.name as ts.Identifier,
            functionExpression.typeParameters,
            functionExpression.parameters,
            functionExpression.type,
            functionExpression.body
          );

          // hoist the function declaration to the top of the containing scope (file)
          context.hoistFunctionDeclaration(functionDeclaration);
        }

        // if the varList (non function expression) is same as the original variable statement, return the node as is.
        // it means there is no function expression in the variable statement
        if (varList.length === node.declarationList.declarations.length) {
          return node;
        }

        // if the varList (non function expression) is empty, return undefined to remove the variable statement node
        if (varList.length === 0) {
          return undefined;
        }

        return ts.factory.updateVariableStatement(
          node,
          node.modifiers,
          ts.factory.createVariableDeclarationList(varList)
        );
      }

      return ts.visitEachChild(node, visit, context);
    };

    return node => {
      // Start a new lexical environment when beginning to process the source file.
      context.startLexicalEnvironment();

      // visit each node in the file.
      const updatedNode = ts.visitEachChild(node, visit, context);

      // End the lexical environment and collect any declarations (function declarations, variable declarations, etc) that were added.
      const declarations = context.endLexicalEnvironment() ?? [];
      const statements = [...declarations, ...updatedNode.statements];

      return ts.factory.updateSourceFile(
        node,
        statements,
        node.isDeclarationFile,
        node.referencedFiles,
        node.typeReferenceDirectives,
        node.hasNoDefaultLib,
        node.libReferenceDirectives
      );
    };
  };
};

A few key terms that you need to know:

• Factory
• Lexical Environment
• Hoisting

Demo for this use case: To run this code, open the terminal at the bottom and run “npx ts-node ./use-cases/replace-function-expression-with-declaration”

Factory

The ts.factory object contains a set of factory functions that can be used to create new nodes or update existing ones. You used ts.factory.createFunctionDeclaration to create a new function declaration node using the information from the function expression node and ts.factory.updateSourceFile to update the source file node statements while keeping the other properties intact.

The factory functions are useful to manipulate the AST in a safe way, without worrying about the details of the AST structure.

Lexical Environment

The lexical environment refers to the scope or context in which variables and function declarations are hoisted and managed during the transformation process.

Here are the key points regarding the lexical environment in your transformer function:

• Scope Management: The lexical environment helps in managing the scope of variables and function declarations. When you start a new lexical environment with context.startLexicalEnvironment(), you are essentially marking the beginning of a new scope. When you end it with context.endLexicalEnvironment(), you are closing off that scope and collecting any declarations that were hoisted to this scope during the transformation process.

• Hoisting: The lexical environment provides the facilities for hoisting function declarations using context.hoistFunctionDeclaration(). Hoisting in this scenario means moving function declarations to the appropriate scope in the source file (the current running scope). For instance, if you have a function declaration inside a function, it will be hoisted to the top of the function, if you have a function declaration inside a for loop, it will be hoisted to the top of the for loop, and so on.

  The previous code only handles the function expression at the top level

• Declaration Collection: The lexical environment also collects the hoisted declarations through context.endLexicalEnvironment(). This method returns an array of Statement nodes that represent the declarations hoisted to the current lexical environment, which can then be included in the SourceFile node.

These methods ensure that the transformer has a mechanism to correctly manage scope and collect hoisted declarations, which is crucial for accurately transforming code while maintaining correct scoping and semantics.

Use Case: Detect Third-Party Classes Used as Superclasses

The last use case is a bit more complex, you’re going to use the Typescript Compiler API to detect third-party classes used as superclasses, not to do anything about it, just detect them which means you don’t need to use the transformer function.

const trackThirdPartyClassesUsedAsSuperClass: ts.Visitor = node => {
  if (ts.isClassDeclaration(node)) {
    const superClass = (node.heritageClauses ?? []).find(
      heritageClause => heritageClause.token === ts.SyntaxKind.ExtendsKeyword
    );

    // Not intrested in classes that don't have super class.
    if (!superClass) {
      return node;
    }

    // in case of class declaration, there will always be one heritage clause (extends)
    const superClassType = superClass.types[0].expression;

    // Get the type checker
    const typeChecker = program.getTypeChecker();

    const symbol = typeChecker.getSymbolAtLocation(superClassType);
    if (!symbol) {
      return undefined;
    }

    const superClassDeclaration = (symbol.declarations ?? []).find(
      ts.isClassDeclaration
    );

    if (!superClassDeclaration) {
      // In this case this should never happen (more on that later),
      // but it's here just to satisfy typescript.
      return node;
    }

    const thisSourceCodeFile = node.getSourceFile();
    const sourceCodeInfo = {
      fileName: thisSourceCodeFile.fileName,
      className: node.name?.text,
      ...ts.getLineAndCharacterOfPosition(thisSourceCodeFile, node.pos),
    };
    const thirdPartyCodeInfo = {
      fileName: superClassDeclaration.getSourceFile().fileName,
      className: superClassType.getText(),
    };
    console.log(`
        Class: "${sourceCodeInfo.className}"
        Filename: "${sourceCodeInfo.fileName}"
        SuperClass: "${thirdPartyCodeInfo.className}"
        Filename: "${thirdPartyCodeInfo.fileName}"
        Line: "${sourceCodeInfo.line}"
        Column: "${sourceCodeInfo.character}"
  `);
  }

  // visit each child in this node
  return ts.forEachChild(node, trackThirdPartyClassesUsedAsSuperClass);
};

const files = program.getSourceFiles();

files
  .filter(file => !file.isDeclarationFile)
  .forEach(file => {
    ts.forEachChild(file, trackThirdPartyClassesUsedAsSuperClass);
  });

Key terms that you need to know:

• HeritageClauses
• Type
• Type Checker
• Symbol

Demo for this use case: To run this code, open the terminal at the bottom and run “npx ts-node ./use-cases/3rd-party-classes/”

HeritageClauses

When you extend a class or implement an interface, you use a clause called a heritage clause and it’s represented by the HeritageClause interface.

interface HeritageClause {
  readonly token: SyntaxKind.ExtendsKeyword | SyntaxKind.ImplementsKeyword;
  readonly types: NodeArray<ExpressionWithTypeArguments>;
}

_ token is the keyword used in the clause, it can be extends or implements._ _ types is an array of ExpressionWithTypeArguments nodes, which represents the types specified in the clause._

types will have a single item if you’re extending a class or multiple items if you’re implementing interfaces.

In this example, you’re only interested in the extends hence const [superClass] = node.heritageClauses

Type

It is the “Type” part in TypeScript 😏, representing the type information of a particular symbol in the AST.

A Type object is associated with a Symbol to represent the type of the symbol (named entity).

interface Type {
  flags: TypeFlags;
  symbol: Symbol;
}

• flags: The flags associated with the type which can be used to determine the kind of type -number, string, boolean, etc.-

• symbol: The symbol associated with the type (scroll a bit to read about symbols)

Type Checker

It is an essential part of the TypeScript Compiler API, acting as the engine that powers the type system in TypeScript. The Type Checker traverses the AST, examining the Type and Symbol of nodes to ensure they adhere to the type annotations and constraints defined in the code.

The Type Checker can be accessed via the program object, and it provides a rich set of methods to retrieve type information, check types, and obtain symbol information, among other functionalities.

const typeChecker = program.getTypeChecker();

In this use case, you used typeChecker to get the type of the class declaration node and the symbol associated with it, which you then used to get the value declaration of the symbol.

const type = typeChecker.getTypeAtLocation(node);
const symbol = type.symbol;
const valueDeclaration = symbol.valueDeclaration;

TypeChecker has a lot of methods to facilitate working with types, symbols, and nodes.

Symbol

The term “symbol” refers to a named entity in the source code that could represent variables, functions, classes, and so forth.

named entity is a name that is used to identify something. It could be a variable name, function name, class name, etc. In other words any Identifier Node.

A symbol is represented by the Symbol interface, which has the following properties:

interface Symbol {
  flags: SymbolFlags;
  escapedName: __String;
  declarations?: Declaration[];
  valueDeclaration?: Declaration;
  members?: SymbolTable;
  exports?: SymbolTable;
  globalExports?: SymbolTable;
}

• flags: The flags associated with the symbol which can be used to determine the type of symbol -variable, function, class, interface, etc.-
• escapedName: The name of declarations associated with the symbol.
• declarations: List of declarations associated with the symbol, think of function/method override.
• valueDeclaration: Points to the declaration that serves as the “primary” or “canonical” declaration of the symbol. In simpler terms, it’s the declaration that gives the symbol its “value” or “meaning” within the code. For example, if you have a variable initialized with a function, the valueDeclaration would point to that function expression.
• members: Symbol table that contains information about the properties or members of the symbol. For instance, if the symbol represents a class, members would contain symbols for each of the class’s methods and properties.
• exports: Similar to members, but this is more relevant for modules. It contains the symbols that are exported from the module.
• globalExports: This is another symbol table but it’s used for global scope exports.

Let’s take the following example

class Test {
  runner: string = "jest";
}

The symbol for class “Test” will have the following properties:

const classTestSymbol = {
  flags: 32,
  escapedName: "Test",
  declarations: [ClassDeclaration],
  valueDeclaration: ClassDeclaration,
  members: {
    runner: // Symbol for runner property
  },
};

Symbols let the type checker look up names and then check their declarations to determine types. It also contains a small summary of what kind of declaration it is — mainly whether it is a value, a type, or a namespace.

Diagnostic

When the TypeScript compiler runs, it performs several checks on the code. If it encounters something that doesn’t align with the language rules or the project’s configuration, it creates an object that represents identified issues the compiler has found in the code.

Each diagnostic object contains information about the nature of the problem, including:

• The file in which the issue was found.
• The start and end position of the relevant code.
• A message describing the issue.
• A diagnostic category (error, warning, suggestion, or message).
• A code that uniquely identifies the type of diagnostic.

The issues can range from syntax errors and type mismatches to more subtle semantic issues that might not be immediately apparent. Mainly, there are two kinds of diagnostics:

• Syntactic Diagnostics: These are errors that occur when the compiler parses the code and encounters something that doesn’t align with the language syntax. For instance, if you have a missing semicolon or a missing closing brace, the compiler will generate a syntactic diagnostic.

• Semantic Diagnostics: These are errors that occur when the compiler performs type-checking on the code and encounters something that doesn’t align with the type annotations or constraints defined in the code. For instance, if you have a variable of type string and you try to assign a number to it, the compiler will generate a semantic diagnostic.

Here’s a simple example of how you might encounter a diagnostic in TypeScript:

let greeting: string = 42;

The TypeScript compiler will generate a diagnostic message for the above code, indicating that the type ‘number’ is not assignable to the type ‘string’.

To obtain semantic diagnostics, you can use the getSemanticDiagnostics method on the program object. This method returns an array of diagnostic objects, each of which contains information about the issue, such as the file in which it was found, the start and end position of the relevant code, and a message describing the issue.

program.getSemanticDiagnostics();

Will return

[
  {
    "start": 0,
    "length": 8,
    "code": 2322,
    "category": 1,
    "messageText": "Type 'number' is not assignable to type 'string'.",
    "relatedInformation": undefined
  }
]

To obtain syntactic diagnostics

// missing expression after assignment operator
let greeting: string =;

// get syntactic diagnostics
program.getSyntacticDiagnostics();

Will return

[
  {
    "start": 0,
    "length": 1,
    "messageText": "Expression expected.",
    "category": 1,
    "code": 1109,
    "reportsUnnecessary": undefined
  }
]

Diagnostics are not just for the compiler; they are also used by IDEs and editors to provide real-time feedback to developers. For example, when you see red squiggly lines under code in Visual Studio Code, that’s the editor using TypeScript diagnostics to indicate an issue.

Here’s a snippet that demonstrates how to retrieve diagnostics for a given TypeScript program before emitting the output files:

// Retrieve and concatenate all diagnostics
const allDiagnostics = ts.getPreEmitDiagnostics(program);

// Iterate over diagnostics and log them
allDiagnostics.forEach(diagnostic => {
  if (diagnostic.file) {
    let { line, character } = diagnostic.file.getLineAndCharacterOfPosition(
      diagnostic.start!
    );
    let message = ts.flattenDiagnosticMessageText(diagnostic.messageText, "\n");
    console.log(
      `${diagnostic.file.fileName} (${line + 1},${character + 1}): ${message}`
    );
  } else {
    console.log(ts.flattenDiagnosticMessageText(diagnostic.messageText, "\n"));
  }
});

You might have noticed the function says pre-emit getPreEmitDiagnostics which implies there will be diagnostics after the emit process, and you’re right, there are.

The post-emit diagnostics are generated after the emit process, and they are used to report issues that might arise during the emit process. For instance, there might be issues with generating source maps or writing output files.

Printer

TypeScript provides a printer API that can be used to generate source code from an AST, it is useful when you want to perform transformations on the AST and then generate the corresponding source code in text format.

function print(file: ts.Node, result: ts.TransformationResult<ts.SourceFile>) {
  const printer = ts.createPrinter({
    newLine: ts.NewLineKind.LineFeed,
  });
  const transformedSource = printer.printNode(
    ts.EmitHint.Unspecified,
    result.transformed[0],
    file
  );
  return transformedSource;
}

In use case 3, you’ll need to write the modified AST to the original file back, to do that you’ll need to use the printer API to get the AST into text.

Worth mentioning that the printer doesn’t preserve the original formatting of the source code.

For this exact reason I don’t recommend direct use of the compiler API to modify source code, instead, I use ts-morph which is a wrapper around the compiler API that preserves the original formatting.

How The Compiler Works

The following is a high-level overview.

Lexical Analysis

In this stage, the compiler takes the source code and breaks it down into a series of tokens, a token is a character or sequence of characters that can be treated as a single unit.

function IAmAwesome() {}

Will be transformed into the following tokens:

<function, keyword>
<IAmAwesome, identifier>
<(, punctuation>
<), punctuation>
<{, punctuation>
<}, punctuation>

The module/function that does this is called the Scanner/Tokenizer which is responsible for scanning the source code and generating the tokens. When the term scanner/tokenizer is used, imagine a while loop that loops over the source code character by character and switch case to determine the token type.

function scan(sourceCode: string) {
  const tokens: Token[] = [];
  let currentChar: string;
  let index = -1;
  do {
    index = index + 1;
    currentChar = sourceCode[index];
  } while (index < sourceCode.length);
  {
    switch (currentChar) {
      case "{":
        // ...
        break;
      case "}":
        // ...
        break;
      // ...
    }
  }
}

Syntax Analysis

In this stage, the compiler takes the tokens generated in the previous stage and uses them to build a tree-like structure called an Abstract Syntax Tree (AST). The AST represents the syntactic structure of the source code.

The part of the compiler that does this is called the Parser which is responsible for parsing the tokens and building the AST that is then used by the semantic analysis stage.

So the Tokensizer generates the tokens and the Parser builds the AST. You may be wondering (are you?) how the parser knows what the AST should look like, well, it’s defined in the Language Grammar.

A grammar is a set of rules that define the syntax of a language. Think of English grammar, it defines the rules of the English language, such as how to form a sentence, how to use punctuation, etc.

Grammar is usually represented in a form called Backus-Naur Form (BNF) or Antlr, which is a notation that describes the grammar of a language.

A simple example of grammars:

<identifier> ::= "a" | "b" | "c" | ... | "z"
<function> ::= "function" <identifier>* "(" ")" "{" "}"

The parser will use the grammar to build the AST, in case of writing a function, the parser will look for the function keyword, then the identifier, the (, the ), the {, then the } in case of any of these tokens is missing, the parser will throw an error. Read More on parsing

Another example

 <postal-address> ::= <name-part> <street-address> <zip-part>

    <name-part> ::= <personal-part> <last-name> <opt-suffix-part> <EOL> | <personal-part> <name-part>

  <personal-part> ::= <first-name> | <initial> "."

 <street-address> ::= <house-num> <street-name> <opt-apt-num> <EOL>

       <zip-part> ::= <town-name> "," <state-code> <ZIP-code> <EOL>

<opt-suffix-part> ::= "Sr." | "Jr." | <roman-numeral> | ""
    <opt-apt-num> ::= "Apt" <apt-num> | ""

Semantic Analysis

At this stage, the TypeScript compiler performs semantic analysis on the AST generated during the parsing stage to ensure that the syntactically correct code is also semantically valid. For instance, consider the following TypeScript snippet:

let a = 1;
a();

While the parser confirms this code is free of syntax errors, semantic analysis reveals a type error. This code tries to invoke a as if it were a function, which is semantically incorrect since a is a number.

The semantic analysis phase comprises two key processes: Binding and Type Checking.

Binding: It walks the AST and creates symbols for all declarations it encounters, such as variables, functions, classes, etc.

• A Symbol is created for each declaration, capturing its name, type, scope, and additional attributes.
• The symbol is then stored in a SymbolTable, which is essentially a map that associates symbols with their names and is scoped to a specific block, function, or module.

The internal representation of a SymbolTable in TypeScript might look something like this:

type SymbolTable = Map<__String, Symbol>;

The function to create such a table could be as follows:

export function createSymbolTable(symbols?: readonly Symbol[]): SymbolTable {
  const result = new Map<__String, Symbol>();
  if (symbols) {
    for (const symbol of symbols) {
      result.set(symbol.escapedName, symbol);
    }
  }
  return result;
}

Read more on the binding process

Type Checking: The Type Checker takes over after binding. It uses the symbol table to verify that each symbol’s usage is consistent with its declared type. In the provided code example, the type checker identifies that a is a number and cannot be invoked, raising a compile-time error.

While Binding and Type checking are distinct processes, they are interdependent. The Binder populates the symbol table necessary for the Type Checker to verify the correct usage of types. Conversely, the Type Checker may influence the binding process, particularly in complex scenarios involving type inference or generics.

Emitting

This is where the compiler generates the output files, such as JavaScript files, source maps, and declaration files after the AST has been successfully created and type-checked.

project.emit();

Recall diagnostics? the emit process will fail if there are any errors but that can be overridden by setting noEmitOnError to true.

ESLint

Using ESLint is a better option if you want to enforce rules on your codebase because it’s specifically built for that purpose. However, if you want to build a tool that does something more complex, then the Typescript Compiler API is the way to go.

I have written a guide for you to learn more.

Conclusion

The TypeScript Compiler API is a powerful tool that can be used to build tools around TypeScript. It provides a rich set of functionalities that can be used to perform various operations on the source code, such as type checking, code generation, and AST transformation.

It’s a bit confusing at first, but once you get the hang of it, you’ll be able to build some cool stuff.

You can also generate TypeScript code using the API, think of protobuf to TypeScript code generator.

Next Steps

Practice, practice, practice. The best way to learn is to practice what you’ve learned here. Try to build a tool that does something useful for you or your team.

Think of manual tasks -typescript-related- that you do often and try to automate them. Stuff that is error-prone, time-consuming, or just boring.

I’m writing a few other blog posts that will help you get started with the Typescript Compiler API, follow me on Twitter to get notified when they’re out.

References

• Codebase Compiler Binder
• TypeScript Compiler Notes
• TypeScript Transformer Handbook
• TypeScript Source Code
• What Is Semantic Analysis in a Compiler?

Related Projects

• OXC
• ts-morph
• jscodeshift

Credits

• The graph tree was generated using Mermaid JS
• The drawing was created using Excalidraw
• Image caption were generated using ChatGPT

You know how ChatGPT continuously show the results as it becomes available -generated-, It doesn’t wait for the whole response to get together, rather it shows what the model is generating right away.

• Smooth user experience: User doesn’t have to wait till the whole response is generated.

• Easier to read: When ChatGPT first released I liked that it respond as if you’re talking to someone, which read different to other form of writing.

• Reduce memory usage: This is a benefit of streaming in general, you offload the data without having to buffer it in memory.

In this article, you’re going to learn how you can stream OpenAI response just like how ChatGPT does it.

I’m assuming you’ve basic knowledge in Node.js, and you’re familiar with JavaScript features like async/await, for await loop and async iterators.

Table of Content

• Problem
• Solution
  • The Server Side
  • The Client Side
    • Using Async Iterator
    • Using While Loop
    • Using RxJS
• Decoding The Data
• Canceling The Request
  • The Client Side
  • The Server Side
• Handling Errors
  • Server Side
  • Client Side
• Backpressure & Buffering
• More On Streams
• Conclusion
• Next Steps
• References

Problem

There are few ways to stream data from the server:

• Server-Sent Events
• WebSockets
• Long Polling
• Fetch API

You’re going to use the Fetch API; simple and more suitable for this use case.

Solution

I divided the solution into two parts, the server side and the client side.

The Server Side

You’re going to build a tiny Node.js server that uses OpenAI chat completion API.

Let’s make simple Node.js server first. No routing framework, no fancy stuff.

import http from "http";

const server = http.createServer();
server.listen(3000);

server.on("request", async (req, res) => {
  switch (req.url) {
    case "/":
      res.write("Hello World");
      res.end();
      break;
    default:
      res.statusCode = 404;
      res.end();
  }
});

The res.write will send data to the client, it can be called many times only before the res.end is called as it will close the connection.

import OpenAI from "openai";
const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
});

const config = {
  model: "gpt-4",
  stream: true,
  messages: [
    {
      content: "Once upon a time",
      role: "user",
    },
  ],
};
const completion = await openai.chat.completions.create(config);

The stream option is what you’re looking for, it will stream the response body.

The completion object implements the AsyncIterable interface, which means you can use for await loop to iterate over the response body.

for await (const chunk of completion) {
  console.log(chunk);
}

Putting it all together.

import OpenAI from "openai";
import http from "http";

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
});

const server = http.createServer();
server.listen(3000);

server.on("request", async (req, res) => {
  switch (req.url) {
    case "/":
      const config = {
        model: "gpt-4",
        stream: true,
        messages: [
          {
            content: "Once upon a time",
            role: "user",
          },
        ],
      };
      const completion = await openai.chat.completions.create(config);

      res.writeHead(200, {
        "Content-Type": "text/plain; charset=utf-8",
      });

      for await (const chunk of completion) {
        const [choice] = chunk.choices;
        const { content } = choice.delta;
        res.write(content);
      }

      res.end();
      break;
    default:
      res.statusCode = 404;
      res.end();
  }
});

In the context of streaming data, a chunk refers to a piece or fragment of data that is handled, processed, or transmitted individually as part of a larger stream of data.

The Client Side

There are few ways to receive data coming from the server while using the Fetch API.

• Using for await loop over the body stream.
• Manually using response.body!.getReader().
• Using 3rd party libraries like RxJS.

Of course, those are not the only ways.

Using Async Iterator

The approach I like the most, it is declarative and to the point.

You might want to check browser compatibility iterating over ReadableStream. TypeScript also doesn’t support ReadableStream as an iterable.

const inputEl = document.querySelector("input");
const resultEl = document.querySelector("p");

inputEl.addEventListener("input", async () => {
  const response = await fetch("http://localhost:3000");

  let total = "";
  const decoder = new TextDecoder();
  for await (const chunk of response.body!) {
    const decodedValue = decoder.decode(chunk);
    total += decodedValue;
    resultEl.textContent = total;
  }
});

Using While Loop

You’ll most likely see this approach in the wild, it is a bit imperative and verbose but it is supported by most browsers.

const inputEl = document.querySelector("input");
const resultEl = document.querySelector("p");

inputEl.addEventListener("input", async () => {
  const response = await fetch("http://localhost:3000");

  // Using While Loop
  let total = "";
  const reader = response.body!.getReader();
  const decoder = new TextDecoder();
  while (true) {
    const { done, value: chunk } = await reader.read();
    if (done) break;

    const decodedValue = decoder.decode(chunk);
    total += decodedValue;
    resultEl.textContent = total;
  }
});

Using RxJS

const inputEl = document.querySelector("input");
fromEvent(inputEl, "input")
  .pipe(
    switchMap(() => fetch("http://localhost:3000")),
    switchMap(response => response.body.pipeThrough(new TextDecoderStream())),
    scan((acc, chunk) => acc + chunk, "")
  )
  .subscribe(total => {
    resultEl.textContent = total;
  });

RxJS is a great library, but it is a bit overkill for this use case, unless you’re already using it in your project.

Decoding The Data

A word on the decoding part, you need to decode the data coming from the server, as it is encoded using Uint8Array by default.

To decode the data, you need to use TextDecoder API.

const decoder = new TextDecoder();

By default it uses utf-8 encoding, which is what you need in most cases.

const decoder = new TextDecoder("utf-8");

The decode method accepts buffer (An ArrayBuffer, a TypedArray, or a DataView) and returns a string.

const decodedValue = decoder.decode(chunk);

Another options is using TextDecoderStream which is a transform stream that takes a stream of Uint8Array chunks as input and emits a stream of strings.

The previous implementation can be rewritten as follows.

let total = "";
const decoderTransformStream = new TextDecoderStream();
const reader = response.body.pipeThrough(decoderTransformStream).getReader();
while (true) {
  const { done, value } = await reader.read();
  if (done) break;

  total += value;
  resultEl.textContent = total;
}

Canceling The Request

To cancel the request in progress, you need to use the AbortController API.

The Client Side

• Cancel the request in progress if you allow the user to generate over the request in progress.

let controller = new AbortController();
let inProgress = false;
inputEl.addEventListener("input", async () => {
  if (inProgress) {
    controller.abort();
    controller = new AbortController();
  }

  inProgress = true;

  const response = await fetch("http://localhost:3000", {
    signal: controller.signal,
  });
  // ... rest of the code

  inProgress = false;
});

• Deliberately cancel the request in progress.

const stopGenerationButtonEl = document.querySelector("button");

stopGenerationButtonEl.addEventListener("click", () => {
  if (inProgress) {
    controller.abort();
  }
});

RxJS has a built-in support for canceling the request in progress using switchMap operator.

To deliberately cancel the request in progress, you can use takeUntil operator.

const stopGeneration$ = new Subject();
fromEvent(inputEl, "input").pipe(
  switchMap(() =>
    fetch("http://localhost:3000").pipe(
      takeUntil(fromEvent(stopGenerationButtonEl, "click"))
    )
  )
  // ... rest of the code
);

The Server Side

You need to listent to the close event on the request object which will be emitted when the client closes the connection.

Aborting the request is one of the reasons a connection might close, of other reasons might be the client closed the tab or the browser crashed.

req.on("close", () => {
  completion.controller.abort();
});

The close listener is the place where you want to do clean up, besides aborting OpenAI request you might want to close any database connection or any other resource you’re using.

Handling Errors

Following OpenAI API error codes, you need to handle the most likely error you’ll get, at least.

• Rate Limit Reached - Quota Exceeded (same error code)
• Model Is Overloaded (will be in form of a server error)
• Token Limit Reached

To commuincate the errors to the client, you’re going to prefix the error with ERROR string to simplify handling the error on the client side.

You cannot alter the response status code at this point, it will always be 200 even if the error is in form of a server error.

Server Side

• Rate Limit Reached
• Model Is Overloaded

try {
  const completion = await openai.chat.completions.create(config);
} catch (error) {
  if (error instanceof RateLimitError) {
    res.end("ERROR:rate_limit_exceeded");
  } else if (error instanceof InternalServerError) {
    res.end("ERROR:internal_server_error");
  }
}

• Token Limit Reached

for await (const chunk of completion) {
  const [choice] = chunk.choices;
  const { content } = choice.delta;

  if (choice.finish_reason === "length") {
    res.write("ERROR:token_limit_reached");
  } else {
    res.write(content);
  }
}

Client Side

while (true) {
  const { done, value: chunk } = await reader.read();
  if (done) break;

  const decodedValue = decoder.decode(chunk);

  switch (decodedValue) {
    case "ERROR:rate_limit_exceeded":
      resultEl.textContent = "Rate Limit Exceeded";
      break;
    case "ERROR:internal_server_error":
      resultEl.textContent = "Internal Server Error";
      break;
    case "ERROR:token_limit_reached":
      resultEl.textContent = "Token Limit Reached";
      break;
    default:
      total += decodedValue;
      resultEl.textContent = total;
  }

  total += decodedValue;
  resultEl.textContent = total;
}

Backpressure & Buffering

What if the client is slow to consume the data coming from the server -OpenAI-, you don’t want to buffer too much data in the server memory. Things can get pretty bad when you have many concurrent requests.

In Node.js you can check if the client is ready to receive more data by checking the return value of res.write method.

res.write returns false if the buffer is full, that can be known synchronously; to know when the buffer is ready to receive more data, you need to listen to the drain event on the response object.

// ... rest of the code
const bufferFull = res.write(content) === false;
if (bufferFull) {
  await new Promise(resolve => res.once("drain", resolve));
}

That is actually less than a solution, merely a workaround. A better solution would be to store the data somewhere else, like in-memory database, and stream the data from there when the client is ready to receive more.

More On Streams

Beside the solutions you read, stream is vast concept and have many components, mainly categoriesd into the following.

• ReadableStream: Represents a source of data, from which you can read. In the client-side, the server can be considered as a readable stream that you access using the Fetch API. In the server-side, the OpenAI API can be considered as a readable stream.

• WriteableStream: Opposite to ReadableStream, a WritableStream represents a destination for data, where you can write. In the server-client setup, it represent the response object where you are writing the data to be sent to the client.

• TransformStream: A TransformStream is a type of stream that sits between a ReadableStream and a WritableStream, transforming or processing the data as it passes through. This can be utilized for various purposes such as modifying, filtering, or formatting the data before it reaches the client. For instance, if there’s a need to format the OpenAI’s response before sending it to the client, a TransformStream could be employed.

When combined together, they form a pipeline, where the data flows from the source to the destination, passing through the transform stream. For instance, data from OpenAI’s API (ReadableStream) could be passed through a TransformStream for formatting or filtering, and then written to the client (WritableStream).

Conclusion

So I hope you have good idea how to mimic ChatGPT in your project, and how to stream data from the server to the client.

Streaming is fun but you’ve to keep an eye on the memory usage on both client and server, you don’t want to buffer too much data in the memory, try not to hold any data, always stream quick and early.

In case you’re using more complex form of Prompt Engineering, like ReAct or Generated Knowledge that relys on the output of a previous LLM request, and have processing within like collecting user info, semantic search, you might then want use Server-Sent Events or WebSockets instead of Fetch API.

Next Steps

1. Put that in practice if you haven’t already.
2. Use some kind of in-memory database to store the data and stream it from there.
3. Try to stress load the server and see how it behaves.

References

• Node.js Stream
• Streams API
• An Introduction to Streaming on the Web
• https://web.dev/articles/eventsource-basics
• [https://jakearchibald.com/2016/streams-ftw/#streams-the-fetch-api](2016 - the year of web streams) I have this bookmarked since 2016 😆

You know when you start typing in a search box and it starts suggesting things to you? That’s called typeahead. It’s a great way to help users find what they’re looking for. In this article, you’ll learn how to build a typeahead component using RxJS.

Table of Content

• Problem
• Solution
• Getting Started
• Typeahead Operator
  • Options Interface
  • Typeahead Operator
• Scenario 1: Typical Typeahead
• Scenario 2: Cache Results
• Scenario 3: The Edge Case
• Example
  • Framework Agnostic Example
  • Angular Example
• Backpressure
• Conclusion

Problem

From end-user perspective, the problem is that they want to search for something and they want to see the results as they type. From developer perspective, the problem is that you need to optimize the search logic to not pressure the server with too many requests.

Solution

To balance the user experience and the performance, you need to make sure that you don’t make too many requests. That can be done using RxJS to debounce the user input and only make a request when the user stops typing for a certain amount of time.

Here is how it’ll be used in the end:

const search$ = fromEvent(searchInputEl, "input").pipe(
  map(event => event.target.value),
  typeahead({
    minLength: 3,
    debounceTime: 250,
    loadFn: searchTerm => {
      const searchQuery = searchTerm ? `?title_like=^${searchTerm}` : "";
      return fetch(`https://jsonplaceholder.typicode.com/posts${searchTerm}`);
    },
  }),
  // convert the response to json
  switchMap(response => response.json())
);

Getting Started

Before you start, you need to install RxJS.

npm install rxjs

Note: I’m using TypeScript primarily for clarity in showing what options are available through types. You’re free to omit them, but if you do want to use types, I’d suggest opting for a framework that has built-in TypeScript support.

Typeahead Operator

You’ll write a custom operator that will take an object with the following properties:§

Options Interface

interface ITypeaheadOperatorOptions<Out> {
  /**
   * The minimum length of the allowed search term.
   */
  minLength: number;
  /**
   * The amount of time between key presses before making a request.
   */
  debounceTime: number;
  /**
   * Whether to allow empty string to be treated as a valid search term.
   * Useful for when you want to show defaul results when the user clears the search box
   *
   * @default true
   */
  allowEmptyString?: boolean;

  /**
   * The function that will be called to load the results.
   */
  loadFn: (searchTerm: string) => ObservableInput<Out>;
}

Typeahead Operator

export function typeahead<Out>(
  options: ITypeaheadOperatorOptions<Out>
): OperatorFunction<string, Out> {
  return source => {
    return source.pipe(
      ...operators
      // The implementation goes here
    );
  };
}

The typeahead custom operator accepts options/config object and returns an operator function that takes an observable of “search term” and returns an observable of “result”.

The result is represented by the generic type Out which is the type of the result returned by the loadFn function.

Scenario 1: Typical Typeahead

Note: Valid search term is a search term that have been still for a certain amount of time (debounceTime) and satisfies the minimum length (e.g. at least 3 char).

• The user types valid search term and the request is sent.
• The user types a new valid search term before the current request is finished, the current request is canceled and a new one is sent with the new search term.
• The user types a valid search term then before the debounce time is up, the user reverts back to the previous search term, no request is sent.

return source.pipe(
  debounceTime(options.debounceTime),
  filter(value => typeof value === "string"),
  filter(value => {
    if (value === "") {
      return options.allowEmptyString ?? true;
    }
    return value.length >= options.minLength;
  }),
  distinctUntilChanged(),
  switchMap(searchTerm => options.loadFn(searchTerm))
);

debounceTime: Think of it as sliding window. It will wait for a certain amount of time before emitting the last value. If a new value comes in before the time is up, it will reset the timer and the window will start over. This is useful for preventing requests with every keystroke.

filter: The first filter will only pass values that are of type string, it might sound reddundant but it’s necessary because the debounceTime operator might emit null when the source observable completes. The second filter will only pass values that are longer than the minimum length or empty string (if allowed).

Note: You’ll need empty string to be treated as a valid search term if you want to show default results when the user clears the search box or when the user opens the page/dropdown for the first time.

distinctUntilChanged: It will only emit a value if it’s different from the previous one (default behavior). This is useful for preventing requests with same search term (duplicate request). For example, if the user types “rxjs”, results are fetched. If the user types “rxjs” again, no need to fetch the results again.

switchMap: Cancels the previous observable and subscribe to the new one. If a user types new search term before the current request is finished, it will cancel the current request and start a new one.

For example, if the user types “Typeahead” and then types “Operator” before the request for “Typeahead” is finished, it will cancel the request for “Typeahead” and send a new one for “Typeahead Operator”. Only in the case that the request for “Typeahead” took longer than the debounce time.

Scenario 2: Cache Results

Most of the use cases I’ve personally seen might benefit from caching the results, especially if the results are not going to change frequently.

You can do that by caching the in-flight observables using the shareReplay operator.

const cache: Record<string, Observable<Out>> = {};
return source.pipe(
  // ... same operators
  switchMap(searchTerm => {
    // Initialize Observable in cache if it doesn't exist
    if (!cache[searchTerm]) {
      cache[searchTerm] = options.loadFn(searchTerm).pipe(
        shareReplay({
          bufferSize: 1,
          refCount: false,
          windowTime: 5000,
        })
      );
    }

    // Return the cached observable
    return cache[searchTerm];
  })
);

shareReplay: It will return the source observable (the request associated with the search term) if it exists in the cache, otherwise it will initialize it and cache it for future use.

bufferSize: presuming you’re doing HTTP calls, then it’ll be one response.

windowTime: essentially means that the cached observable will be removed from the cache after 5 seconds.

There are two other optimizations techniques that I can think of:

1. return the cached observable immediately after the user types the search term if it exists in the cache without having to go through the debounce time. I have deliberately decided not to do that so the user can have consistent experience. In my case the debounce time is 1.5s.

2. Inspired by stale-while-revalidate strategy, you can return the cached observable immediately and then send a new request to update the cache. That can be done using concat operator.

Scenario 3: The Edge Case

In most cases you’ll be fine with the previous implementation, but there is an edge case that you might need to handle.

• The user types a valid search term (Angular) and the request is sent.
• The user types an invalid search term (Ng) before the current request is finished “(Angular) request”. Since the search term is invalid, switchMap didn’t get the chance to receive it, hence it didn’t cancel the current request.
• The user typed back (Angular) but the default behaviour of distinctUntilChanged won’t allow it to pass through.

let shouldAllowSameValue = false; // -> 1
return source.pipe(
  distinctUntilChanged((prev, current) => {
    if (shouldAllowSameValue /** -> 3 */) {
      shouldAllowSameValue = false;
      return false;
    }
    return prev === current; // -> 4
  }),
  switchMap(searchTerm =>
    // -> 5
    from(options.loadFn(searchTerm)).pipe(
      takeUntil(
        source.pipe(
          tap(() => {
            shouldAllowSameValue = true; // -> 2
          })
        )
      )
    )
  )
);

Let’s break it down (follow the numbers in the code comments):

1. shouldAllowSameValue is a flag that will be used to allow the same value to pass through the distinctUntilChanged operator. It’s set to false by default.

2. shouldAllowSameValue is set to true when the user types an invalid search term before the current request is finished.

3. If shouldAllowSameValue is true, it means that the user typed an invalid search term before the last request is finished. In that case, we want to allow the same value to pass through the distinctUntilChanged operator.

4. This is the default behavior of distinctUntilChanged, it will only emit a value if it’s different from the previous one.

5. Converts the loadFn function to an observable, subscribes to it, and cancels it when the source observable emits a new value while the request is still in progress.

distinctUntilChanged: It will only emit a value if it’s different from the previous one (default behavior). This is useful for preventing requests with same search term (duplicate request). For example, if the user types “hello”, results are fetched. If the user types “hello” again, we don’t want to fetch the results again.

switchMap: Cancels the previous observable and subscribe to the new one. If a user types new search term before the current request is finished, it will cancel the current request and start a new one.

Complete code

export function typeahead<Out>(
  options: ITypeaheadOperatorOptions<Out>
): OperatorFunction<string, Out> {
  let shouldAllowSameValue = false;
  return source => {
    return source.pipe(
      debounceTime(options.debounceTime),
      filter(value => typeof value === "string"),
      filter(value => {
        if (value === "") {
          return options.allowEmptyString ?? true;
        }
        return value.length >= options.minLength;
      }),
      distinctUntilChanged((prev, current) => {
        if (shouldAllowSameValue) {
          shouldAllowSameValue = false;
          return false;
        }
        return prev === current;
      }),
      switchMap(searchTerm =>
        from(options.loadFn(searchTerm)).pipe(
          takeUntil(
            source.pipe(
              tap(() => {
                shouldAllowSameValue = true;
              })
            )
          )
        )
      )
    );
  };
}

Example

Framework Agnostic Example

import { fromEvent } from "rxjs";
import {
  debounceTime,
  distinctUntilChanged,
  filter,
  switchMap,
} from "rxjs/operators";

const searchInputEl = document.getElementById("search-input");
const resultsContainerEl = document.getElementById("results-container");

const search$ = fromEvent(searchInputEl, "input").pipe(
  map(event => searchInputEl.value),
  typeahead({
    minLength: 3,
    debounceTime: 1000,
    loadFn: searchTerm => {
      const searchQuery = searchTerm ? `?title_like=^${searchTerm}` : "";
      return fetch(`https://jsonplaceholder.typicode.com/posts${searchTerm}`);
    },
  }),
  // convert the response to json
  switchMap(response => response.json())
);

search$.subscribe(results => {
  resultsContainerEl.innerHTML = results
    .map(result => `<li>${result.title}</li>`)
    .join("");
});

<input type="text" id="search-input" />
<ul id="results-container"></ul>

Keep in mind that it’s the practice to use switchMap and not other flattening operators like mergeMap or concatMap in this scenario.

Note: I have deliberately left out the error handling for the sake of simplicity, you might want to implement retry logic or show an error message to the user.

Angular Example

import { Component } from "@angular/core";
import { HttpClient } from "@angular/common/http";
import { FormControl } from "@angular/forms";
import { Observable } from "rxjs";
import {
  debounceTime,
  distinctUntilChanged,
  filter,
  switchMap,
} from "rxjs/operators";

@Component({
  selector: "app-search-bar",
  template: `
    <input type="text" [formControl]="searchControl" />
    <ul>
      <li *ngFor="let result of results$ | async">{{ result.title }}</li>
    </ul>
  `,
})
export class SearchBarComponent {
  searchControl = new FormControl();
  results$: Observable<any[]>;

  constructor(private http: HttpClient) {
    this.results$ = this.searchControl.valueChanges.pipe(
      typeahead({
        minLength: 3,
        debounceTime: 300,
        loadFn: searchTerm => {
          const searchQuery = searchTerm ? `?title_like=^${searchTerm}` : "";
          return this.#http.get<any[]>(
            `https://jsonplaceholder.typicode.com/posts${searchQuery}`
          );
        },
      })
    );
  }
}

Backpressure

Simply put, it’s the pressure of too much incoming data that our system can’t handle at once. Think of conveyer belt in a factory, if the belt is moving too fast, the workers won’t be able to keep up with the incoming products.

In a typeahead scenario, If we let every keystroke from every user hit our server for query processing, we’re going to overwhelm it faster than a JavaScript framework becomes outdated. In technical terms, this rapid influx of data can create a bottleneck, leading to increased latency and resource consumption.

This is less of a concern to the frontend developer, as their main focus is usually on user experience rather than backend scalability. However, it’s essential to understand that the choices made on the frontend, like how often to trigger server requests, can have a direct impact on backend performance.

Conclusion

In this article, we learned how to build a typeahead component using RxJS. We also learned about backpressure and how it can affect our application’s performance. I hope you found this article helpful and that it will help you build better applications in the future.

You can take this further and apply Infinite Scroll along with typeahead to have a unique user experience.

Have you ever experienced slow loading or lag on a webpage while trying to load a large amount of data? If so, you’re not alone. An effective solution to improve the experience is to use infinite scrolling, which works similarly to how your Twitter feed continuously loads more tweets as you scroll down.

What is Infinity Scroll

A web design technique where, as the user scrolls down a page, more content automatically and continuously loads at the bottom, eliminating the user’s need to click to the next page.

Scroll down for the result, Or see the complete code

Using Angular? Here’s a detailed implementation

Table Of Content

• Problem
• Solution
• Getting Started
• RxJS operators
  • Common RxJS Operators
    • filter
    • map
    • tap
    • finalize
    • debounceTime
    • startWith
    • fromEvent
    • takeUntil
    • pipe
  • Flattening Operators
    • switchMap
    • concatMap
    • mergeMap
    • exhaustMap
• Scroll API
  • Scroll and ScrollEnd Events
  • Wheel Event
  • Size Properties
• The Code
  • ensureScrolled
  • fetchData
• Example
  • Vertical Scrolling
  • Horizontal Scrolling
  • RTL Horizontal Scrolling
• UX and Accessibility Consideration
• Other Pagination Strategies
• Bonus: What About Other Flattening Operators?
  • Using mergeMap
  • Using switchMap
  • Using concatMap
  • Using exaustMap
• Next Step
• Summary
• References

Problem

Infinite scrolling is often used for a few key reasons:

• Data Fetching: Loading large data sets all at once can lead to latency issues or even browser crashes.

• Mobile Usability: On mobile platforms, scrolling is more intuitive than navigating through multiple pages.

• Resource Optimization: Incremental data loading is generally more resource-efficient, reducing the load on both the server and the client, which can lead to faster load times and a better user experience.

Solution

You’re going to build a minimal yet efficient function using RxJS. It will include:

• Support for vertical scrolling
• Horizontal scroll support for both LTR and RTL
• A threshold for determining when to fetch more data
• Loading state

The writing assumes you have a basic understanding of RxJS. No worries though, I’ll explain any special code or RxJS features along the way. So get ready, because you’re about to dive into some RxJS operators! 😄

For those already comfortable with RxJS, you can skip the next section or jump to The Code

Well, Let’s start!

Getting Started

The only thing you need to get started is RxJS. Install it with this command:

npm i rxjs

Note: I’m using TypeScript primarily for clarity in showing what options are available through types. You’re free to omit them, but if you do want to use types, I’d suggest opting for a framework that has built-in TypeScript support.

RxJS operators

RxJS operators are functions that manipulate and transform observable sequences. These operators can be used to filter, combine, project, or perform other operations on an observable sequence of events.

Common RxJS Operators

There are a lot of them, most used (by me 😆) are tap, map, filter, switchMap, and finalize. You might already know how to use those but lucky you, we’re going to learn about other useful operators!

Take a look at the following observable:

const source$ = from([1, 2, 3, 4, 5]);
source$.subscribe(event => console.log(event));

The result would be 1 2 3 4 5. -Each in a new line-

filter

To only log odd numbers

const source$ = from([1, 2, 3, 4, 5]);
source$.pipe(filter(event => event % 2)).subscribe(event => console.log(event));

Say there’s a possibility that the source$ might emit a null value. You can use a filter to stop it from passing through the rest of the sequence.

const source$ = from([1, 2, 3, null, 5]);
source$
  .pipe(filter(event => event !== null))
  .subscribe(event => console.log(event));

map

To change the sequence of events, you can use the map operator.

source$
  .pipe(map(event => (event > 3 ? `Large number` : "Good enough")))
  .subscribe(event => console.log(event));

What if I want to inspect an event without changing the source sequence

tap

source$
  .pipe(
    tap(event => {
      logger.log("log an event in the console");
      // you can perform any operation as well, however return statment are ignore in tap function
    })
  )
  .subscribe(event => console.log(event));

finalize

To monitor the end of an observable’s lifecycle, you can use the finalize operator. It gets triggered when the observable completes.

It is usually used to perform some cleanup operations, stop the loading animation, or debug the memory, for example, add a log statement to ensure that the observable is complete and doesn’t stuck in the memory 🥲.

debounceTime

Imagine you’re building a login form and upon the user typing its password you want to hit the backend server to ensure the password conforms to certain criteria.

condt source$ = fromEvent(passwordInput, 'input').pipe(
  map((event) => passwordInput.value),
  switchMap((password) => checkPasswordValidaity(password))
)
source$.subscribe(event => console.log(event));

This example might work fine with one key caveat; on every keystroke, a request is sent to the backend server, thanks to switchMap it’ll cancel previous requests so there might not be as much harm, however, using debounceTime you can ignore input events till the dueTime -argument- is pass.

const source$ = fromEvent(passwordInput, 'input').pipe(
  debounceTime(2000)
  map((event) => passwordInput.value),
  switchMap((password) => checkPasswordValidaity(password))
)
source$.subscribe(event => console.log(event));

Adding debounceTime essentially implies creating 2 seconds between each keystroke, so a user enters “hello” and then before 2 seconds pass enters “world” and only one request will be sent. In other words, each event has to have a 2 seconds distance from the last event.

startWith

An observable might not have value immediately and you need an event readily available for the new source$ subscribers.

const defaultTimezone = '+1'
condt source$ = fromEvent(timezoneInput, 'input').pipe(
  map((event) => timezoneInput.value),
  startWith(defaultTimezone)
)
source$.subscribe(event => console.log(event));

This sample will immediately log “+1” even if timezoneInput value is never entered

fromEvent

You could rewrite the previous example to be as follows

const timezoneInputController = new Subject<string>();
const timezoneInputValue$ = timezoneInputController.asObservable();
timezoneInput.addEventListener("input", () =>
  subject.next(timezoneInputController.value)
);

const source$ = timezoneInputValue$.pipe(
  map(event => event.target.value),
  startWith(defaultTimezone)
);
source$.subscribe(event => console.log(event));

Thanks to RxJS you can use fromEvent that will encapsulate that boilerplate, all you need to do is to say which event to listen to and from what element. Of course fromEvent returns an observable 🙂

takeUntil

I admit this one might be difficult to digest, it was for me. Taking the same previous example, Let’s say that you have a form, an input, and a submit button. When the user clicks on the submit button you want to stop listening to the timezoneInput element input event. Yes, takeUntil as it sounds, it lets the subscribers take events until the provided observable emits at least once.

const defaultTimezone = '+1'
condt source$ = fromEvent(timezoneInput, 'input').pipe(
  map((event) => timezoneInput.value),
  startWith(defaultTimezone)
)
// normally, this subscriber will keep logging the event even if the users clicked on the submit button
source$.subscribe(event => console.log(event));

// Now, once the submit button are clicked the subscriber subscription will be canceled
const formSubmission$ = fromEvent(formEl, 'submit')
source$
  .pipe(takeUntil(formSubmission))
  .subscribe(event => console.log(event));

pipe

The pipe function in RxJS is a utility for composing operations on observables. Use it to chain multiple operators together in a readable manner, or to create reusable custom operators. This is crucial when the source sequence is complex to manage.

import { pipe } from "rxjs"; // add it to not to be confused with Observable.pipe

// Create a reusable custom operator using `pipe`
const doubleOddNumbers = pipe<number>(
  filter(n => n % 2 === 1),
  map(n => n * 2)
);

const source$ = from([1, 2, 3, 4, 5]);

source$.pipe(doubleOddNumbers).subscribe(x => console.log(x));
// result: 1, 6, 10

Flattening Operators

Sometimes, you need to fetch some data with every incoming event, say from the backend server. There are a few methods to do this.

switchMap

Like a regular map, the switchMap operator uses a project function that returns an observable -its first argument-, known as the inner observable. When an event occurs, switchMap subscribes to this inner observable, creating a subscription that lasts until the inner observable completes. If a new event arrives before the previous inner observable completes, switchMap cancels the existing subscription and starts a new one. In other words, it switches to a new subscription.

const source$ = from([1, 2, 3, 4, 5]);

function fetchData(id: number) {
  return from(fetch(`https://jsonplaceholder.typicode.com/todos/{id}`));
}

source$
  .pipe(switchMap(event => fetchData(event)))
  .subscribe(event => console.log(event));

In this sample, only the todo with id 5 will be logged because switchMap works by switching the priority to the recent event as explained above. from([...]) will emit the events after each other immediately thereby switchMap will switch (subscribe) to the next event inner observable as soon as it arrives without regard to the previous inner observable subscription. The switch operation essentially means unsubscribing from the previous inner observable and subscribing to the new one.

concatMap

It blocks new events from going through the source sequence unless the inner observable completes. It is particularly useful for database writing operations or animating/moving an element where it’s important to complete one action before starting another.

source$
  .pipe(concatMap(event => fetchData(event)))
  .subscribe(event => console.log(event));

This sample will log all todos in order. Essentially what happens is concatMap blocks the source sequance till the inner observable at hand completes.

mergeMap

It doesn’t cancel the previous subscription nor blocks the source sequence. mergeMap will subscribe to the inner observable without regard to its completion, so if an event comes through and the previous inner observable hasn’t been completed yet that’s fine, mergeMap will subscribe to the inner observable anyway.

source$
  .pipe(mergeMap(event => fetchData(event)))
  .subscribe(event => console.log(event));

This sample will log all todos but in uncertain order, for instance, the second request might resolve before the first one and mergeMap doesn’t care about the order, If that is important then use concatMap.

exhaustMap

The final one and the most important in this writing is exhaustMap: it is like switchMap but with one key difference; it ignores the recent events in favor of the current inner observable completion in contrary to switchMap which cancels the previous inner observable subscription in favor of a new one.

source$
  .pipe(exhaustMap(event => fetchData(event)))
  .subscribe(event => console.log(event));

This sample will only log the first todo as the first todo request hasn’t been completed yet other events came through therefore they’ve been ignored.

To summarise

1. switchMap will unsubscribe from the existing subscription (if the previous inner observable hasn’t been completed) in favor of a new one when a new event arrives.
2. concatMap will block the source sequence so the inner observable at hand must be complete before allowing other events to flow.
3. mergeMap doesn’t care about the status of the inner observable so it’ll subscribe to the inner observable as events come through.
4. exhaustMap will ignore any event till the current inner observable is complete.

Okay, that is a lot, isn’t it? I understand that if you’re new to RxJS you might not be able to digest all this info, Your best bet is to practice and that’s what you’re trying to do here.

Wow, I really did it, and you did too 😎

Time to talk about some of the Scroll API(s)

Scroll API

You already know the Scroll Bar, it’s at the right end of the page 🥸, no really, when the user scroll in any direction the browser emits a few events, like scroll, scrollend, and wheel.

You are going to learn enough that to tackle the problem at hand.

Let’s start with scroll and scrollend:

Scroll and ScrollEnd Events

The scroll event fires while an element is being scrolled and scrollend fires when scrolling has completed.

element.addEventListener("scroll", () => {
  console.log(`I'm being scrolled`);
});

element.addEventListener("scrollend", () => {
  console.log(`User stopped scrolling`);
});

Keep in mind that this only works if the element—the one that has the event listener (handler)—is scrollable, not its parent or any ancestor or descendant elements.

Wheel Event

The wheel event fires while an element or any of its children is being scrolled using the mouse/trackpad wheel which means trying to scroll down/up using the keyboard won’t trigger it.

Size Properties

For the task at hand, the scroll event will be the primary focus. However, I’ve also outlined some additional events and properties to give you a well-rounded understanding. Now, let’s look at the key size properties you’ll need to know:

• element.clientWidth: The inner width of the element, excluding borders and scrollbar.
• element.scrollWidth: The width of the content, including content not visible on the screen. If the element is not horizontally scrollable then it’d be the same as clientWidth.
• element.clientHeight: The inner height of the element, excluding borders and scrollbar.
• element.scrollHeight: The height of the content, including content not visible on the screen. If the element is not vertically scrollable then it’d be the same as clientHeight.
• element.scrollTop: The number of pixels that the content of an element is scrolled vertically.

Note: When I say “the content,” I mean the entirety of what’s contained within the HTML element.

Let’s take the following example, Calculate the remaining pixels from the user’s current scroll position to the end of the scrollable element.

function calculateDistanceFromBottom(element: HTMLElement) {
  const scrollPosition = element.scrollTop;
  const clientHeight = element.clientHeight;
  const totalHeight = element.scrollHeight;
  return totalHeight - (scrollPosition + clientHeight);
}

Take a look at the below image.

Presuming the totalHeight is 500px, clientHeight 300px, and the scrollPosition is 100px, deducting the sum of scrollPosition and clientHeight from totalHeight would result in 100px which is the remaining distance to reach the bottom of the element. A similar formula when calculating the remaining distance to the end horizontally

function calculateRemainingDistanceOnXAxis(element: HTMLElement): number {
  const scrollPosition = Math.abs(element.scrollLeft);
  const clientWidth = element.clientWidth;
  const totalWidth = element.scrollWidth;
  return totalWidth - (scrollPosition + clientWidth);
}

Presuming the totalWidth is 750px, clientWidth 500px and the scrollPosition is 150px, deducting the sum of scrollPosition and clientWidth from totalWidth would result in 100px which is the remaining distance to reach the end of the XAxis. You might have noticed the Math.abs being used and that due to RTL direction where the user has to go in the reverse direction which would make the scrollPosition value to be negative so using Math.abs to unify it in both directions.

Side tip: Using the information you have about the element’s sizes, you can also make a function to check if the element can be scrolled or not.

type InfinityScrollDirection = "horizontal" | "vertical";

function isScrollable(
  element: HTMLElement,
  direction: InfinityScrollDirection = "vertical"
) {
  if (direction === "horizontal") {
    return element.scrollWidth > element.clientWidth;
  } else {
    return element.scrollHeight > element.clientHeight;
  }
}

Simply put, if the element scroll’s size is the same as its client’s size then it isn’t scrollable.

The Code

I know you’ve been looking around to find this section, finally, we’ll put all the learnings into action, let’s start by creating a function named infinityScroll that accepts options argument

export interface InfinityScrollOptions<T> {
  /**
   * The element that is scrollable.
   */
  element: HTMLElement;
  /**
   * A BehaviorSubject that emits true when loading and false when not loading.
   */
  loading: BehaviorSubject<boolean>;
  /**
   * Indicates how far from the end of the scrollable element the user must be
   * before the loadFn is called.
   */
  threshold: number;
  /**
   * The initial page index to start loading from.
   */
  initialPageIndex: number;
  /**
   * The direction of the scrollable element.
   */
  scrollDirection?: InfinityScrollDirection;
  /**
   * The function that is called when the user scrolls to the end of the
   * scrollable element with respect to the threshold.
   */
  loadFn: (result: InfinityScrollResult) => ObservableInput<T>;
}

function infinityScroll<T extends any[]>(options: InfinityScrollOptions<T>) {
  // Logic
}

As promised, you now can customize the infinite scroll function to your liking. Next, you’ll learn how to attach an event listener to the specific element that contains your infinitely scrollable list of items.

function infinityScroll<T extends any[]>(options: InfinityScrollOptions<T>) {
  return fromEvent(options.element, "scroll").pipe(
    startWith(null),
    ensureScrolled,
    fetchData
  );
}

• fromEvent listens to scroll event of the scrollable element.
• startsWith starts the source sequence to fetch the first batch of data.
• ensureScrolled is a chainable operator that confirms the scroll position surpasses the predefined threshold before proceeding.
• fetchData is another chainable operator that fetches data based on the pageIndex, more on that later.

ensureScrolled

const ensureScrolled = pipe(
  filter(() => !options.loading.value), // ignore scroll event if already loading
  debounceTime(100), // debounce scroll event to prevent lagginess on heavy scroll pages
  filter(() => {
    const remainingDistance = calculateRemainingDistance(
      options.element,
      options.scrollDirection
    );
    return remainingDistance <= options.threshold;
  })
);

function calculateRemainingDistance(
  element: HTMLElement,
  direction: InfinityScrollDirection = "vertical"
) {
  if (direction === "horizontal") {
    return calculateRemainingDistanceOnXAxis(element);
  } else {
    return calculateRemainingDistanceToBottom(element);
  }
}

• filter only passes the scroll events if the element is scrollable, otherwise, it might lead to unexpected behavior.
• debounceTime will skip any event, in our case scroll events from flowing the sequence
• filter is checking if the remainingDistance either to the bottom (in case of vertical scrolling) or to the end of XAxis in case of horizontal scrolling is less than the threshold. Presuming threshold is 100px then when the scroll position is within 100 pixels of reaching the end (either vertically or horizontally, depending on the configuration), loadMore.next() will be invoked, signaling that more content should be loaded.

fetchData

const fetchData = pipe(
  exhaustMap((_, index) => {
    options.loading.next(true);
    return options.loadFn({
      pageIndex: options.initialPageIndex + index,
    });
  }),
  tap(() => options.loading.next(false)),
  // stop loading if error or explicitly completed (no more data)
  finalize(() => options.loading.next(false))
);

• exhaustMap ignores any event till the loadFn completes. If exhaustMap project function (its first argument) has been called, that implies the previous (if any) observable is completed and is ready to accept new events -load more data-.
• tap is signaling data loading is finished.
• finalize does the same as tap in our case, however, tap won’t be called if loadFn -request to the backend server- had responded with an error, and in case of an error, the source observable completes hence finalize. In other words, if the source sequence errored or the user explicitly completed the source then stop the loading.

Notice how exhaustMap indicates the loading state. You might question why not place the loading signal in a tap operator right before exhaustMap. Doing so would cause the loading observable to emit true whenever loadMore triggers. But this doesn’t necessarily mean it’s time to load more data -the previous inner observable from loadFn hasn’t finished yet-. To avoid this, exhaustMap is used to confirm that it’s ready to load more data.

The real piece of code; incrementing the page index to fetch the next patch of data

exhaustMap((_, index) => {
  return options.loadFn({
    pageIndex: options.initialPageIndex + index,
    // ...code
  });
});

The exhaustMap project function has two arguments

1. The event from the source sequence.
2. The index corresponds to the most recent event (this number signifies the position of the latest event).

In this specific case, you’ll be focused on the event position or index. Check out the following example for a clearer understanding of how it operates.

• At first time the source loadMore emits the index will be zero.
• 3 batches of data are loaded, so the next index will be 4.
• Assuming initialPageIndex is 1 and the data is to be loaded for the first time then the pageIndex is 1
• Assuming initialPageIndex is 1 and the data is to be loaded for the fifth time then the pageIndex is 6
• Assuming initialPageIndex is 4 and the data is to be loaded for the first time then the pageIndex is 4

The last case might be off; usualy you might have initialPageIndex 0, but let’s say you’re scrolling the Twitter feed, and for some reason, the browser reloaded, so instead of loading data from the beginning, you decided to store the pageIndex in some state (URL query string) so in such cases only the data from the last pageIndex will be there so the experience continues as if nothing happened. Prior data needs to be there as well either by loading it till the pageIndex or via implementing an opposite scroll direction data loading 🥲

Example

Vertical Scrolling

Horizontal Scrolling

RTL Horizontal Scrolling

UX and Accessibility Consideration

Infinite scrolling isn’t a magic fix. I know some folks who strongly advise against using it. Here’s why:

1. Bad for Keyboard Users: If you’re using a keyboard to get around a website, infinite scrolling can mess that up and get you stuck.Especially if the infinity scrolling is the main way of navigating the website

2. Hard to Pick Up Where You Left Off: Without page numbers, it’s tough to go back to where you were. This makes it hard for users and a headache for developers to implement.

3. Unreachable Content: Makes certain content like footers hard to reach.

4. Confusing Screen Readers: If someone’s using a screen reader, the constant loading can make the page structure confusing.

5. Too Much, Too Fast: For some people, like those who get easily distracted, the never-ending flow of content can be overwhelming. This one’s just my take, but it’s something to think about.

When building an infinite scroll you’ve to consider important factors such as:

1. Placing content correctly and making them accessible like footer, and contact information.
2. Allowing users to return to their previous spot.
3. Offering the ability to jump ahead.
4. Ensuring the experience is navigable for users who rely solely on keyboards.

I recognize that these tasks present significant developmental challenges. However, as the saying goes, quality comes at a cost.

This isn’t to say that infinite scrolling is bad; instead, the emphasis is on applying it with caution.

Other Pagination Strategies

• Traditional Pagination: This approach uses a combination of numbered pagination and ‘Previous’/‘Next’ buttons to offer both specific and sequential page access.

• “Load More” Button: Includes a button at the end of the visible content; clicking it appends additional items to the list.

• Content Segmentation: Utilizes tabs or filters to categorize content, enabling quick navigation to topic-specific data—e.g., segmenting tweets into categories like Science, Tech, Angular, 2021, etc.

Bonus: What About Other Flattening Operators?

What about using mergeMap, switchMap, or concatMap? You might have thought about that already!

Given the following scenario: A user scrolled down to the end of the page, but the request to load more data is still pending. The user kept scrolling down to the end of the page but the data was not yet resolved. What do you think would happen?

Note: The following recordings use a slow 3g network speed

Using mergeMap

With each scroll event, mergeMap subscribes to the inner observable without regard to the previous subscription, essentially leading to a new request -loadMode- with each verified scroll event -below the threshold-

Edit in CodePen

Using switchMap

With each scroll event, switchMap will cancel/unsubscribe from the previous subscription and subscribe to the inner observable again, essentially leading to a new request but the previous unresolved one will be canceled so only one request will be pending at a time. That might be okay, however, the event position index will increment each time switchMap subscripes to the inner observable which leads to incorrect data being loaded.

Edit in CodePen

Using concatMap

With each scroll event, concatMap will subscribe to the inner observable, blocking the source sequence till the current subscription completes -loadMore request resolves-, essentially leading to a new request with every verified scroll but holding them onto till it can process a new event. The event position index will increment each time concatMap subscribes to the inner observable which leads to requesting more data than needed. See the recording below and take a good look at what happens in the Network Tap when the user stops scrolling.

Edit in CodePen

Using exaustMap

It is the winner in this scenario because it effectively manages pending requests. When a scroll event triggers a new request, exhaustMap will ignore any subsequent scroll events until the current request (inner observable) is complete. This ensures that only one request is pending at a time, and it prevents the index from incrementing incorrectly.

That being said, a simple workaround would be to explicitly ignore any scroll even while the data is loading.

const fetchData = pipe(
  filter(() => options.loading.value === false),
  // mergeMap, switchMap and concatMap should work now.
  exaustMap((_, index) => {
    // ...
  })
  // ...
);