Jesus Rodriguez is the CEO of IntoTheBlock, a market intelligence platform for crypto assets.

In a recent article, I discussed the relevance of the machine learning techniques powering the famous OpenAI’s GPT-3 could have for the crypto market. GPT-3 – which can answer questions, perform language analysis and generate text – might be the most famous achievements in recent years of the deep learning space.

But, by no means, is it the most applicable to the crypto space. In this article, I would like to discuss some novel areas of deep learning that can have a near immediate impact in the quant models applied to crypto. He has held leadership roles at major technology companies and hedge funds. He is an active investor, speaker, author and guest lecturer at Columbia University in New York.

Models such as GPT-3 or Google’s BERT are the result of a massive breakthrough in deep learning known as language pretrained and transformer models. These techniques, arguably, represent the biggest milestone in the last few years of the deep learning industry and their impact hasn’t gone unnoticed in capital markets.

In the last year, there have been active research efforts in quantitative finance exploring how transformer models can be applied to different asset classes. However, the results of these efforts remain sketchy showing that transformers are far from ready to operate in financial datasets and they remain mostly applicable to textual data. But there is no reason to feel bad. While adapting transformers to financial scenarios remains relatively challenging, other new areas of the deep learning space are showing promise when applied in quant models on various asset classes including crypto.

From many angles

, crypto seems to be like the perfect asset class for deep learning-based quant models. That’s because of the the digital DNA and the transparency of crypto assets and that the rise of crypto has coincided with a renaissance of machine learning and the emergence of deep learning. After decades of struggle and a couple of so-called “artificial intelligence (AI) winters,” deep learning has finally become real and somewhat mainstream across different areas of the software industry. Quantitative finance has been one of the fastest adopters of new deep learning technologies and research. It is very common for some of the top quant funds in the market to experiment with the same types of ideas coming out of high tech AI research labs such as Facebook, Google or Microsoft.

Some of the most exciting developments in modern quant financing are not coming from flashy techniques like transformers, but from exciting machine learning breakthroughs that are more developed for quant scenarios. Many of those methods are perfectly applicable to crypto-asset quant techniques and are starting to make inroads in crypto quant models. Below, I’ve listed five emerging areas of deep learning that are particularly important to crypto quant scenarios. I tried to keep the explanations relatively simple and tailored to crypto scenarios

1) Graph neural networks

Blockchain datasets are a unique source of alpha for quant models in the crypto space. From a structural perspective, blockchain data is intrinsically hierarchical and is represented by a graph with nodes representing addresses connected by edges representing transactions. Imagine a scenario in which a quant model is trying to predict volatility in bitcoin in a given exchange based on the characteristics of addresses transferring funds into the exchange. That kind of model needs to operate efficiently over hierarchical data. But most machine learning techniques are designed to work with tabular datasets, not graphs. Graph neural networks (GNNs) are a new deep learning discipline that focuses on models that operate efficiently on graph data structures. GNNs are a relatively new area of deep learning being invented only in 2005. However, GNNs have seen a lot of adoptions from companies like Uber, Google, Microsoft, DeepMind and others. In our sample scenario, a GNN could use a graph as input representing the flows in and out of exchanges and infer relevant knowledge relevant to its impact on price. In the context of crypto assets, GNNs have the potential of enabling new quant methods based on blockchain datasets.

2) Generative models

One of the limitations of machine learning quant models is the lack of large historical datasets. Suppose that you are trying to build a predictive model for the price of chainlink (LINK) based on its historical trading behavior. While the concept seems appealing, it might prove to be challenging as LINK has a little over a year of historical trading data in exchanges like Coinbase. That small dataset will be insufficient for most deep neural networks to generalize any relevant knowledge.

Generative models are a type of deep learning method specialized in generating synthetic data that matches the distribution of a training dataset. In our scenario, imagine that we train a generative model in the distribution of the link orderbook in Coinbase in order to generate new orders that match the distribution of the real orderbook. Combining the real dataset and the synthetic one, we can build a large enough dataset to train a sophisticated deep learning model. The concept of generative model is not particularly new but has gotten a lot of traction in recent year with the emergence of popular techniques such as generative adversarial neural networks (GANs), which have become one of the most popular methods in areas such as image classification and have been used with relevant success with time series financial datasets.

3) Semi-supervised learning

Labeled datasets are scarce in the crypto space and that severely limits the type of machine learning (ML) quant models that can be built in real world scenarios. Imagine that we are trying to build an ML model that makes price predictions based on activity of over-the-counter (OTC) desks. To train that model, we would need a large labeled dataset with addresses belonging to OTC desks which is the type of dataset that only a few entities in the crypto market possesses. Semi-supervised learning is a deep learning technique that focuses on the creation of models that can learn with small labeled datasets and a large volume of unlabeled data. Semi-supervised learning is analogous to a teacher presenting a few concepts to a group of students and leaves the other concepts to homework and self-study.

In our sample scenario, imagine that we train a model with a small set of labeled trades from OTC desks and a large set of unlabeled ones. Our semi-supervised learning model will learn key features from the labeled dataset such as trade size or frequency and will use the unlabeled dataset to expand the training.

4) Representation learning

Feature extraction and selection are a key component of any quant machine learning model and is particularly relevant in problems that are not very well understood such as crypto asset predictions. Imagine that we are trying to build a predictive model for the price of bitcoin based on order book records. One of the most important aspects of our effort is to determine which attributes or features can act as predictors. Is it the mid-price, the volume or a hundred other factors? The traditional approach is to rely on subject matter experts to handcraft these features but that can become hard to scale and maintain over time. Representation learning is an area of dep learning focused on automating the learning of solid representations or features in order to build more effective models. Instead of relying on human feature modeling, representation learning tries to extrapolate features directly from unlabeled datasets. In our example, a representation learning method could analyze the order book and identify hundreds of thousands of potential features that can act as predictors for the Bitcoin prices. That level of scaling and automation is impossible to achieve in manual feature engineering.

5) Neural architecture search

The process of creating quant machine learning models remains highly subjective in many aspects. Let’s take the scenario of a model that is trying to forecast the price of Ethereum based on activity in a set of DeFi protocols. Given the nature of the problem, data scientists will have certain preferences about the type of model and architecture to use.

In our scenario, most of those ideas would be based on domain knowledge and subjective opinions about the way the activity in DeFi protocols can impact the price of Ethereum. Given that machine learning is based on building knowledge and knowledge is not a discrete unit, its almost impossible to debate the merits of one method versus another for a given problem.

Neural architecture search (NAS) is one area of deep learning that tries to automate the creation models using machine learning. Sort of using machine learning to create machine learning. Given a target problem and dataset, NAS methods will evaluate hundreds of possible neural network architectures and output the ones with the most promising results. In our sample scenario, a NAS method can process a dataset that incorporates trades in decentralized exchanges and produce a few models that can potentially predict the price of Ethereum based on those records.

More to come The methods described above represent some emerging and more developed areas of deep learning that are likely to have an impact in the crypto quant models in the short term. And those are by no stretch the only areas of deep learning crypto quant should pay attention to. Other deep learning disciplines such as reinforcement learning, self-supervised learning and even transformers are rapidly making inroads in the quant space. Research and experimentation about deep learning techniques applied to quant models is happening everywhere and crypto stands to be a great beneficiary of that wave of innovation.