Since the launch of the Incentivized Testnet marked the coming of the Shelley era last year, the Cardano platform has entered a fast-moving period of development. The Ouroboros Classic consensus protocol has supported Cardano Byron and the ada cryptocurrency for the past 30 months, and we’ll soon be switching to Ouroboros Praos. This is the version of our proof-of-stake (PoS) protocol that will initially power Shelley as Cardano decentralizes. It builds in the staking process with monetary rewards for ada holders and stake pool owners.

We upgraded Cardano on February 20 with a hard fork that switched the mainnet from the original consensus protocol, Ouroboros Classic, to an updated version, Ouroboros BFT. This BFT hard fork began a transition period under Ouroboros BFT, a slimmed-down version of the protocol designed to help us make the switch to Praos, while still preventing any malicious behaviour. Many probably didn’t even notice. For Daedalus wallet users, it meant a standard software update. Exchanges had to upgrade manually, but they had several weeks to do this and we were on hand to help.

The next event was the ‘Byron reboot’ on March 30. This released totally new code for many of the Cardano components, including a new node to support delegation and decentralization, and future Shelley features. A big advantage of the new code base is that it has been redesigned to be modular, so many components can be changed without affecting the others.

In turn, the BFT will act as the jumping off point for the Shelley hard fork, which will happen once we’re happy with the Haskell testnet. This second hard fork will be a similar process to the February one for exchanges, ada holders and wallet users, and, hopefully, just as much of a non-event.

However, while everything looks smooth on the surface, there is a lot of hidden activity going on. Like a duck serenely swimming across a pond – while its feet are furiously paddling below the calm waters – our blockchain engineers are hard at it.

So, we sat down two of the leading engineers on the Cardano project, Duncan Coutts and Edsko de Vries, to find out how they’ve done it. Duncan has been Cardano’s architectural lead for the past three years, and between them, Duncan and Edsko have spent 35 years using Haskell, the programming language being used to develop Cardano.

Duncan, how did you do it?

As described in the Cardano roadmap, IOHK’s blockchain engineers believe in smooth code updates. Instead of trying to do the jump from Ouroboros Classic to Praos in a single update – which would be an incredibly complex task – it’s been a two-stage approach using Ouroboros BFT as an intermediary (Figure 1). The BFT code is compatible with both the Byron-era federated nodes and the Shelley-style nodes released in the Byron reboot. It's like a relay race: one runner (in our case, running one protocol) enters the handover box where the other runner is waiting; they synchronise their speeds (so they're perfectly compatible with each other) and then hand over the baton (operating the mainnet), and then the new runner with the baton continues from the handover box for the next lap.

The Daedalus Flight process has helped us quickly develop and test a new wallet and, once everyone is running that on the mainnet, and once we finish swapping over the core nodes, the old code is redundant. We are in that transition phase right now, with a new mainnet Daedalus wallet released on April 24.

Our aim is to have a ‘graceful entry into Shelley’, as IOHK chief Charles Hoskinson describes in his whiteboard video about the hard fork. A vital tool in making this move has been creating a hard fork combinator.

That sounds like farm machinery. What is it?

A combinator is just a technical term for something that combines other things. For example, addition is a combinator on numbers. A hard fork combinator combines two protocols into a single protocol. We call this a sequential combination of the two protocols because it runs the first protocol for a while and at some point it switches over to the second. In our case, this is two versions of Ouroboros as we move from BTF to Praos.

The clever part of all this has been the use of discrete modules that do their job, while knowing as little as possible about each other and the blockchain. Simplicity is the key here and this process of taking out the details we call ‘abstraction’. Most of the consensus modules don’t even have to know that they’re dealing with a cryptocurrency and could be putting pretty much anything on a blockchain. For example, we’ve done seminars using the example of a Pokémon ledger on a Ouroboros blockchain. The only thing that’s different is the ledger rules; the consensus is all the same. You just set it up – ‘instantiate’ it in the programming jargon – with the rules for playing Pokémon rather than for UTXO-style accounting.[For readers with a technical interest, watch out for Edsko delving further into the ‘abstraction’ process and combinators in a future blog post.]

You make it sound simple

In fact, it’s tricky because Cardano is running the ada cryptocurrency, and a pile of other things, at the same time. Think of it as changing all the wheels on a car while you’re driving along and towing a caravan. So we have to be sure we can do this in a totally reliable way.

We could have tackled this as a one-off task, but it made sense to do it in a generic way using a protocol combinator. We chose this route because we get a better result and the testing that is vital to ensuring the code works is made far easier. On top of that, there will be more hard forks to come, which made the choice even clearer. For example, as we near the culmination of Cardano’s development and move through the Goguen, Basho, and Voltaire eras, there will be at least one hard fork at each stage.

So how did you cope with the tricky bits?

Well, first off, we had to do it without research to turn to. The researchers describe a single protocol as a free-standing, perfect thing. But that’s not where we are. We are trying to run Praos after having started with a chain that was using something else. What Edsko’s working on, going from one protocol to another in a generic way, is just not covered in the research. And it’s hard, it’s complicated. All the details need a lot of thinking, a lot of scratching your head. But switching between Cardano code bases is not the sort of thing the academics can expect to get published. It doesn’t have a novel aspect and is seen as just an implementation issue.

Edsko, can you give us an example?

As Duncan says, for the researchers, these implementation issues are trivial but dealing with them is 99% of what we do. Take the problem of time for a blockchain. It’s what I’ve been banging my head against for a couple of weeks. Time is divided into slots where the chain can contain at most one block per slot. We often need to convert between slot numbers and time in the real world, for example when a node needs to know ‘Is it my turn?’ to generate the next block. This is fundamental to Cardano, but the length of a slot changes after the hard fork. For Byron, a slot is 20 seconds; for Shelley, it will be two seconds, or perhaps one. To really complicate things, the exact point of time when the hard fork is made is decided on the chain itself. Yet, I need to know when the changeover point is. It’s a quandary: to do slot conversions I need to know the state of the blockchain, but to know the state I need to know the slot conversions!

This is real chicken-and-egg territory with many complex things to disentangle. We have to be very precise with how we do things. It might be trivial in theory, but it’s very difficult to disentangle things and make sure it’s not a circular problem.

We can’t afford for it to be wrong, so how do you know you’re right?

Duncan: That’s an excellent question. My reply is that you come to the answer on two levels. The first is intellectual: you analyse the problem, you do the maths, you talk to colleagues and wrestle with it until you can see how it all fits together. Second, we do all our QuickCheck testing to give us the confidence that this does what we think it does. We do extensive testing that really takes us into the unusual cases that you might never think of, including this changeover. We can do 100,000 tests every time we change a line of code. [Lars Brünjes has written about how John Hughes, one of the creators of Haskell, has helped IOHK develop its testing strategies.]

Edsko: Yes, I agree with those two points. In terms of the combinator, I resolve these things by thinking about the guarantees that the code I write needs to provide, and which guarantees that it, in turn, needs from the ledger. I sketch a mathematical proof that this ‘if-then’ reasoning is indeed justified, and then turn to the formal method teams. The formal methods team are the people who set up the mathematical rules that describe the blockchain, and they can then tweak the rules in such a way that they provide the required guarantees.

In terms of Duncan’s second point, I know the time issue I mentioned above is correct by thinking hard mathematically, and by testing. Timing decisions are easy when we have the full blockchain, but are hard when we have to make predictions about the future. Fortunately, the way we set things up means I can easily create testing blockchains. So, I can create a full blockchain, then slice this chain in half. I take the first half and consider that to be in the present; and set the other half in the future. Then I can use the ‘present’ (first half) to make predictions about the ‘future’ (the second half) and verify them against the whole thing (on which the calculations are easy). If they match, then I know everything is OK.

When did you start on this?

Right after Duncan came up with the genius idea of the OBFT. So I’ve been thinking about the combinator on and off for about 18 months. It was a design goal from the very beginning of our modular rewrite of Ouroboros starting in October 2018, with my first commit to the GitHub repository. We had a prototype demonstration with OBFT and Praos soon after, in December 2018.

And how many people have been involved?

Duncan: Many people have been working on the consensus code, but whenever we have anything really hard, like this combinator, we give it to Edsko: he’s our software engineer extraordinaire! This is Haskell programming as free climbing, rather than climbing with the ropes of formal methods for support.

Any final thoughts?

Duncan: The code that was running is right now being phased out and pretty soon the code that was running Cardano a month ago will no longer exist. Once everyone is running the new code on the mainnet and once we finish swapping over the core nodes, the old code is redundant. We are in that transition phase right now and no one is shouting that the sky is falling. No one has noticed it.

Edsko: That is quite an achievement in its own right. The idea of OBFT was crucial in making the transition, but it’s not relevant any more once we make that transition to Shelley. This has been a way for us genuinely to ditch legacy code, which is often very difficult to do, as the banks know to their cost.

Duncan: And if it all works fine, you won’t notice anything.

Duncan and Edsko, thank you for your time. I think we’d better let you both get back to it.

The Daedalus team is opening up IOHK’s wallet testing program to ada owners. The aim is to seek the help of a broad range of people who can test – on a rolling basis – the latest interface features being readied for the next release version of Daedalus. This will be a fully-functioning version of the wallet, called Daedalus Flight, so you will be able to spend and receive ada as usual – and give the team valuable feedback to improve the experience with each release.

We put some questions to Daedalus Product Manager, Darko Mijić, to tease out the details for people who want to take part.

Darko, can you expand on the thinking behind the program?

I’ve been working at IOHK for almost four years. It’s been amazing to see the way the Incentivized Testnet has galvanized the Cardano community in the past few months. The community helped us develop the stake pools for Shelley, and, of course, to test and develop a version of the Daedalus wallet for the ITN. So, building on that strategy, we came up with the idea of Daedalus Flight. This is a pre-release version of the Daedalus wallet that users can test with real ada transactions on the mainnet. People can join the program by downloading and using the Flight wallet alongside their usual Daedalus – or Yoroi – wallet to find and report issues so we can fix them before the actual release. We’ve seen that people want to help – and they will gain early access to the new features of the next production Daedalus release for the mainnet.

It’s similar to beta testing, or the ‘canary’ approach Google uses for its Chrome browser. Adding this tactic to our release process creates another valuable source of testing data. We will be able to improve the quality assurance of our new software versions and move more quickly into full production by first trialing these changes in Daedalus Flight with a subset of participating users.

But is it for everyone?

This program is especially important because it will give us a bigger sample of users who are using Daedalus in various ways on a wide variety of software and hardware configurations. We want to bring confident ada owners on board to help us develop the wallet by testing features and suggesting their own ideas. People who volunteer for this must already be familiar with using the Daedalus wallet. People need to be skilled in using computers – copying and backing up files, moving things around between folders – and prepared to deal with any issues that arise. We’re not expecting IT experts, but you do need to be confident about computers and making ada transactions.

How will the process work?

We complete a Daedalus development sprint every two weeks and quality assurance is a continuous effort during the development of every feature. At the end of each sprint, we will now create a new Daedalus Flight release, and, starting with the first flight candidate, we do our tests internally. But testing in a lab for a mass-market desktop product is not a real test. Under the new flight process, we will release a series of flight candidates and work on testing, finding issues and fixing issues with members of the Cardano community who have joined the program to help us. Flight candidate releases will be delivered through the Daedalus newsfeed, after an initial download from a new Flight option on the official Daedalus website, which is the only source of the Flight wallet.

Once we reach our final candidate, and we are confident that the release ‘can fly’, we will be releasing it to production for all Daedalus users.

This process will repeat every two weeks. There will be instances when we don’t release the production version after completing a flight release. This will happen when we are building big, multi-sprint features that cannot be delivered partially.

When you download Daedalus Flight, you can compare balances and transaction histories and see that all your wallet data has moved across. If there’s a problem, you just report it and go back to using your usual wallet while we fix things and release another candidate.

Why is this on the mainnet? Won’t it put our ada at risk?

Your ada will not be at risk. The Flight wallet is a real wallet and can do everything your usual wallet can. We’re adding features to make things easier for users. We could have done this on a testnet but the real test is where the real wallets live, on the mainnet. ‘Real’ user testing only happens after a wallet update is released to the mainnet. This flight process is secure because it is a completely separate wallet installation. We import your wallets from your production version of Daedalus. The production version of Daedalus stays untouched and fully functional.

So, we keep our present wallets?

Yes, you keep all your present wallets. You can use them alongside each Daedalus Flight candidate.

How will we give feedback?

There is an option to open a support request directly from the Daedalus interface. These requests will be handled separately by the support desk, and you can attach a wallet log so we can investigate your problem. If you just want to suggest an idea, you can do that too by clicking ‘Support request’ in the help menu.

Can you give us a sneak preview of the features?

Well, first of all, there will be Yoroi support in the new Flight wallet. Yoroi users will be able to use the same wallet both in Flight and Yoroi. Transactions will be in sync. Then, there will be a transaction filtering function in your transaction history so you can filter transactions by type, date, and ada amount. There will also be a warning in the transaction confirmation window making sure users understand that Daedalus Flight transactions are real ada transactions on the mainnet. Other neat touches will include parallel wallet restoration and a resync wallet function. Later on, we will be adding cool features like hardware wallet support.

How long will the Daedalus Flight program last?

The program started with the Byron reboot on March 31, with the first flight release and its candidate #1 release. If we discover and fix issues, we will issue a second candidate. When we get to a fully stable candidate, we will release it to all Daedalus users. April 6 is the planned date for the first production-ready Daedalus, but this depends on the results and user data from Daedalus Flight.

The next release and its series of candidates can be expected for April 14, and then every two weeks. Because this is a new product, we will try this process out for a month or two. If successful, we plan to make this permanent practice in the long term. It is important to note that the special Daedalus version for the Incentivized Testnet will still exist – that is a completely separate product.

Any final message?

People should be clear that the Daedalus Flight wallet is used with your mainnet ada. It’s a real wallet with all the core functions of Daedalus. It’s just the new features, which are not to do with transactions, that we’re testing. Going back to the Chrome comparison, think of it as buying something from Amazon using a beta test release of a web browser; you are doing real things with test software.

We hope you’ll be as excited as we are by this innovation – and can help our Daedalus development really fly! And remember, for the latest information and news, visit the official Daedalus website.

The Incentivized Testnet (ITN) has been running since mid-December, and the results have produced some fascinating insights into stake pools and a steep learning curve for the blockchain engineers at IOHK, as well as the companies and individuals setting up stake pools, and ada owners. The strategy of using a fast development team writing in the Rust language to act as pathfinders for the heavyweight Haskell developers looks to be paying off. IOHK now has an enormous amount of information about the use — and misuse — of the protocol to take to the next stage: the Haskell testnet. Alongside that, the Cardano community has shown what it is capable of — supporting, experimenting, and providing solid feedback throughout.

Before the ITN went live on December 13, 158 stake pools had registered with the Cardano Foundation and were setting themselves up. Yet, within three days, the number of pools had shot up to 325. By the end of January, the total was well past the 600 mark. There had been some scepticism when IOHK chief Charles Hoskinson talked of 1,000 stake pools last year, but we’re well on the way to that total.

As Scott Darby’s world of stake pools animation shows, the nodes are spread from Brazil to South Africa and Australia; from Japan and China to San Francisco via Europe — and, with nodes in Bodø and Fauske in Norway, we’re even in the Arctic Circle.

Many from the crypto press remarked on the fast results: CryptoSlate pointed out that the testnet had 10 times more pools than Eos or Tron within a week. NewsBTC summed it up with the headline: ‘Cardano testnet success shows how decentralization should work.’ The headlines, of course, don’t tell the whole story, and there were plenty of bumps in the road. But it’s going well, and we’ve received positive feedback about the improvements made to date (with more to come). That said, the network’s success isn’t just about what we do: it’s about the work of stake pool operators. Here, we take a look at the stake pools bringing this decentralized network to life, and explore the business of running a stake pool.

Stake pool tools

Thanks to the efforts of the Cardano community, anyone can delve into the workings of the system and explore what is happening. AdaPools, run by the Cardanians group alongside its pool, has a dashboard based on data from IOHK’s GitHub registry with tools such as a mapping of decentralization, notifications of saturation, and a test for whether a pool is forked and off the main blockchain. Cardano Pool Tool run by StakeLove is based around a table that can rank staking providers by 16 measures, from pool name to ada staked to return on investment.

The information shown in these tools comes from the blockchain data. Beyond that, the decentralized nature of the blockchain means we cannot know the identity of stake pool operators until they reveal themselves through their pool’s website or social media. So, the biggest pool early on, with 737m ada staked — twice as much as any of the IOHK pools — had ZZZ as its ticker but, initially, its name was simply its identity extracted from the blockchain. ZZZ soon split itself into several pools and revealed its name as TripleZ, based in Japan.

People staking their ada may prefer to know more about who’s running their pool, but they might not. This is one of the things that IOHK — and ada holders because Cardano is going to become their network once it’s decentralized — will get a feel for from the ITN. There are various forums where all this is being discussed, such as on Telegram and the Cardano forums. It’s been fascinating to see the debate inspired by the testnet, much of which reflects debates within IOHK about how best to build a community-driven, decentralized network and the role that incentives should play. The balance between community contribution and personal profit motive has been discussed at length. So, too, has how much the community should police itself. This is new territory, and, through the community, we’re able to test our assumptions about how blockchain social dynamics play out, and to what extent the protocol should be responsible for preventing adversarial behavior.

Community and operator reactions

Alongside the technical learnings, gathering community feedback and input has been an essential part of the Incentivized Testnet, to help us on the journey to deploying Shelley on the mainnet. Even before stake pools had set up their nodes to join the testnet, users began to provide feedback and have their say. Max, a Cardano ambassador, ran three ‘What the pool?’ interviews in the run-up to the testnet launch on his Gerolamo blog, and has since added a fourth. The Cardano Effect also interviewed four operators. Another website, Stake Pool Showcase, asked five standard questions and encourages pool owners to sign up and make their case:

• Who operates the pool?
• What is your history with the Cardano project?
• What is the setup of your pool?
• What are your plans for the future of your pool?
• Why should people delegate to your pool?
  

The answers demonstrate a range of operators. In terms of size of stake, the nine listed by February ranged from 1 ada to 50 million ada. Eight of the pools were run by one or two people who worked in computing and most dated their involvement with Cardano back to 2017. Three did not give their names, one stating: ‘The pool is run in an anonymous fashion, in order to make it impossible to influence me. This is part of the security, to make it much harder to attack the pool.’ They were in places such as France, Honolulu, London, Manchester, and Norway.

As well as giving information about their experience, most listed their hardware set-up and seemed to know what to expect from a testnet: ‘Of course, within the testnet the pool can only run as stable as the software stability allows, but I will do my best — and, moving forward, code stability will improve for sure.’ Another said: ‘We have been tinkering with the settings all the time and have achieved very good uptime in the last few epochs — after a lot of lost sleep.’

One operator was sensitive to the power expenditure of running cryptocurrencies: ‘Overall, I am very pleased I still only draw 35-45 watts in day-to-day operations, so it's eco-friendly.’ A second was running a backup server on a Rock Pi single board computer, which uses as little as 10W, as demonstrated at last year’s IOHK Summit. Looking beyond the testnet, another pool operator raised the challenge of governance in the Voltaire era of development and saw smart contracts as the way forward: ‘We have Marlowe for a financial DSL [domain-specific language], why not a legal DSL to help with governance issues?’

The Cardano Shelley Testnet & StakePool Best Practice Workgroup on Telegram received several mentions as the place for operators to go for tips.

All in all, as Kyle Solomon at AdaFrog told this blog: ‘Being a stake pool operator has been both a highly challenging and amazingly fulfilling journey. The most important takeaways I’ve learned as a pool operator are: first, that the protocol is very close to a production quality that achieves IOHK’s original goals for Cardano; and second that the Cardano community is utterly and hands-down amazing. Even though we compete amongst each other, every pool operator is eager to help one another.’

The next post in this three-part series will delve deeper into the experiences of the stake pools and what’s been learnt.

As with everything IOHK does, we cannot give advice on how you use your ada and we’re not recommending any of these pools. As always, though, please keep getting in touch and let us know your thoughts.