Jekyll2018-10-19T12:50:39+00:00https://iohk.io/Input OutputCascading disruptionOuroboros Genesis presented at flagship conference2018-10-18T00:00:00+00:002018-10-18T00:00:00+00:00https://iohk.io/blog/ouroboros-genesis-presented-at-flagship-conference<p><img src="/images/blog/ccs.jpg" alt="CCS 2018 Toronto" class="fullwidth" /></p>
<p>A third major paper from the Ouroboros line of research was presented at a leading computer security and cryptography event yesterday, a recognition of the contribution the work makes to the field of computer science.</p>
<p>The paper, <a href="https://eprint.iacr.org/2018/378.pdf" title="Ouroboros Genesis, eprint.iacr.org">Ouroboros Genesis</a>, was presented by researcher Christian Badertscher at the 25th <a href="https://www.sigsac.org/ccs/CCS2018/" title="sigsac.org">ACM Conference</a> on Computer and Communications Security, held in Toronto this week. The conference is five days of presentations, workshops and tutorials for hundreds of delegates, who are information security researchers, industry professionals, and developers from around the world. The annual event, organised by the Special Interest Group on Security, Audit and Control (SIGSAC) of the Association for Computing Machinery (ACM), is a forum for delegates to come together for discussions and to explore cutting edge research.</p>
<p>This year CCS sponsors included the US government agency, the National Science Foundation, and major global technology companies such as Baidu, Cisco, Samsung, Google, Facebook. The hardware wallet maker, Ledger, was also present. CCS is the highest rated computer security and cryptography conference according to Google Scholar ratings, meaning that collectively, the papers selected to appear at the conference are more cited by academics than papers for any other conference.</p>
<p>IOHK’s paper appeared in one of the two sessions that were dedicated to blockchain, with a total of six papers on the subject overall. These included a <a href="https://arxiv.org/abs/1805.05288" title="The Gap Game, arxiv.org">paper</a> on what will happen with blockchains such as Bitcoin as rewards get smaller and the potential problems that stem from that. Scalability was in focus too, with a <a href="https://eprint.iacr.org/2018/460.pdf" title="RapidChain: Scaling Blockchain via Full Sharding, eprint.iacr.org">paper</a> on scaling blockchains through sharding and another on <a href="https://eprint.iacr.org/2018/320" title="General State Channel Networks, eprint.iacr.org">state channel</a> networks.</p>
<figure class="alignright">
<img src="/images/blog/genesis-presentation.jpg" alt="" width="350" height="" />
<figcaption>Ouroboros Genesis Presentation</figcaption>
</figure>
<p>Since Bitcoin demonstrated the disadvantages of using an energy-intensive proof-of-work protocol to run a public distributed ledger, many researchers have turned to explore proof of stake. The Ouroboros research is an attempt to systematically surmount the challenges that proof of stake poses and describe a secure, efficient and sustainable protocol for blockchains.</p>
<p>This effort has been led by Professor Aggelos Kiayias, IOHK Chief Scientist and Chair in Cybersecurity and Privacy at the University of Edinburgh’s School of Informatics. In the space of only a couple of years, the team have made significant advances. Ouroboros was the first peer reviewed, provably secure proof-of-stake protocol, and it is already running in the real world, underpinning Cardano, a top 10 global cryptocurrency.</p>
<p><a href="https://www.youtube.com/watch?v=LCeK_4o-NCc" title="Ouroboros Genesis: A Provably Secure Proof-of-Stake Blockchain Protocol, youtube.com">Ouroboros Genesis</a> is the third paper in the Ouroboros family of proof-of-stake protocols, and the third paper from this important line of IOHK research to be heard at a flagship international computer science conference. The first paper, Ouroboros, was presented at <a href="https://iohk.io/blog/proof-of-stake-protocol-ouroboros-at-crypto-17/" title="Proof-of-stake protocol, Ouroboros, at Crypto 17, iohk.io">Crypto 2017</a> in California, and the second, Ouroboros Praos, was at <a href="https://iohk.io/blog/ouroboros-praos-presented-at-leading-cryptography-conference/" title="Ouroboros Praos presented at leading cryptography conference, Eurocrypt, iohk.io">Eurocrypt 2018</a> in Tel Aviv. Further papers are to come from the research team, including on sharding, a means to provide scalability for Cardano.</p>
<p>Using Ouroboros Genesis, new users joining the blockchain will be able to do so securely based only on an authentic copy of the genesis block, without the need to rely on a checkpoint provided by a trusted party. Though common in proof-of-work protocols like Bitcoin, this feature was previously unavailable in existing proof-of-stake systems. This means that Ouroboros can now match the security guarantees of proof-of-work protocols like Bitcoin in a way that was previously widely believed to be impossible.</p>
<figure class="alignright">
<img src="/images/blog/ccs-christian.jpg" alt="" width="350" height="" />
<figcaption>Christian Badertscher (left) with Charles Hoskinson (right)</figcaption>
</figure>
<p>Aggelos said: “Ouroboros Genesis resolves an important open question in the PoS blockchain space, namely how it is possible to securely connect to the system without any information beyond the genesis block. This is a significant step forward that enables a higher degree of decentralization that seemed unattainable for PoS protocols before our work.</p>
<p>“Our security analysis is also in the “universal composition” setting that provides, for the first time in the PoS space, a modular way of building secure applications on top of the ledger.”</p>
<p>Christian said: “It is exciting to present Ouroboros Genesis at a top security conference and very rewarding to see how theoretical research can make a significant impact on practice. Avoiding the need of a trusted checkpoint, and still being secure in a setting with a variable level of participation, has been a challenging problem to solve in the PoS space.”</p>
<p>Published on May 3 this year, the paper’s full title is Ouroboros Genesis: Composable Proof-of-Stake Blockchains with Dynamic Availability. The research team was comprised of Christian Badertscher, Peter Gaži, Aggelos Kiayias, Alexander Russell, and Vassilis Zikas.</p>jane.wild@iohk.ioAn Open Letter to the Cardano Community from IOHK and Emurgo2018-10-12T00:00:00+00:002018-10-12T00:00:00+00:00https://iohk.io/blog/an-open-letter-to-the-cardano-community-from-iohk-and-emurgo<div class="videoframe">
<iframe width="100%" height="315" src="https://www.youtube.com/embed/brmocm3ttQo" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen=""></iframe>
</div>
<p>To the Cardano Community,</p>
<p>Cardano is an amazingly diverse and vibrant project that is rightfully being recognised throughout the world. Our community contains tens of thousands of engaged and passionate volunteers, advocates, contributors and fans in countries ranging from Argentina to Zimbabwe. This growth is due to our commitment to innovation, transparency, balance of power and embracing the scientific community. To IOHK and Emurgo, <a href="https://github.com/input-output-hk/cardano-sl" title="Cardano SL, github.com">Cardano</a> is so much more than a product we work on. Cardano is a mission to deliver a financial operating system to the three billion people who do not have one.</p>
<p>As with all movements, occasionally issues occur that require careful and rational discussion. When the Cardano movement began in 2015, instead of launching an all-powerful foundation that would raise funds, manage development, encourage adoption and address the concerns of the community, we diligently split the governance of Cardano into three legal entities: <a href="https://iohk.io" title="iohk.io">IOHK</a>, <a href="https://emurgo.io" title="emurgo.io">Emurgo</a> and the <a href="https://cardanofoundation.org" title="cardanofoundation.org">Cardano Foundation</a>. This separation of powers was to ensure that the failure of one legal entity, if any, could not jeopardise or destroy the <a href="https://cardanoroadmap.com" title="Cardano Roadmap">Cardano project</a>.</p>
<h3 id="iohk-and-emurgo">IOHK and Emurgo</h3>
<p>IOHK’s primary responsibility was and continues to be, developing the core collection of protocols that compose <a href="https://cardano.org" title="cardano.org">Cardano</a>, from academic inception to applying formal methods to verify correct implementation. This task is enormous in scope and has led to the creation of three research centers, many peer reviewed papers, engagement with half a dozen development firms and one of the most active cryptocurrency GitHub repositories.</p>
<p>As a company that accepts its critical role in this effort, IOHK has attempted to be as transparent and focused as possible. That acceptance is why we launched the Cardano Roadmap website, produce many videos on our <a href="https://www.youtube.com/channel/UCBJ0p9aCW-W82TwNM-z3V2w" title="IOHK, youtube.com">IOHK YouTube channel</a>, publish a weekly technical report, have dedicated project managers who produce videos on progress, hold special events and have AMA (Ask Me Anything) sessions.</p>
<p>Emurgo has been responsible for building partnerships with developers and instigating projects for the Cardano protocol around the world. Emurgo has grown from a small entity of just a few employees to a multinational effort with an ever-increasing investment portfolio.</p>
<p>Emurgo has been collaborating with IOHK on products such as the <a href="https://yoroiwallet.com/" title="yoroiwallet.com">Yoroi wallet</a>, improving the developer experience for smart contracts and DApps, and holding discussions on high-value markets to drive adoption, as well as other efforts within its mandate. These collaborations will continue to grow and become even more meaningful as we move into 2019, with Cardano achieving decentralization, multi-asset accounting and full smart contract support.</p>
<p>This acceptance of its role is also why IOHK has retained firms such as <a href="https://iohk.io/blog/functional-correctness-with-the-haskell-masters/" title="Functional Correctness with the Haskell Masters">Quviq</a>, <a href="https://www.tweag.io/" title="tweag.io">Tweag</a> and <a href="https://paymentweek.com/2018-7-30-iohk-launches-iele-virtual-machine-testnet-cardano-blockchain/" title="VM Testnet, paymentweek.com">Runtime Verification</a> to help build Cardano, refine processes and speed up development. Our collective development efforts have resulted in three codebases (Scala, Haskell and Rust), some of the first examples of applied formal methods with our new wallet backend and incredibly sophisticated techniques for modeling performance and reliability with deployed distributed systems.</p>
<p>Finally, our protocols are based on scientific inquiry. Such work should be done by scientists who have the requisite domain experience and wisdom. Thus we have directly engaged leaders in their respective fields with years to decades of experience to write our foundational papers. We have also vetted these papers through the peer review process accepted by the computer science community.</p>
<p>Like every other project, IOHK’s efforts aren’t without their flaws and setbacks. The initial release of Cardano wasn’t perfect. There were many issues ranging from some users having difficulty connecting to peers, to exchanges having trouble with the Cardano wallet. These teething problems are expected to be solved with all new codebases. However, the most important observation is that IOHK has never accepted any status quo and continues to work diligently to improve the code, the user’s experience and broaden the utility of Cardano.</p>
<p>Like IOHK, Emurgo has had its own challenges. Navigating 2017 – during a period of utterly irrational valuations, ICO mania, many poorly led ventures as well as continued regulatory uncertainty – was difficult. As with all ventures, staffing a great executive team is also a tremendous task. But as 2018 comes to a close, Emurgo has retained some <a href="https://emurgo.io/about/" title="emurgo.io">great talent</a> such as their CTO Nicolas Arqueros, chief investment officer Manmeet Singh, and one of our community’s best educators, Sebastien Guillemot. Emurgo’s collaborations with IOHK have been both meaningful and productive.</p>
<h3 id="cardano-foundation">Cardano Foundation</h3>
<p>The Cardano Foundation was created to promote the Cardano protocol, to grow and inform our community and address the needs of the community. These are broad aims and cross demographics and borders.</p>
<p>Being more specific about the needs of the Cardano community, all cryptocurrency communities need accurate, timely and comprehensive information about events, technology and progress of the ecosystem. All cryptocurrency communities need stable and moderated forums to discuss their ideas, concerns and projects. All cryptocurrency communities need liquidity and thus require access to exchanges.</p>
<p>The Cardano protocol also requires community-led efforts to gradually decentralize the protocol beyond what Bitcoin and Ethereum have achieved. A core focus outlined in the <a href="https://whycardano.com" title="whycardano.com">Why Cardano</a> white paper is the desire to establish a treasury and a blockchain-based voting system for ratifying Cardano improvement proposals.</p>
<p>This effort cannot just rely on technological and scientific innovation. Rather, it requires a well-organized and informed community that is representative of the users of Cardano and is geographically diverse. Among other things, it is the Foundation’s responsibility to invest in the creation of this community.</p>
<h3 id="lack-of-performance-by-the-cardano-foundation">Lack of performance by the Cardano Foundation</h3>
<p>For more than two years there has been great frustration in the Cardano community and ecosystem. This has been caused by a lack of activity and progress on the assigned responsibilities of the Cardano Foundation and its council. Furthermore, there has been no clear indication of improvement, despite many fruitless attempts and approaches to the Foundation’s chairman and council to change this.</p>
<p>Dissatisfaction and frustrations about the Foundation’s performance stem primarily from:</p>
<ol>
<li>A lack of strategic vision from the council.
There are no KPIs or public strategy documents outlining how the Foundation will accomplish the above goals or any discernible goal.
</li>
<br />
<li>
The absence of a clear public plan for how the Foundation will spend its funds to benefit the community.
</li>
<br />
<li>
The lack of transparency in the Foundation’s operations (for example publication of its board minutes and director remuneration).
</li>
<br />
<li>
Material misrepresentations and wrongful statements by the Foundation’s council including a claim that it owned the trademark in Cardano. The council has even tried to assume the power to decide who speaks for the protocol, what should be deployed on the protocol and how the press should represent relationships between Emurgo, IOHK, the Foundation and third party projects.
<br /><br />
Having identified the legal dubiousness and profound consequences of the Foundation’s claims in respect of trademark ownership, IOHK ceased collaboration with the Foundation until it published a fair use policy for the trademark. This process took weeks.
<br /><br />
The unpredictable conduct and lack of action by the board of the Foundation has been puzzling. For example, when IOHK went to Ethiopia to <a href="https://bitcoinmagazine.com/articles/where-coffee-just-grows-connecting-ethiopian-agritech-blockchain/" alt="bitcoinmagazine.com">sign an MOU with the Ministry of Science and Technology</a>, the Foundation originally agreed to attend and jointly sign. Unexpectedly, the Foundation decided to back out the week before and claimed in an email to IOHK’s communications director that it – without any basis or underlying agreement – was to be the single guardian of the Cardano brand and protocols.
<br /><br />
<a target="_blank" href="https://static.iohk.io/comms/CF-email.pdf">Read the email from the Foundation to IOHK here</a>.
</li>
<br />
<li>Lack of financial transparency. As of October, despite several requests the Foundation has still refused to publish the addresses holding its allocation of Ada. Neither has the Foundation published audited financial statements. And, the Foundation has not provided any information on remuneration of directors and officers.
</li>
<br />
<li>The lack of a complete and diverse Foundation council. At its incorporation (September 2016) the council consisted of 4 members, with Michael Parsons as chairman. Ten days after his appointment, a council member (Mr Parsons’ stepson, Bruce Milligan) resigned. Instead, Mr Milligan became the general manager of the Foundation. His vacancy on the council, however, was never filled. Ten months after the Foundation’s incorporation, the third council member resigned, thus reducing the council from the 4 members as intended by its founders to only 2 (Mr Parsons and a professional Swiss council representative).
<br /><br />
The vacancies have not been filled by the remaining council members. As a consequence, since 14 July, 2017, the Foundation has, in effect, been controlled by Mr Parsons. He has been acting as the Foundation’s de facto sole decision-maker in respect of the day-to-day business of the Foundation and ruling its staff like a monarch. For more than 15 months, there appear to have been no reasonable attempts to fill the 2 council vacancies. There appears to be no oversight and there appear to be no checks and balances beyond those required by Swiss law.
<br /><br />
A sound council board in the opinion of the ecosystem should consist of several active and competent and independent members. These should be domain experts from the cryptocurrency community who fairly represent the holders of Ada and users of the Cardano protocol. They should be committed to maintaining reasonable checks and balances. Although not imposed under Swiss law, the council appointment process should ideally be open to the community and include their feedback and suggestions.
<br /><br />
Despite over 90 percent of the original Ada voucher purchasers residing in Japan, the Cardano Foundation has yet to appoint anyone from Japan into a position of power. Also, the Foundation has yet to engage a lobbyist to assist with getting Ada listed on Japanese exchanges. And, the Foundation has no significant presence or personnel from Japan or even Asia.
</li>
<br />
<li>Lack of any concept of how the millions of dollars committed to the Foundation will benefit the Cardano community. Instead of working on meaningful projects such as law and policy research for ICO and STO standards for assets that will be issued on Cardano, thereby offering an alternative to Ethereum’s tokens, or studying ways to deploy Cardano’s improvement proposal process, the Foundation’s council has decided to invest its provided research capital in the <a target="_blank" href="https://cardanofoundation.org/en/distributed-futures/" alt="Distributed Futures">Distributed Futures</a> program.
<br /><br />
No explicit case has been made as to how the Distributed Futures research will benefit the Cardano protocol or the ecosystem. No funds have been committed to commercialize the research. No apparent effort has been made by council members of the Foundation to annotate the Distributed Futures reports with specifics on how the findings will be applied to our community.
<br /><br />
Furthermore, members of the ecosystem worry about potential conflicts of interest because both Robert McDowall, an adviser and contributor to Distributed Futures research, and Michael Mainelli, leader of Distributed Futures, have pre-existing relationships with Mr Parsons. Indeed, we are not aware of any process within the Cardano Foundation to analyze potential conflicts of interest and require recusal where necessary.
</li>
<br />
<li>Absence/unawareness of any meaningful internal governance system at the Cardano Foundation. In our many interactions with Foundation staff, it has never become clear how decisions are made and reviewed. It has also never been clear how the chain of command operates beyond Chairman Parsons. </li>
</ol>
<h3 id="our-call-for-action">Our call for action</h3>
<p>Emurgo and IOHK are calling for the Foundation council: to voluntarily subject itself to the Swiss authorities; for a complete audit of all of the Foundation’s financial transactions and major decisions to be conducted; and for the results to be released to the general public. This audit should include direct and indirect remuneration paid (in the light of actual and agreed performance or services delivered for the benefit of the Foundation) to Mr Parsons; his stepson Bruce Milligan who acted as a general manager; and his wife, Julie Milligan, who acted as an assistant to Mr Parsons.</p>
<p>The Cardano Foundation is an independent legal entity governed by its council, thus the Cardano community, IOHK and Emurgo cannot force the chairman to resign. Nevertheless, we can only hope that reason will persuade Mr Parsons to voluntarily step down. This would allow for regulatory oversight and avoid the Foundation continuing to be an ineffective entity.</p>
<h3 id="offer-of-iohk--emurgo">Offer of IOHK & Emurgo</h3>
<p>The Foundation and its council have not been able to execute their purpose in promoting and supporting the Cardano ecosystem. So, to provide the Cardano ecosystem with the support and services it requires and deserves, in the Foundation’s stead, IOHK and Emurgo are committed to the following actions until at least 2020:</p>
<ol>
<li>
<p>IOHK and Emurgo will begin hiring dedicated community managers for the Cardano ecosystem and assign them to growing and informing our community through meetup groups, events, educational efforts and other metrics that can be tracked.</p>
</li>
<li>
<p>IOHK is willing to hire, subject to reasonable due diligence and negotiations, Cardano Foundation personnel directly engaged in community management should they desire to leave the Foundation.</p>
</li>
<li>
<p>IOHK will work with Emurgo to start efforts in Japan to improve exchange access and community understanding of Cardano.</p>
</li>
<li>
<p>IOHK and Emurgo will scale up its educational and marketing efforts to include more content about the Cardano protocols, developer resources and USPs of our ecosystem.</p>
</li>
<li>
<p>IOHK has hired an open source community manager to draft the Cardano improvement proposal process and begin its rollout.</p>
</li>
<li>
<p>IOHK has expanded its research scope to include the areas originally forseen for the Cardano Foundation.</p>
</li>
<li>
<p>IOHK will start a research agenda to design a decentralized Foundation built as a DAO to be deployed on the Cardano computation layer. We will announce a dedicated research center at a later date.</p>
</li>
</ol>
<h3 id="final-thoughts">Final thoughts</h3>
<p>First, IOHK and Emurgo’s funding for the Cardano project is fully secured, independent, and not connected to the Cardano Foundation. The Foundation is not in a position to mandate or compel changes in the operations of the Cardano platform, IOHK, or Emurgo.</p>
<p>Second, the original intention of separating powers within the Cardano ecosystem was to ensure that the failure of one entity would not destroy the project. This resilience has allowed us to thrive, despite the Foundation’s lack of progress and vision.</p>
<p>Third, the real strength of Cardano stems from its exceptional community, which continues to grow and impress us. The Foundation’s role is similar to the Bitcoin Foundation’s, in that its purpose is to add value to the community. Like the Bitcoin Foundation for Bitcoin, the Cardano Foundation is not necessary for Cardano to succeed as a project.</p>
<p>And last, but not least, for IOHK, Cardano is more than a product. Cardano is a mission to deliver a financial operating system to the three billion people who need a new one. Our personnel have been to more than 50 countries over the past three years representing Cardano. We will continue to do so over the coming years because we see the power of this technology and the people it can help.</p>
<p>As the CEOs of IOHK and Emurgo, we are deeply disappointed that we have not been able to activate and increase the performance of the Foundation. We have not been able to resolve the above outstanding matters in another way. We are also deeply disappointed that our community has been repeatedly let down by the Foundation, yet we are determined to ensure that the community will be served in the manner it deserves to be served.</p>
<p>Regardless of the above, we believe our best days are ahead of us. We believe Cardano will become the best technology to deliver financial infrastructure to the billions who lack it.</p>
<div class="col-md-12" style="height:110px">
<img style="border: 1px solid #ddd;padding: 1px;float: left;width: 90px !important;height: 90px !important;background: #eee;margin-right: 15px" class="img-circle" src="/images/team/charles.png" />
Charles Hoskinson,<br />
Chief Executive Officer,<br />
Input Output HK Ltd.</div>
<div class="col-md-12" style="height:110px">
<img style="border: 1px solid #ddd;padding: 1px;float: left;width: 90px !important;height: 90px !important;background: #eee;margin-right: 15px" class="img-circle" src="/images/team/ken-kodama.png" />
Ken Kodama,<br />
Chief Executive Officer,<br />
Emurgo</div>
<p><br /><br /><br /><br /></p>
<p><em>This article has been corrected to reflect the fact that Bruce Milligan is Michael Parson’s stepson, rather than son-in-law, as previously stated.</em></p>
<p><small>Artwork, <a href="https://creativecommons.org/licenses/by/4.0/" target="_blank"><i class="fa fa-creative-commons" aria-hidden="true"></i> <a href="http://www.beeple-crap.com">Mike Beeple</a></a></small></p>iohk-emurgoFunctional correctness with the Haskell masters2018-09-26T00:00:00+00:002018-09-26T00:00:00+00:00https://iohk.io/blog/functional-correctness-with-the-haskell-master<p><img src="/images/blog/functional.jpg" alt="IOHK Haskell and Cryptocurrency Course in Barbados" class="fullwidth" /></p>
<p>At IOHK, we are proud of our scientific approach and close collaboration with academia. We publish in peer reviewed scientific journals and present our results at acclaimed international conferences to ensure that our protocols and algorithms are built on rock-solid foundations. Our software must reflect this scientific excellence and quality, which means that we need a process to go from scientific results to actual code written in the Haskell programming language. We therefore decided to run internal training on “functional correctness”, so that the quality of our theoretical foundations can translate into equal quality for our code.</p>
<p>We ran the first course over four days in Regensburg, Germany, two weeks ago. This training is aimed at everybody writing Haskell at IOHK, so we decided to run four sessions, roughly based on geography – there are IOHK engineers in 16 countries. We plan to do a second session in Regensburg in November and then two more early next year in the US. The lecturers were Andres Löh, co-founder of the Well-Typed consultancy, and John Hughes, the founder of <a href="http://www.quviq.com/" title="quiq.com">QuviQ</a>, who are both prominent in the Haskell world.</p>
<iframe width="100%" height="415" src="https://www.youtube.com/embed/dIVAinPWypE" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen=""></iframe>
<p>John is one of the creators of Haskell and the co-inventor of <a href="http://hackage.haskell.org/package/QuickCheck" title="hackage.haskell.org/package/QuickCheck">QuickCheck</a>, the Haskell testing tool. Most mainstream software companies (if they do testing at all, which, sadly, is not always the case), use unit tests. For this, developers write down a number of tests by hand, cases that they deem typical or relevant or interesting, and then use a unit test framework to run the tests and report whether they yield the expected results. QuickCheck is different. Instead of specifying a handful of tests, developers using QuickCheck state the properties that their code should have. QuickCheck then generates many random test cases and checks the property for each of these. If QuickCheck finds that a property is violated, it first tries to simplify the test, then reports the simplest failing case back to the user.</p>
<figure class="alignright">
<img src="/images/blog/haskell-training.jpg" alt="Learning in Regensburg" width="400" height="" />
<figcaption>Haskell students in class</figcaption>
</figure>
<p>As a simple example, let’s say you wrote a program to sort a list of names. Using unit tests, you would check the program against a few handcrafted examples of lists of names (something like [“Tom”, “Dick”, “Harry”] and [“Dora”, “Caesar”, “Berta”, “Anton”] ). With QuickCheck, on the other hand, you would sit down and carefully think about properties your program should have In the example of sorting lists of names, what properties would you expect? Well, after running the program, you should get a list that is sorted alphabetically. Oh, and that list should contain all the names you entered. And yes, it should only contain those names you entered. You can write down these properties as Haskell programs, then hand them over to QuickCheck. The tool checks your properties against as many randomly generated lists of names as you wish (usually hundreds or thousands) and identifies any violations.</p>
<p>In practice, QuickCheck often manages to find problems that are overlooked by less rigorous methods, because their authors tend to overlook obscure cases and complicated scenarios. In our example, they may, for example, forget to test an empty list of names. Or there may be a bug in the program that only occurs for long lists of names, and their unit tests only check short lists. John had many ‘war stories’ of this happening in real life with real customers, where bugs were only revealed after a series of complex interleaved operations that no human unit test writer would have imagined.</p>
<p>Every Haskell developer has heard of QuickCheck and understands the basic ideas, but in complex real-world programs like Cardano, it is sometimes not so easy to use the tool properly. It was therefore great to have the intricacies and finer points explained by John himself, who has been using QuickCheck for 20 years and has worked with many industries, including web services (Riak, Dropbox and LevelDB), chat servers (Ejabberd), online purchasing (Dets), automotive (Autosar specification), and telecommunications (MediaProxy, Ericsson and Motorola). He helps find bugs and guarantee correctness every day. Given John’s experience, the training participants were able to spend about half of their time learning the finer points of QuickCheck from the master himself. It was tremendous fun enjoying John’s obvious enthusiasm for, and deep knowledge of, the subject. The rest of the session was dedicated to understanding the link between formal specifications, written in a mathematical style, and Haskell implementations.</p>
<figure class="alignright">
<img src="/images/blog/iohk-regensburg.jpg" alt="Exploring Regensburg" width="400" height="" />
<figcaption>IOHK in Regensburg</figcaption>
</figure>
<p>At IOHK, we work very hard on writing correct code. For example, we specify program behavior and properties using rigorous mathematics. In the end, of course, we can’t deploy mathematics to a computer. Instead, our developers have to take the specification, translate the mathematics into Haskell and produce executable, efficient code. This process is easier for Haskell, because it is firmly rooted in mathematical principles, than for most languages, but it is still a conceptual leap. The specification talks about mathematical objects like sets and relations, which have to be translated into data types and functions as faithfully as possible. Nobody wins if your beautiful mathematics is ‘lost in translation’ and you end up with bug-ridden code. For example, when mathematicians talk about integers (…, −2, −1, 0, 1, 2,…) or real numbers (such as π, and √2), how do you express this in Haskell? There are data types like Int or Double that seem related, but they are not the same as the mathematical concepts they were inspired by. For example, a computer Int can overflow, and a Double can have rounding errors. It is important to understand such limitations when translating from mathematics to code. This is where the mathematician and renowned Haskell expert Andres Löh came in. He taught the participants how to read mathematical notation, how mathematical concepts relate to Haskell and how to translate from the one to the other.</p>
<p>For example, Andres presented the first pages of our formal blockchain specification and talked the participants through understanding and implementing this piece of mathematics as simple (and correct!) Haskell code, which led to interesting questions and lively discussions: How do you represent hashing and other cryptographic primitives? What level of detail do you need? Is it more important to stay as faithful to the mathematics as possible or to write efficient code? When should you sacrifice mathematical precision for simplicity?</p>
<p>In addition to their great lectures, John and Andres also provided challenging practical exercises, where participants could immediately apply their newly-gained knowledge about testing and specifications. Finally, there was plenty of opportunity for discussions, questions and socializing. Regensburg is a beautiful town, founded by the Romans two thousand years ago and a Unesco World Heritage Site. The city offered participants a perfect setting to relax after the training, continuing their discussions while exploring the medieval architecture or sitting down for some excellent Bavarian food and beer.</p>
<p><small>Artwork, <a href="https://creativecommons.org/licenses/by/4.0/" title="Creative Commons"><i class="fa fa-creative-commons" aria-hidden="true"></i></a> <a href="http://www.beeple-crap.com">Mike Beeple</a></small></p>lars.bruenjes@iohk.ioIOHK releases Icarus to the Cardano community2018-08-15T00:00:00+00:002018-08-15T00:00:00+00:00https://iohk.io/blog/iohk-release-icarus-to-the-cardano-community<p><img src="/images/blog/icarus-sun.jpg" alt="Icarus" class="fullwidth" /></p>
<p>Today IOHK releases Icarus, a reference implementation for a lightweight wallet developed by the IOHK engineering team.</p>
<p>We hope that this code base will be used as a point of reference to enable developers to create their own secure light and mobile wallets for Cardano. <a href="https://github.com/input-output-hk/project-icarus" title="Project Icarus, github.com">Icarus</a> is a fully open source code base that will be the first step in a range of open source initiatives to provide developers with a suite of tools for Cardano.</p>
<p>Icarus was born out of a series of proof of concepts which began back in March of this year. A small section of the IOHK engineering team were interested to find out if they could demonstrate that it would be possible to create a lightweight Cardano wallet with all the features of <a href="https://daedaluswallet.io/" title="Daedalus Wallet, daedaluswallet.io">Daedalus</a> but that was easy to use and fast to set up. Whilst we are improving Daedalus synchronization speeds all the time, notably in the recent 1.3 release, we wanted to see if we could build something fast for Ada users who might not require the full features of Daedalus, or might not have the bandwidth or machine requirements to easily run Daedalus.</p>
<p>Therefore, investigating whether it would be possible to build a wallet where the user did not have to download the whole blockchain – and could run in a browser or on a mobile device – was worth the effort of a small dedicated team.</p>
<p>To build a wallet like this, we would need to prove that we could safely and securely store private keys and execute client-side cryptographic operations in the browser. In tandem, we would need to communicate with the Cardano nodes to provide users with their current UTxO state. If this could be accomplished, it would be no mean feat, hence the name Icarus – from the beginning we knew we would be flying close to the sun.</p>
<p>The team set out in at the beginning of March with ambitious goals to see if, within a month, we could build a skeleton Chrome extension application and verify that Cardano cryptography could be run in the browser using WebAssembly compiled from Rust. The Rust library of Cardano primitives has been developed by Vincent Hanquez and Nicolas Di Prima, IOHK Specialised Cryptographic Engineers, and has already been used for the paper wallet certificate feature in Daedalus.</p>
<p>To build this Chrome extension, we would need to successfully demonstrate we could import and track a wallet balance. Of course, we would have to do all this without sacrificing the IOHK engineering principles of quality and security.</p>
<p>The demo at the end of March went well and produced a functional prototype that we could develop. Once each demo had been given, the wider IOHK engineering team had a chance to review, critique and provide feedback about the design decisions the Icarus project team was taking, which proved invaluable to the process. After proof of concept 1, it was felt that good progress was being made and another month’s effort from the team would be worthwhile.</p>
<p>Proof of concept 2 was delivered in mid-April. The Rust engineers had spent the intervening time extending the Rust library to support the Cardano primitives for the creation, signing, and broadcast of transactions, and providing an API so that these could be run in the browser. On the application side, we wanted to see if we could reuse the UX/UI components of Daedalus to provide a smooth user experience. Luckily, the IOHK Daedalus development team has maintained a high-quality, portable UI framework for React, called React-Polymorph, which we found to be easily portable to the Chrome extension.</p>
<p>Proof of concept 3 in late May involved making Icarus fully interoperable with the Daedalus wallet. The team worked to develop a new Hierarchical Deterministic (HD) address scheme that Daedalus will use in future and will ensure ongoing compatibility. One important feature we built at this point was to allow the user to enter their Daedalus wallet recovery phrase in Icarus, and for their Ada in Daedalus to be transferred to the Icarus wallet. In effect, this allows users to retrieve their Ada without using the Daedalus wallet. We also optimised wallet restoration times. Finally, after only three months and three demo’s we had a fully functional prototype lightweight Cardano wallet!</p>
<p>Before we could ensure this was a reference implementation we could release to the community, we wanted to ensure that it performed at scale. This, along with some code clean-up, was the main task of the final proof of concept 4 in early June. We called upon the experience of Philipp Kant, in IOHK benchmarking, and Neil Davies, leading networking, and successfully conducted a series of rigorous stress and failover tests on the architecture.</p>
<p>The code base has been quality assured by Allied Testing, a leading quality assurance and testing company. We also engaged Kudelski Security, a cybersecurity consultancy, to perform a full security audit of the code – their report will be published soon.</p>
<figure class="fullwidth">
<img src="/images/blog/emurgo-trip.jpg" alt="emurgo trip" width="" height="" style="display:block; margin: 0 auto;" />
<figcaption>L-R: Nicolás Arqueros, Alan Verbner, Brian McKenna, Sebastien Guillemot</figcaption>
</figure>
<p>We knew that Emurgo, the organisation that supports new Cardano ventures, was interested in releasing the first implementation of Icarus to the community. To that end, we invited two Emurgo staff – Nicolás Arqueros, chief technology officer, and Sebastien Guillemot, technical manager – to Buenos Aires to meet lead Icarus developer Alan Verbner and his team in July. The goal of this trip was to see if the code could be understood and deployed by open source community members. Emurgo provided feedback on how we could make the reference implementation ready to release as a product and they wrote a technical specification for the code base. We are excited that <a href="https://www.youtube.com/watch?time_continue=9&amp=&v=GLNgpr-3t2E" target="_blank">Emurgo</a> will soon be launching their implementation of Icarus, the Yoroi wallet, and look forward to seeing how they carry through their vision for the product.</p>
<p>In mid-July, Hayley McCool Smith, IOHK Product Marketing Manager, visited Emurgo at their offices in Tokyo. One of the purposes of the trip was to take part in a naming workshop which would help Emurgo bring their product to life. After spending a day working through a plethora of contenders that Emurgo had shortlisted, it was decided that “Yoroi” was the perfect fit. In Japanese, Yoroi means “great armour” and is a prominent example of the type of secure armament that Samurais would wear to protect themselves. With the name decided, it was up to the team to create a logo that would reflect a new lightweight wallet, while also incorporating the traditional Samurai meaning of the word.</p>
<figure class="fullwidth">
<img src="/images/blog/emurgo-tokyo.jpg" alt="emurgo tokyo" width="850" height="" />
<figcaption>Emurgo team in Tokyo</figcaption>
</figure>
<p>The Rust library that was used to bring the Cardano cryptography into the browser has spawned another IOHK project, the Cardano Rust project. (This has been known as Project Prometheus internally.) IOHK will be releasing more information on this in due course. The Cardano Rust project will maintain the open source spirit of Icarus and further extend the toolbox of Rust modules. The project will be made available to the open source community to easily build high-quality applications using Cardano. The first product of the project will be a full command line interface wallet, which you can expect to see in September.</p>
<p>The segmented development team and rapid iteration approach to software development has worked well on Project Icarus and we will be employing this strategy again. We are happy that Ada holders will have the ability to store their Ada in the really cool Yoroi wallet and that developers have a high-quality reference implementation on which to base their own new light and mobile wallets for Cardano. The project has also given rise to Project Prometheus which is the natural evolution of the spirit of Icarus.</p>
<p>We feel that we have developed, in quite a short time, a very useful quality assured and security audited reference implementation for a lightweight Cardano wallet. We encourage the open source community to fork the Icarus code base, compile it, and maybe even build your own wallet for Cardano. We welcome contributions and hope that this effort will benefit the entire community.</p>
<p><em>This blog has been amended to update the name of the Cardano Rust project from Project Prometheus.</em></p>
<p><small>Artwork, <a href="https://creativecommons.org/licenses/by/4.0/" target="_blank"><i class="fa fa-creative-commons" aria-hidden="true"></i> <a href="http://www.beeple-crap.com">Mike Beeple</a></a></small></p>brian.mckenna@iohk.ioHow does Casper compare to Ouroboros?2018-08-09T00:00:00+00:002018-08-09T00:00:00+00:00https://iohk.io/blog/how-does-casper-compare-to-ouroboros<p><img src="/images/blog/circular.jpg" alt="Circular" class="fullwidth" /></p>
<p><strong>TL;DR</strong></p>
<p>In response to recent discussions in social media, we give a brief comparison of the Ouroboros and Casper proof-of-stake protocols.</p>
<p>Ouroboros is a formally specified and analysed protocol with mathematically proven security guarantees based on clearly specified assumptions. The protocol description, models and proofs are all public. Hence, the underlying assumptions, the target protocol properties, and the respective correctness proofs can be publicly scrutinised. Ouroboros offers stake-based finality with the strongest possible guarantees in terms of the amount of stake backing up honest operation. It also provides a solid foundation over which services such as near instant finality of transactions can be offered in optimistic network conditions.</p>
<p>Regarding Casper, we are not aware of any currently published source that sufficiently describes the protocol’s mode of operation nor any provable guarantees about it. Still, from what has been presented about Casper until now, as compared to Ouroboros, we can safely conclude that Casper provides much weaker guarantees in terms of how much stake the adversary needs to control in order to disrupt the protocol. Below, we compare the two protocols along several dimensions; for lack of proper documentation, many properties of Casper have to be assumed to the best of our knowledge.</p>
<hr />
<p>In response to a discussion <a href="https://www.reddit.com/r/ethereum/comments/92f1u0/eli30_differences_between_casper_and_ouroboros/" title="Differences between Casper and Ouroboros, Reddit">here</a> and <a href="https://www.reddit.com/r/cardano/comments/92r3si/vitalik_allegations_against_ouroboros/" title="Vitalik allegations against Ouroboros, Reddit">here</a>, we give a brief comparison of the Ouroboros proof-of-stake (PoS) protocol and Casper PoS. For Ouroboros, we refer to the <a href="https://eprint.iacr.org/2016/889" title="Ouroboros: A Provably Secure Proof-of-Stake Blockchain Protocol, eprint">original version</a> underlying the Cardano Settlement Layer (published at <a href="https://www.iacr.org/conferences/crypto2017/" title="Crypto 2017, iacr.org">Crypto 2017</a>), however most of our comments apply to later versions <a href="https://eprint.iacr.org/2017/573" title="Ouroboros Praos: An adaptively-secure, semi-synchronous proof-of-stake protocol, eprint">Ouroboros Praos</a> and <a href="https://eprint.iacr.org/2018/378" title="Ouroboros Genesis: Composable Proof-of-Stake Blockchains with Dynamic Availability, eprint">Ouroboros Genesis</a> as well. For Casper, we primarily refer to the Casper Friendly Finality Gadget (FFG) as described in the <a href="https://arxiv.org/abs/1710.09437" title="Casper the Friendly Finality Gadget, arxiv.org">white paper</a>, being the most recent Casper proposal that is sufficiently descriptive to draw a full comparison (other references include <a href="https://docs.google.com/document/d/1maFT3cpHvwn29gLvtY4WcQiI6kRbN_nbCf3JlgR3m_8/edit" title="Ethereum 2.0 Mauve Paper">Ethereum Mauve</a>, <a href="https://notes.ethereum.org/SCIg8AH5SA-O4C1G1LYZHQ?view" title="Casper+Sharding chain v2.1, ethereum.org">Casper+Sharding v2.1, FFG-RPJ</a>, <a href="https://github.com/ethereum/research/blob/master/papers/CasperTFG/CasperTFG.pdf" title="Casper the Friendly Ghost, github.com">Casper TBG/CBC</a>).</p>
<p>Any PoS ledger consensus protocol should satisfy two fundamental properties: persistence and liveness. The first ensures that the ledger is final and immutable. The second ensures that transactions broadcasted by honest parties are eventually included in the (immutable) ledger. Such properties, typically, cannot be proven unconditionally: they will rely on certain conditions, some of them cryptographic, e.g., that digital signatures cannot be forged, while others are related to the behaviour of the participants, e.g., that the players who follow the protocol control a majority of the stake. There are other desirable properties that a PoS protocol should satisfy (such as that executing the protocol as prescribed is the only rational strategy for the participants), but persistence and liveness as defined above constitute the bare minimum pair of fundamental properties necessary for ledger consensus.</p>
<p>Let us now discuss some of the differences between the two protocols and their analyses.</p>
<h3 id="execution-model-and-falsifiability-of-claims">Execution Model and <a href="https://en.wikipedia.org/wiki/Falsifiability" title="Falsifiability, wikipedia.org">Falsifiability</a> of Claims</h3>
<p>The Ouroboros protocol is analyzed in a model that is fully described: it unambiguously defines all the participants’ programs, their execution and interactions, their communication – including network properties – and the potential corruption by an adversarial entity of any set of parties controlling a minority of the stake. Such a model allows the formulation of mathematically precise security guarantees satisfied by any execution, such as the persistence and liveness properties proven for Ouroboros. In particular, the formal modeling of Ouroboros permits precise, quantitative statements about stake bounds and settlement times; see below. <strong>This makes all the claims we make about Ouroboros entirely concrete; there is nothing left up to interpretation or reader perspective.</strong> Without such a model (notably missing in the Casper FFG white paper or in any other available sources related to Casper), it is impossible to prove the correctness of any claims about the protocol. Consensus protocols, in general, are complex objects; designing them without the development of rigorous mathematical arguments that establish the required properties can prove to be precarious as prior practice in secure systems design has shown. Good design intuition and best effort are just not sufficient when a ledger consensus protocol is supposed to carry assets worth billions.</p>
<h3 id="a-comprehensive-solution-to-pos-ledger-consensus">A comprehensive solution to PoS ledger consensus</h3>
<p>Given the above, it is important to appreciate that the Ouroboros protocol is proven to provide persistence and liveness under clearly defined assumptions such as honest stake majority which is the bare minimum assumption needed in the PoS setting. On the other hand, Casper FFG, as described in the white paper, is an enhancement on top of a pre-existing “block proposal mechanism”, e.g., a PoW blockchain (namely Ethereum); in particular, its security guarantees as a ledger consensus protocol depend on the security of this proposal mechanism. As the authors of Casper FFG observe, “a wholly compromised block proposal mechanism will prevent Casper from finalizing new blocks”, hence the honest-majority-of-hashing power assumption is still necessary for Casper FFG’s liveness. Similarly, other versions of the Casper protocol, such as Casper FFG-RPJ, are incomplete and/or not accompanied by any proofs of security.</p>
<h3 id="stake-assumptions">Stake Assumptions</h3>
<p>Ouroboros is proven to achieve persistence and liveness under the assumption of honest majority of all stake in the system, even in the case that some significant portions of stakeholders are not participating in the protocol (see e.g., Theorem 1 in the <a href="https://eprint.iacr.org/2018/378" title="Ouroboros Genesis: Composable Proof-of-Stake Blockchains with Dynamic Availability, eprint">Ouroboros Genesis</a> paper for the most comprehensive statement on Ouroboros security). In contrast, Casper requires a ⅔-fraction of deposited stake to be controlled by honest parties (see section 2.1 of the <a href="https://arxiv.org/abs/1710.09437" title="Casper the Friendly Finality Gadget, arxiv.org">white paper</a>). Since the deposited stake is blocked and cannot be used for other purposes in the meantime, it is reasonable to assume that the deposited stake will be a small fraction of the total stake in the system. Naturally, larger amounts of stake are more difficult to control so that basing security on the total stake in the system, as in Ouroboros, is a more prudent choice. As a concrete example, in the current sharded version of Ethereum (Ethereum Mauve paper or Casper+Sharding chain v2.1), a minimum of 32 ETH per validator is required with 100-128 validators per shard depending on the reference, without any other restriction. It follows that if the total deposited stake among all prospective validators turns out to be minimal and is not otherwise restricted then just a few thousand ETH would be enough to register a set of <a href="https://en.wikipedia.org/wiki/Sybil_attack" title="Sybil Attack, wikipedia.org">sybil</a> validators that could disrupt the ledger consensus security properties.</p>
<h3 id="finality">Finality</h3>
<p>Though the notion is not formally defined in the Casper FFG white paper, it is easy to see that the property of “stake-based finality” is subsumed by persistence, the property that ensures that transactions become permanently part of the public immutable ledger; the stake-based adjective on finality used in Casper FFG refers to the fact that the condition under which finality is to be attained is based on stake as opposed to, e.g., a hashing power assumption. As mentioned above, no protocol can be deemed to solve the ledger consensus problem without providing persistence (and hence finality). In fact, all PoS protocols provide such properties only with a high probability – if for no other reason, cryptography can always fail with (very) small probability (for example, someone may guess your key). We do in fact know that Bitcoin and (pre-Casper) Ethereum provide finality (shown by the works of <a href="https://eprint.iacr.org/2014/765" title="The Bitcoin Backbone Protocol: Analysis and Applications, eprint">GKL15</a>, <a href="https://eprint.iacr.org/2016/1048" title="The Bitcoin Backbone Protocol with Chains of Variable Difficulty">GKL17</a> and <a href="https://eprint.iacr.org/2016/454" title="Analysis of the Blockchain Protocol in Asynchronous Networks">PSS17</a>) assuming honest majority of computational power), and so does Ouroboros, assuming honest majority of stake as shown in <a href="https://eprint.iacr.org/2016/889" title="Ouroboros: A Provably Secure Proof-of-Stake Blockchain Protocol, eprint">KRDO17</a>, <a href="https://eprint.iacr.org/2017/573" title="Ouroboros Praos: An adaptively-secure, semi-synchronous proof-of-stake protocol, eprint">DGKR18</a>, <a href="https://eprint.iacr.org/2018/378" title="Ouroboros Genesis: Composable Proof-of-Stake Blockchains with Dynamic Availability, eprint">BGKRZ18</a>.</p>
<p><strong>Put simply, Ouroboros provides stake-based finality</strong> and it does so with the strongest possible guarantee in terms of stake: against a malicious coalition controlling any amount of the total stake existing in the system as long as it is bounded below 50%. In the Casper FFG white paper, where Casper operates over the Ethereum blockchain, stake-based finality is provided every 100 blocks under the assumption that ⅔ of the deposited stake is honest. As a concrete example, in the same window of time, which is a little over half an hour in our current deployment, we can derive from our formal analysis that Ouroboros will offer finality against, say, a 10% stake adversary with probability of error less than 2^(-44). This is less than 1/10000000000000, one over ten trillion. To appreciate such small numbers, consider that it is expected to have one large asteroid hit the earth once every 100 million years (<a href="https://www.scientificamerican.com/article/what-is-the-chance-of-an/" title="What is the chance of an asteroid hitting Earth and how do astronomers calculate it?, scientificamerican.com">Scientific American</a>). Thus, it is 10 thousand times more likely that a big asteroid will hit the earth next month than that Ouroboros will reorganise its chain to drop a particular transaction after it has been included in the ledger for about half an hour.</p>
<h3 id="eventual-consensus-vs-near-instant-finality">Eventual Consensus vs. (near-)Instant finality</h3>
<p>Blockchain protocols like Bitcoin and Ouroboros are called eventual-consensus since they ensure that the irreversibility of a block increases gradually with the number of blocks that are added on top of it. This means that finality is more nuanced than just a true or false value, and is quantified by the probability of reverting a transaction as a function of the strength of the adversary and the length of time that has passed since the block containing that transaction was added. This design enables these protocols to work in the strongest possible adversarial settings and still be very efficient in terms of the number of messages that need to be exchanged; furthermore, they have the feature that the recipient of a transaction can decide for herself how important a transaction is and adjust her own notions of stability per transaction. Their downside is that they do not provide near-instant finality, or in other words, a fast assurance that the transaction will be finalised. This may be a potential advantage of classical BFT protocols that have inspired the design of Casper FFG as well as other protocols in the space including Algorand.</p>
<p>However, near-instant finality typically also comes with significant downsides in terms of the security model such as a much higher requirement of honest stake or, perhaps more importantly, a high degree of guaranteed online presence that must be offered by the participants following the protocol. This hurts the dynamic availability of the participants (see below) which is one of the hallmarks of the bitcoin era of consensus protocols. On the other hand, near-instant finality can be built as a service on top of Ouroboros and this is something that we will be releasing in due course. Moreover, we can argue that this is the best possible way forward: use the Ouroboros eventual consensus protocol which is secure under the strongest possible stake-based guarantees as the solid foundation over which services such as near-instant settlement in optimistic network conditions can be safely built.</p>
<h3 id="incentives-and-dynamic-availability">Incentives and dynamic availability</h3>
<p>Casper FFG is inspired by pre-Bitcoin era standard BFT consensus protocols and as such it cannot handle uncertainty in terms of the number of participating entities once the set of validators becomes fixed. This means that the protocol cannot operate in the “<a href="https://eprint.iacr.org/2016/918" title="The Sleepy Model of Consensus, eprint">sleepy setting</a>” and “<a href="https://eprint.iacr.org/2018/378" title="Ouroboros Genesis: Composable Proof-of-Stake Blockchains with Dynamic Availability, eprint">dynamic availability</a>” setting, where a significant number of parties that are supposed to act in the protocol are unavailable due to network conditions, hardware failure or simply lack of interest. This is a significant concern in a decentralized setting where the execution of the protocol is not meant to be left in the hands of a few centralized-power actors, but is rather distributed proportionally among a great number of smaller players. The Casper-FFG white paper acknowledges this as the “Catastrophic Crash” scenario and observes that in this case “no future checkpoints can be finalized”. The authors propose a mitigation in the form of the so-called “inactivity leak.” This idea is only described informally as draining “the deposit of any validator that does not vote for checkpoints, until eventually its deposit sizes decrease low enough that the validators who are voting are a supermajority.” Unfortunately, this modification would in turn negate any potential advantage Casper can claim in face of network splits, as the authors also recognise: “The inactivity leak introduces the possibility of two conflicting checkpoints being finalized without any validator getting slashed.” This also affects the incentives running the protocol. Ouroboros allows for a natural and incentive-driven aggregation of stake into stake pools that will be performed over a period of time using our <a href="https://arxiv.org/abs/1807.11218" title="Reward Sharing Schemes for Stake Pools, arxiv.org">stake pool reward mechanism</a>, without forcing the behaviour of stakeholders onto a predetermined structure, while Casper has to impose preset numbers of block validators.</p>
<h3 id="randomness">Randomness</h3>
<p>While the original Ouroboros protocol does not use VRFs to generate protocol randomness (instead it uses a guaranteed-output-delivery coin-tossing protocol based on verifiable secret-sharing), the follow-up versions Praos and Genesis do so for performance gains. The VRFs proposed for use in Ouroboros Praos and Genesis are proven secure under standard cryptographic assumptions (such as the <a href="https://en.wikipedia.org/wiki/Computational_Diffie%E2%80%93Hellman_assumption" title="Computational Diffie Hellman assumption, wikipedia.org">Computational Diffie Hellman assumption</a>) while the security analysis we have performed ensures Ouroboros’ resilience to randomness manipulation (see <a href="https://eprint.iacr.org/2017/573" title="Ouroboros Praos: An adaptively-secure, semi-synchronous proof-of-stake protocol, eprint">Ouroboros Praos</a> and <a href="https://eprint.iacr.org/2018/378" title="Ouroboros Genesis: Composable Proof-of-Stake Blockchains with Dynamic Availability, eprint">Ouroboros Genesis</a>).</p>
<h3 id="network-assumptions">Network Assumptions</h3>
<p>Ouroboros is analysed in the “partially synchronous” setting where messages are delivered to the majority of the parties executing the protocol within a time window upper bounded by a network delay Δ which is unknown to the parties. The order of messages is adversarial and it is not guaranteed that two honest parties will receive messages in the same order. The adversary is allowed to inject arbitrary messages selectively to any of the parties. Casper makes no explicit claims about the network setting it operates in, nevertheless, when describing defenses against long range revisions it alludes to a similar type of model.</p>
<h3 id="sharding">Sharding</h3>
<p>This property refers to the ability of a database or ledger consensus protocol to scale its processing power as more nodes (or processing capacity) enter the system, ideally with a linear speedup in the number of nodes added. Ouroboros Hydra, the scalable version of Ouroboros is in development and will be released in due time following our usual mode of discourse, i.e., the release of a full paper containing complete mathematical formulations of the problem that we solve, a full description of our protocol solution, as well as concrete statements about the protocol’s properties that are accompanied by all necessary proofs. At present, the version of Casper that enables sharding, (<a href="https://notes.ethereum.org/SCIg8AH5SA-O4C1G1LYZHQ?view" title="Casper+Sharding chain v2.1, ethereum.org">Casper+Sharding v2.1</a>), is incomplete even in terms of protocol description, and as such, it cannot allow any proof of security.</p>
<p><a href="https://www.youtube.com/watch?v=Nlmv4fg4NQk&list=PLnPTB0CuBOBw9H7dynFu9U25vqFWRw1UX" title="Ouroboros: A Provably Secure Proof-of-Stake YouTube">Learn more about Ouroboros</a>.</p>
<p><em>Team effort is a hallmark of IOHK research and this blog post is no exception. I am grateful to Christian Badertscher, Matthias Fitzi, Peter Gaži, Alexander Russell, Jeremy Wood, and Vassilis Zikas for various suggestions, comments, and corrections to the above text.</em></p>
<p><small>Artwork, <a href="https://creativecommons.org/licenses/by/4.0/" target="_blank"><i class="fa fa-creative-commons" aria-hidden="true"></i> <a href="http://www.beeple-crap.com">Mike Beeple</a></a></small></p>aggelos.kiayias@iohk.ioFrom free algebras to free monads2018-08-07T00:00:00+00:002018-08-07T00:00:00+00:00https://iohk.io/blog/from-free-algebras-to-free-monads<!-- This post is written in HTML not Markdown and uses MathJax.org javascript from a cdn to display algebraic equations-->
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
tex2jax: {inlineMath: [['$','$'], ['\\(','\\)']]},
TeX: {
extensions: ["AMSmath.js"]
},
});
</script>
<style>
.mjx-chtml.MathJax_CHTML {
font-size: 1.1em !important;
}
.mjx-chtml.MJXc-display {
display: block;
font-size: 0.9em;
padding: 10px 20px;
color: #505050;
}
</style>
<script type="text/javascript" async="" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/latest.js?config=TeX-AMS-MML_HTMLorMML"></script>
<p><img src="/images/blog/blob.jpg" alt="Blob" class="fullwidth" /></p>
<p>In <a href="https://en.wikipedia.org/wiki/Universal_algebra">universal algebra</a> <strong>freeness</strong> is a well defined algebraic property. We will explore equational theories which are tightly connected to free algebras. We will consider free monoids. Then we’ll explain how monads can be brought into the picture in the context of monoidal categories. This will lead to a precise definition of a free monad as a free monoid.</p>
<p>This post requires familiarity with some very basic Category Theory and does not assume any knowledge on universal algebra. Most mathematical notions will be introduced but you might want to dig into literature for some more examples; though most of the books are quite lengthy and not suited for non-mathematicians - you’ve been warned ;). Knowing this I tried to bring all the required definitions, together with some very basic examples. As you read you may want to read about semigroups, monoids, groups, $G$-sets, lattices, Boolean or Heyting algebras from Wikipedia articles or try to find info on <a href="https://ncatlab.org/nlab/show/HomePage">nCatLab</a> (though this is is a heavy resource, with mostly with higher categorical approach, so probably better suited for more familiar readers).</p>
<h2>Preliminaries</h2>
<p>
We will need some preliminary definitions. Let's begin with a definition of algebra. For a set $A$ we will denote $A^n$ the $n$th cartesian product of $A$, i.e. $A^2=A\times A$.
</p>
<div class="definition">
<h5><span>Definition</span> Algebra</h5>
<div>
An <b>algebra</b> $\underline{A}$ is a set $A$ together with a finite set of operations $f_i^{\underline{A}}:A^{j_i}\rightarrow A$ ($i=1,\ldots,n$), usually simply written as$\underline{A}=(A, (f_i^{\underline{A}})_{i=1,\ldots,n})$. For operation $f_i^{\underline{A}}$ the natural number $j_i$ is called its arity, which is a finite natural number. The set $A$ is usually called <b>universum</b> of the algebra $\underline{A}$. The set of symbols $f_i$ is called <b>type of the algebra</b> $\underline{A}$.
</div>
</div>
<p>Examples includes many classical algebraic structures, like <i>semigroups</i>, where there is only a single operation of arity 2, <i>monoids</i> which in addition have one operation of arity $0$ - the unit element of multiplication. Other source of examples are <i>Boolean algebras</i> with two 2-ary operations $\wedge$ and $\vee$ or more generally <i>lattices</i>, <i>Heyting algebras</i>. Also <i>rings</i>, <i>modules</i>, <i>fields</i>, <i>vector spaces</i> and countless other structures. Universal algebra has a very general theory describing common concepts but also deals with very special cases of some of more esoteric algebras.</p>
<div class="definition" id="homomorphism">
<h5><span>Definition</span> Homomorphism</h5>
<div>
A <b>homomorphism</b> between two algebras $\underline{A}=(A, (f_i^{\underline{A}})_{i=1,\ldots,n})$ and $\underline{B}=(B, (f_i^{\underline{B}})_{i=1,\ldots,n})$ (of the same type) is a map $h:A\rightarrow B$ with the property that for every $i$: $$ h(f^{\underline{A}}_i(a_1,\ldots, a_{i_j})) = f^{\underline{B}}_i(h(a_1),\ldots,h(a_{i_j}))$$
</div>
</div>
<p>This means that <i>homomorphism preserve operations</i>. For example a homomorphism of monoids is a map that preserves the multiplication and unit. For boolean algebras, it means that a homomorphism preserves the $\vee$ (also called <i>join</i>) and $\wedge$ (usually called <i>meet</i>) operations, etc.</p>
<p>
It is an easy observations that homomorphism are closed under composition and since the identity map is always a homomorphism this leads to well defined categories, e.g. category of monoids, category of boolean algebras, category of rings, ...
</p>
<h2>Free algebras and Equational theories</h2>
<div class="definition">
<h5>Free Algebra</h5>
<div>
An algebra $\underline{A}=(A,(f_i^{\underline{A}})_{i=1,\dots,n})$ is <b>free</b> in a class of algebras $\mathcal{C}$ over a subset $S\subset A$ if for every $\underline{B}=(B,(f_i^{\underline{B}}){i=1,\dots,n})\in\mathcal{C}$ every map $S\rightarrow B$ uniquely extends to a <a href="#homomorphism">homomorphism</a> $\underline{A}\rightarrow \underline{B}$. The set $S$ is called the <b>set of generators</b>.
</div>
</div>
<p>
As you can see the definition of a free algebra requires a context, this interesting in its own!
<!-- TODO: EXPAND -->
There are free monoids in the class of all monoids and there are free commutative monoids in the class of commutative monoids (i.e. monoids in which $m\cdot n=n\cdot m$ for each elements $m,n$).
</p>
<p>Many theories allow free algebras. Let’s see some other examples:</p>
<ul>
<li>
The monoid of natural numbers $\mathbb{N}$ with addition and $0$ as its unit element is a free monoid generated by $\{1\}$. It is both free in the class of all monoids and in the class of commutative ones. The $n$-th cartesian product $\mathbb{N}^n$ is a free commutative monoid generated by the set $\{(1,0,\ldots,0),(0,1,0,\ldots,0),\ldots,(0,\ldots,0,1)\}$, but it's not a free monoid in the class of all monoids.
</li>
<li>
The additive group of integers $\mathbb{Z}$ is a free group with one generator, it is also free in the class of commutative groups. As in monoids: $\mathbb{Z}^n$ is a free commutative group with $n$ generators.
</li>
<li>
A <a href="https://en.wikipedia.org/wiki/Free_group">free group</a> with two generators can be pictured as the <a href="https://en.wikipedia.org/wiki/Cayley_graph">Cayley graph</a> (which is a fractal) (note that its first quarter is the free monoid with two generators).
</li>
<li>
Every <i>vector space</i> is free, since every vector space admits a basis.
</li>
<li>
In the class of <a href="https://en.wikipedia.org/wiki/Group_action#Definition">$G$-sets</a>, free $G$ sets are exactly all the cartesian products of $G^n$.
</li>
<li>
In the class of <a href="https://en.wikipedia.org/wiki/Ring_(mathematics)#Definition_and_illustration">rings</a>, polynomial rings with integer coefficients, usually denoted by: $\mathbb{Z}[X]$ or $\mathbb{Z}[X_1,\dots,X_n]$ for polynomials with many variables) are free (you likely have learned quite a lot about them in school, you just haven't been told the really interesting part ;)). This example was the motivation for terms, their algebra and term functions which we will discover next.
<p>
This is also true for <a href="https://en.wikipedia.org/wiki/Semiring">semi-rings</a>. You might have used this fact when using <a href="https://pursuit.purescript.org/packages/purescript-validation/4.0.0/docs/Data.Validation.Semiring">purescript validation</a> library. A free semiring generated by a type <code>a</code> has type <code>[[a]]</code>; for example <code>[[()]]</code> is isomorphic to $\mathbb{N}[X]$, since (please excuse mixing Haskell and mathematical notation): $$[[()]]\simeq[\mathbb{N}]\simeq\mathbb{N}[X]$$
</p>
</li>
</ul>
<p>
<i>Free algebras</i> play an essential role in a proof of beautiful and outstanding <a href="http://mathworld.wolfram.com/BirkhoffsTheorem.htmp">Birkhoff theorem</a>. It states that a class of algebras $\mathcal{C}$ is an <b>equational theory</b> if and only if the class is closed under cartesian products, <a href="#homomorphism">homomorphic</a> images and subalgebras. <i>Equational theories</i> are classes of algebras which satisfy a set of equations; examples includes: <i>semigroups</i>, <i>monoids</i>, <i>groups</i> or <i>boolean</i> or <i>Heyting algebras</i> but also <i>commutative (abelian) semigroups</i> / <i>monoids</i> / <i>groups</i>, and many other classical algebraic structures.
</p>
<p>
We need to be a little bit more precise language to speak about <i>equational theories</i> in the full generality of universal algebra, which we are going to introduce.
</p>
<h3>Terms, term functions and their algebra</h3>
<div class="definition" id="term">
<h5><span>Definition</span> Term</h5>
<div>
Let’s consider an algebra type $(f_i)_{i=1,\ldots,n}$.
Then the set of <strong>terms</strong> on a set $X$
(set of variables) is inductively defined as:
<ul>
<li id="term-1">
each $x\in X$ is a <b>term</b> (of arity $0$)
</li>
<li id="term-2">
each $f_i(x_1,\dots ,x_{j_i})$ is a <b>term</b> of arity $j_i$ for $x_1,\dots ,x_{j_i}\in X$
</li>
<li id="term-3">
if $g_1,\dots g_n$ are <i>terms</i> of arities $j_1$ to $j_n$ respectively, and $g$ is a <i>term</i> of arity $n$ then $g(g_1(x_{11},\dots,x_{1j_1}),\dots, g_n(x_{n1},\dots,x_{nj_n}))$ is a <b>term</b> of arity $j_1+\dots+j_n$ with $x_{kl}\in X$.
</li>
</ul>
<p>
We will denote the set of terms on $X$ by $\mathsf{T}^{(f_i)_{i=1,\dots,n}}(X)$ or simply $\mathsf{T}(X)$.
</p>
</div>
</div>
<p>
For example in groups: $x^{-1}\cdot x$, $x\cdot y$ and $1$ (the unit of the group) are terms. Terms are just abstract expressions that one can build using algebraic operations that are supported by the algebra type. Each term $t$ defines a <b>term function</b> on every algebra of the given type. In groups the following terms are distinct but they define equal term function: $x^{-1}\cdot x$ and $1$; on the other hand the two (distinct) terms $(x\cdot y)\cdot z$ and $x\cdot (y\cdot z)$ define equal term functions. The two terms $x\cdot y$ and $y\cdot x$ define distinct term functions (on non commutative groups or commutative monoids). Another example comes from boolean algebras (or more broadly lattice theory) where the two terms $x\wedge (y\vee z)$ and $(x\wedge y)\vee(x\wedge z)$ define equal term functions on Boolean algebras (or more generally <a href="https://en.wikipedia.org/wiki/Distributive_lattice">distributive lattices</a>). If $t$ is a term then the associated term function on an algebra $\underline{A}$ we let denote by $\tilde{t}^{\underline{A}}$. Term functions are natural to express equalities within a theory. Now we are ready to formally define equational classes of algebras.
</p>
<div class="definition">
<h5><span>Definition</span> Equational Theory</h5>
<div>
A class of algebras $\mathcal{C}$ is an <strong>equational
theory</strong> if and only if there exists a set of
pairs of terms $\mathbf{E}\subset\mathsf{T}(X)^2$ such
that the class consists exactly of algebras
$\underline{A}=(A,(f_i^{\underline{A}})_{i=1,\dots,n})$ for which the
following condition is satisfied: for each
pair of terms $(t, s)\in \mathbf{E}$ two corresponding
term functions $\tilde{t}^{\underline{A}}$ and
$\tilde{s}^{\underline{A}}$ are equal.
<div>
</div>
<p>
For example the class of monoids is an equational theory
for
$$\mathbf{E}=\bigl\{(1\cdot x,\, x),\; (x\cdot 1,\, x),\; \bigl((x\cdot y)\cdot z,\, x\cdot (y\cdot z)\bigr)\bigr\}$$
i.e. all the algebras with two operations: one of arity
0 (the unit) and one of arity 2 (the multiplication), such
that the $1$ is the unit for multiplication
$\cdot $ and multiplication is associative. The class
of commutative monoids is also an equational theory with
one additional equation $(x\cdot y,\, y\cdot x)$. Groups,
Boolean or Heyting algebras, lattices are also equational
theories.
</p>
<p>
Coming back to free algebras: it turns out that the set of
<em>terms</em> $\mathsf{T}^{(f_i)}(X)$ on a given set of
variables $X$ has an algebra structure of type
$(f_i)_{i=1,\dots,n}$: it is given by the inductive step in
the definition of <a href="#term-3">terms</a>:
if $t_i\in \mathsf{T}^{(f_i)}(X)$ for $i=1,\dots,j_i$ then
$$
f_j^{\underline{\mathsf{T}^{(f_i)}(X)}}(t_1,\ldots,t_{j_i}) := f_j(t_1,\ldots,t_{j_i})\in \mathsf{T}(X)
$$
Furthermore $\underline{\mathsf{T}^{(f_i)}(X)}$ is a free
algebra over $X$ in the class of all algebras of the given
type $(f_i)_{i=1,\dots,n}$. An extension of a map
$h:X\rightarrow\underline{A}=(A,(f_i^{\underline{A}})_{i=1,\ldots,n})$
can be build inductively following the definition of
<a href="#term">terms</a> and using the
<a href="#homomorphism">homomorphism property</a>:
$$
h(f_i(t_1,\ldots,f_{i_j})) := f_i^{\underline{A}}(h(t_1),\ldots,h(t_{i_j}))
$$
The map $h$ is indeed a homomorphism:
$$
\begin{array}{ll}
h\bigl(f_i^{\underline{\mathsf{T}(X)}}(t_1,\ldots,t_{i_j})\bigr) & = h(f_i(t_1,\ldots, t_{i_j}) \\\\
& = f_i^{\underline{A}}(h(t_1),\ldots, h(t_{i_j})) \\\\
\end{array}
$$
Note that the class of algebras of the same type is usually
very broad, but this is the first approximation to build
free algebras in an equational theory. This is just the
equational theory for the empty set $\mathbf{E}$.
</p>
<p>
Let’s see this on an example and let us consider algebras
of the same type as a monoid: with one nullary operation
(unit $1$ or <code>mempty</code> if you like) and one 2-ary
operation (multiplication / <code>mappend</code>). Let $X$
be a set of variables. Then $1$ is a valid term, and also
if $t_1$ and $t_2$ are terms on $X$ then also $t_1\cdot
t_2$ is a term, but also $t_1\cdot 1$ and $1\cdot t_2$ are
valid and distinct terms. $\mathsf{T}(X)$ resembles
a monoid but it isn't. It is not associative and the
unitality condition is not valid since $t\cdot 1\neq
t\neq 1\cdot t$ as terms. We still need a way to enforce
the laws. But note that if you have a map
$f:X\rightarrow M$ to a monoid $M$ which you'd like to extend to
a homomorphism $\mathsf{T}(X)\rightarrow M$ that preserves
$1$ (which is not the unit, yet) and multiplication (even
though it is not associative), you don’t have much choice:
$\mathsf{T}(X)\rightarrow M$: $t_1\cdot t_2$ must be mapped
to $f(t_1)\cdot f(t_2)\in M$.
</p>
<p>
We need a tool to enforce term equations. For that one can use
<div class="definition">
<h5><span>Definition</span> Congruence relation</h5>
<div>
Let $\underline{A}=(A,(f^A_i)_{i=1,\dots,n})$ be an
algebra and let $\sim$ be an <i>equivalence relation</i> on $A$,
i.e. a subset of $A\times A$ which is:
<ul>
<li><b>reflexive</b>: for each $a\in A$: $a\sim a$</li>
<li><b>symmetric</b>: for each $a,b\in A$: if $a\sim b$ then $b\sim a$</li>
<li><b>transitive</b>: for each $a,b,c\in A$: if
$a\sim b$ and $b\sim c$ then $a\sim c$</li>
</ul>
An equivalence relation is a <b>congruence relation</b>
if for all operations $f_i$ and any
$a_1,\dots,a_{i_j}\in A$ and $b_1,\dots,b_{i_j}\in
A$ the following implication holds:
$$
a_1\sim b_1,\dots,a_{i_j}\sim b_{i_j}\Rightarrow f_i^{\underline{A}}(a_1,\dots,a_{i_j})\sim f_i^{\underline{A}}(b_1,\dots,b_{i_j})
$$
</div>
</div>
If you have an equivalence relation $~$ on a set $A$ then
you can always construct the <a href="https://en.wikipedia.org/wiki/Equivalence_relation#Quotient_set">quotient set</a> $A/\sim$. An
equivalence class of $a\in A$ is the set $[a]:={x\in A:\;
x\sim a}$, then $A/\sim$ is just the set of equivalence
classes. However if you have a congruence then the
quotient $A/\sim$ carries algebra structure which turns the
quotient map $A\rightarrow A/\sim$ into a homomorphism.
</p>
<p>
Equivalence relations and congruences form complete
lattices (partial ordered which have all suprema and
minima, also infinite). If you have two equivalence
relations (congruences) then their intersection (as subsets
of $A^2$) is an equivalence relation (congruence).
</p>
<p>
The set of equations that defines the class of monoids
generates a
<a href="https://en.wikipedia.org/wiki/Congruence_relation">congruence relation</a>
on the term algebra $\underline{\mathsf{T}^{f_i}(X)}$ (i.e. an
<a href="https://en.wikipedia.org/wiki/Equivalence_relation">equivalence relation</a>
which is compatible with operations: $x_1\sim y_1$ and
$x_2\sim y_2$ then $(x_1\cdot y_1) \sim (x_2\cdot
y_2)$). One can define it as the smallest congruence
relation which contains the set $\mathbf{E}$.
<a href="https://en.wikipedia.org/wiki/Equivalence_relation">Equivalence relation</a>
on a set $A$ is just a subset of the cartesian
product $A\times A$ (which satisfy certain axioms), so it
all fits together! One can describe this congruence more
precisely, but we'll be happy with the fact that it
exists. To show that, first one need to observe that
intersection of congruences is a congruence, then the
smallest congruence containing the set $\mathbf{E}$ is an
intersection of all congruences that contain $\mathbf{E}$.
This intersection is non empty since the set $A\times A$ is
itself a congruence relation.
</p>
<p>
The key point now is that if we take the term algebra and
take a quotient by the smallest congruence that contains
all the pairs of terms which belong to the set $\mathbf{E}$
we will obtain a free algebra in the equational class
defined by $\mathbf{E}$. We will leave the proof to
a curious reader.
</p>
<h3>Free monoids</h3>
<p>
Let’s take a look on a free monoid that we can build this
way. First let us consider the free algebra
$\underline{\mathsf{T}(X)}$ for algebras of the same type as monoids
(which include non associative monoids, which unit does not
behave like a unit). And let $\sim$ be the smallest
relation (congruence) that enforces $\mathsf{T}(X)/\sim$ to
be a monoid.
</p>
<p>
Since monoids are associative every element in
$\underline{\mathsf{T}(X)}/\sim$ can be represented as $x_1\cdot(
x_2\cdot (x_3\cdot\ldots \cdot x_n))$ (where we group
brackets to the right). Multiplication of $x_1\cdot(
x_2\cdot (x_3\cdot\ldots \cdot x_n))$ and $y_1\cdot(
y_2\cdot (y_3\cdot\ldots \cdot y_m))$ is just
$x_1\cdot (x_2\cdot (x_3\cdot\ldots\cdot(x_n\cdot (y_1\cdot
(y_2\cdot (y_3\cdot\ldots\;\cdot y_m)\ldots)$. In Haskell if you’d
represent the set $X$ as a type
$a$ then the free monoid is just the list type $[a]$ with
multiplication: list concatenation and unit element: the
empty list. Just think of
<pre class="haskell"><code>-- A set with `n` elements corresponds
-- to a type with `n` constructors:
data X = X_1|⋯|X_n</code></pre>
<h2 id="monads">Free Monads</h2>
<p>
It turns out that monads in $\mathcal{Hask}$ are also an
equational theory. Just the terms are higher kinded:
$*\rightarrow*$ rather than $*$ as in monoids. The same
construction of a free algebra works in the land of monads,
but we need to look at them from another
perspective. Let us first take a mathematical definition
of view on monads.
</p>
<div class="definition" id="monad">
<h5><span>Definition</span> Monad</h5>
<div>
A <b>monad</b> is an (endo) <i>functor</i> <code>m</code>
with two
<a href="https://en.wikipedia.org/wiki/Natural_transformation"><i>natural transformations</i></a>:
<pre class="haskell"><code>class Monad m where
return :: a -> m a
join :: m(m a) -> m a</code></pre>
which is unital and associative, i.e. the following law holds:
<pre class="haskell"><code>-- | associativity
join . join == join . fmap join
-- | unitality
join . return = id = join . fmap return</code></pre>
<p>
These axioms are easier to understand as diagrams:
</p>
<p>
<div class="diagram">
<img src="/images/blog/monad-associativity.svg" style="display:block" class="center" width="350px" />
</div>
</p>
and
<p>
<div class="diagram">
<img src="/images/blog/monad-unitality.svg" style="display:block" class="center" width="350px" />
</div>
</p>
</div>
</div>
<p>
It is a basic lemma that this definition a monad is
equivalent to what we are used to in Haskell:
<pre class="haskell"><code>class Monad m where
return :: a -> m a
>>= :: m a -> (a -> m b) -> m b</code></pre>
<p>Having <code>join</code> one defines <code>>>=</code> as
<pre class="haskell"><code>ma >>= f = join $ f <$> ma</code></pre>
<p>and the other way, having <code>>>=</code> then
<pre class="haskell"><code>join = (>>= id)`</code></pre>
Not only these two constructions are reverse to each other,
but also they translate the monad laws correctly.
</p>
<h3>Monoids in monoidal categories</h3>
To define a monoid $M$ in the category $\mathcal{Set}$ (of
sets) one needs the product $M\times M$. Abstraction of this
structure leads to monoidal categories.
<div class="definition" id="monoidal-category">
<h5><span>Definition</span> Monoidal Category</h5>
<div>
Category $\mathcal{C}$ with a <a href="https://ncatlab.org/nlab/show/bifunctor">bifunctor</a>
$-\otimes-:\mathcal{C}\times\mathcal{C}\rightarrow\mathcal{C}$
is called <b>strict monoidal category</b> if `\otimes` is
<i>associative</i> and <i>unital</i>, i.e. for all
$a,b,c\in\mathcal{C}$ $(a\otimes b)\otimes c = a\otimes
(b\otimes c)$ and there exists a unit object $1$ such that
$1\otimes a=a=a\otimes 1$.
</div>
</div>
<p>
Most examples of monoidal categories are not strict but are
associative and unital up to a
<a href="https://en.wikipedia.org/wiki/Natural_transformation">natural transformation</a>.
Think of $(A\times B)\times C\simeq A\times(B\times C)$ in
$\mathcal{Set}$ (or any category with (finite) products,
like $\mathcal{Hask}$). Let me just stress out that since
$\otimes$ is a bifunctor, for any two maps
$f:\;a_1\rightarrow b_1$ and $g:\;a_2\rightarrow b_2$ we
have a map $f\otimes g: a_1\otimes a_2\rightarrow
b_1\otimes b_2$, and moreover it behaves nicely with
respect to composition: $(f_1\otimes g_1) \cdot (f_2\otimes
g_2) = (f_1\cdot f_2)\otimes(g_1\cdot g_2)$ for composable
pairs of arrows $f_1,\;f_2$ and $g_1,\;g_2$.
</p>
<p>
Now we can generalise a definition of a monoid to such categories:
</p>
<div class="definition" id="monoid-in-a-monoidal-category">
<h5><span>Definition</span> Monoid in a Monoidal Category</h5>
<div>
A <b>monoid</b> in a monoidal category $\mathcal{C}$ with monoidal
product $-\otimes-$ and a unit $1$ is an object $m$ with a pair of morphisms
$$
\mathrm{mappend}:\;m\otimes m\rightarrow m\quad\mathrm{mempty}:\;1\rightarrow m
$$
such that
<p>
<div class="diagram">
<img src="/images/blog/monoid-in-moncat-associativity.svg" style="display:block" class="center" width="350px" />
</div>
</p>
and
<p>
<div class="diagram">
<img src="/images/blog/monoid-in-moncat-unitality.svg" style="display:block" class="center" width="350px" />
</div>
</p>
</div>
</div>
<p>
The main point of this section is that these diagrams have
exactly the same shape as associativity and unitality for
<a href="#monad">monads</a>. Indeed, a monoid in the
category of endo-functors with functor composition as
a monoidal product $\otimes$ and unit the identity functor
is a monad. In category theory this category is strict
monoidal, if you try to type this in Haskell you will end
up with a non strict
<a href="https://ncatlab.org/nlab/show/monoidal+category">monoidal structure</a>,
where you will need to show
<a href="https://ncatlab.org/nlab/show/pentagon+identity">penthagon equation</a>.
</p>
<p>
These consideration suggest that we should be able to build
a free monad using our algebraic approach to free algebras. And
this is what we will follow in the next
<a href="free-monad">section</a>.
</p>
<h3 id="free-monad">Free monads in $\mathcal{Hask}$</h3>
<p>
Firstly, what should replace the set of generators $X$ in
$\mathsf{T}(X)/\sim$?
First we generalised from the category of sets $\mathcal{Set}$ to
a monoidal category $(\mathcal{C},\otimes, 1)$: its clear
that we just should pick an object of the category
$\mathcal{C}$. Now since our category is the category of
(endo) functors of $\mathcal{Hask}$ the set of generators
is just a functor. So let's pick a functor <code>f</code>.
</p>
<p>
To get a <i>free monad</i> we need to decypher
$\mathsf{T}(f)/\sim$ in the context of a monoid in
a monoidal category of endofunctors.
Note that here $\mathsf{T}(f)$ and $\mathsf{T}(f)/\sim$ are
functors! To simplify the notation, let
$\mathsf{Free}(f):=\mathsf{T}(f)/\sim$. So what is a term in this setting? It should be an expressions of
a Haskell's type:
$$
\begin{equation}
\begin{array}{c}
\bigl(\mathsf{Free}(f)\otimes\mathsf{Free}(f)\otimes\ldots\otimes \mathsf{Free}(f)\bigr)(a) \\\\
\quad\quad = \mathsf{Free}(f)\bigl(\mathsf{Free}(f)\bigl(\ldots (\mathsf{Free}(f)(a)\bigr)\ldots\bigr)
\end{array}
\end{equation}
$$
In our setup the monoidal product $-\otimes-$ is just the
functor composition, thus $\mathsf{Free}(f)(a)$ must be
a type which (Haskell's) terms are of Haskell's types:
<pre class="haskell"><code>a, f a, f (f a), f (f (f a)), ...</code></pre>
</p>
<p>
The monadic <code>join</code> will take something of
type $\mathsf{Free}(f)\;(\mathsf{Free}(f)\;(a))$, e.g. $f^n(b)=f\;(f\;(\dots f\;(b)\dots)$ (by abusing the notation $f^n$)
where $b$ has type
$f^m(a)=(f\;(f\;(\dots(f\;(a)\dots)$ and return something
of type $\mathsf{Free}(f)(a)$ and it should be quite clear
how to do that: just take the obvious element of type
$f^{n+m}(a)$. Altogether, this is a good trace of a monad,
so let us translate this into a concrete Haskell type:
<pre class="haskell"><code>data Free f a
= Return a
-- ^ the terms of type a
| Free (f (Free f a))
-- ^
-- recursive definition which embraces
-- `f a`, `f (f a)` and so on
instance Functor f => Functor (Free f) where
fmap f (Return a) = Return (f a)
fmap f (Free ff) = Free (fmap (fmap f) ff)</code></pre>
</p>
<p>
<code>Free f</code> is just a tree shaped by the functor
<code>f</code>. This type indeed embraces all the terms of
types: <code>a, f a, f (f a), ...</code> into a single
type. Now the monad instance:
<pre class="haskell"><code>instance Monad (Free f a) where
return = Return
join (Return ma) = ma
-- ^ stitch a tree of trees into a tree
join (Free fma) = Free $ join <$> fma
-- ^ recurs to the leaves</code></pre>
As you can see, takes a tree of trees and outputs a bigger
tree, that's what <code>join</code> does on the
<code>Return</code> constructor.
</p>
<p>
Before formulating the next result let's describe morphisms
between monads. Let <code>m</code> and <code>n</code> be
two monads then a <a href="https://en.wikipedia.org/wiki/Natural_transformation">natural transformation</a>
<code>f :: forall a. m a -> n a</code> is a <i>homomorphism of monads</i> iff the
following two conditions are satisfied:
<pre class="haskell"><code>f . return == return
join . f == f . fmap f . join</code></pre>
Note that this two conditions are satisfied iff
<code>f</code> is a monoid homomorphism in the category of (endo)functors
of $\mathcal{Hask}$.
<div class="proposition">
<h5><span>Proposition</span></h5>
Let <code>f</code> be a functor, then <code>Free
f</code> then there exists a morphism:
<pre class="haskell"><code>foldFree :: Functor f => (forall x. f x -> m x) -> (Free f a -> m a)</code></pre>
which restricts to an isomorphism of natural
transformations on the left hand side and monad
homomorphisms on the right hand side, and thus
<code>Free f</code> is rightly colled free monad..
</div>
<div class="proof">
<h5>Proof</h5>
Let start with a defintion of <code>foldFree</code>.
<pre class="haskell"><code>foldFree :: Functor f => (forall x. f x -> m x) -> (Free f a -> m a)
foldFree _ (Return a) = return a
foldFree f (Free ff) = join $ f $ foldFree f <$> ff</code></pre>
It's inverse is:
<pre class="haskell"><code>liftF :: Functor f => (forall x. Free f x -> m x) -> (f a -> m a)
liftF f fa = f $ Free $ Return <$> fa
</code></pre>
First let's check that <code>foldFree f</code> is a morhpism of monads:
<pre class="haskell"><code>foldFree f (Return a)
-- | by definition of (foldFree f)
= return a
foldFree f (join (Return a))
= foldFree f a
-- | by monad unitality axiom
= join $ return $ foldFree f $ a
-- | by definition of (foldFree f)
= join $ foldFree f (Return $ foldFree f a)
-- | by definition of functor instance of (Free f)
= join $ foldFree f $ fmap (foldFree f) $ Return a
foldFree f (join (Free ff)
-- | by definition of join for (Free f)
= foldFree f (Free $ fmap join $ ff)
-- | by definition of foldFree
= join $ f $ fmap (foldFree f) $ fmap join $ ff
= join $ f $ fmap (foldFree f . join) $ ff
-- | by induction hypothesis
= join $ f $ fmap (join . foldFree f . fmap (foldFree f)) $ ff
= join $ f $ fmap join $ fmap (foldFree f)
$ fmap (fmap (foldFree f)) $ ff
-- | f is natural transformation
= join $ fmap join $ f $ fmap (foldFree f)
$ fmap (fmap (foldFree f)) $ ff
-- | monad associativity
= join $ join $ f $ fmap (foldFree f)
$ fmap (fmap (foldFree f)) $ ff
-- | by definition of (foldFree f)
= join $ foldFree f $ Free
$ fmap (fmap (foldFree f)) $ ff
-- | by functor instance of (Free f)
= join $ foldFree f $ fmap (foldFree f) $ Free ff</code></pre>
And we have
<pre class="haskell"><code>foldFree . liftF :: (forall x. Free f x -> m x) -> (Free f a -> m a)
(foldFree . liftF $ f) (Return x)
-- ^ where f is a morphism of monads
= foldFree (liftF f) (Return x)
= return x
= f (Return x) -- since f is assumed to be a morphism of monads
(foldFree . liftF $ f) (Free ff)
-- ^ where f is a morphism of monads
= foldFree (liftF f) (Free ff)
= join $ liftF f $ fmap (foldFree (liftF f)) $ ff
-- | by induciton hypothesis
= join $ liftF f $ fmap f $ ff
-- | by definition of (liftF f)
= join $ f $ Free $ fmap Return $ fmap f $ ff
-- | by functor instance of (Free f)
= join $ f $ fmap f $ Free (Return ff)
-- | since f is a morphism of monads
= f $ join $ Free (Return ff)
= f $ Free ff</code></pre>
<pre class="haskell"><code>liftF . foldFree :: (forall x. f x -> m x) -> (f a -> m a)
(liftF . foldFree $ f) fa
-- ^ where f is a natural transformation
= liftF (foldFree f) $ fa
-- | by definition of liftF
= (foldFree f) $ Free $ fmap Return $ fa
-- | by definition of (foldFree f)
= join $ f $ fmap (foldFree f) $ fmap Return $ fa
= join $ f $ fmap (foldFree f . Return) $ fa
-- | by defintion of (foldFree f)
= join $ f $ fmap return $ fa
-- | since f is a natural transformation
= join $ fmap return $ f fa
-- | by monad unitality axiom
= f fa</code></pre>
</div>
</p>
<p>
<code>foldFree</code> corresponds to <code>foldMap</code> which is
defined in a very similar way
<pre class="haskell"><code>foldMap :: Monoid m => (a -> m) -> [a] -> m
foldMap _ [] = mempty
foldMap f (a : as) = mappend (f a) (foldMap f as)</code></pre>
Note that <code>foldMap</code> is an isomorphism onto
monoid homomorphisms with an inverse
<pre class="haskell"><code>g :: Monoid m => ([a] -> m) -> a -> m
g f a = f [a]</code></pre>
</p>
<p>
Furthermore, if we had polymorphic functions over monoidal
categories in our type system, <code>foldMap</code> and
<code>foldFree</code> would be specialisations of the
same function!
</p>
<h3>Some examples of free monads</h3>
Let us study some simple examples of free monads
<ul>
<li>
First let us consider the constant functor:
<pre class="haskell"><code>data Const a b = Const a</code></pre>
Then <code>Free (Const a)</code> is isomorphic to <code>Either a</code>
<pre class="haskell"></code>toEither :: Free (Const a) b -> Either a b
toEither (Return b) = Right b
toEither (Free (Const a)) = Left a
fromEither :: Either a b -> Free (Const a) b
fromEither (Right b) = Return b
fromEither (Left a) = Free (Const a)</code></pre>
Since <code>Either ()</code> is isomorphic with
<code>Maybe</code> also <code>Maybe</code> is a free
monad.
</li>
<li>
<code>Free Identity</code> is isomorphic to:
<pre class="haskell"><code>data Nat = Zero | Succ Nat
newtype Writer m a = Writer { runWriter :: (m, a) }
deriving Functor
toFree1 :: Free Identity a -> Writer Nat a
toFree1 (Return a) = Writer (Zero, a)
toFree1 (Free (Identity fa)) = case toFree1 fa of
Writer (n, a) -> (Succ n, a)
fromFree1 :: (Nat, a) -> Free Identity a
fromFree1 (Writer (Zero, a))
= Return a
fromFree1 (Writer (Succ n, a))
= Free (Identity (fromFree1 (Free1 n a)))</code></pre>
Note that <code>Nat</code> is the free monoid with one
generator (<code>Nat</code>$\simeq$<code>[()]</code>) in
the cateogry $\mathcal{Hask}$, and so is <code>Free
Identity</code> but in the monoidal category of
endofunctors of $\mathcal{Hask}$!
</li>
<li>
If you take a functor with two constructors
<pre class="haskell"><code>data F2 a = FX a | FY a
deriving Functor</code></pre>. Then we have
<pre class="haskell"><code>data S2 = SX | SY
toFree2 :: Free F2 a -> Writer [S2] a
toFree2 (Return a) = Writer ([], a)
toFree2 (Free (FX fa)) = case toM2 fa of
Writer (m, a) -> Writer (SX : m, a)
toM2 (Free (FY fa)) = case toM2 fa of
Writer (m, a) -> Writer (SY : m, a)
fromFree2 :: Writer [S2] a -> Free F2 a
fromFree2 (Writer ([], a))
= Return a
fromFree2 (Writer (SX : xs, a))
= Free (FX (fromM2 $ Writer (xs, a)))
fromFree2 (Writer (SY : xs, a))
= Free (FY (fromM2 $ Writer (xs, a)))</code></pre>
<code>toFree2</code> and <code>fromFree2</code> are isomorphisms.
I think you see the pattern: if you take a functor with $n$
constructors you will end up with a writer monad over
a free monoid with $n$ generators. You might ask if all
the monads are free then? The answer is no: take a non
free monoid <code>m</code> then the monad <code>Writer
m</code> is a non free monad. You can prove your self that
the writer monad <code>Writer m</code> is free if and only
if the monoid <code>m</code> is a free monoid in
$\mathcal{Hask}$.
</li>
</ul>
<h2>Final remarks</h2>
<p>
I hope I convinced you that monads are algebraic constructs and I hope
you'll find universal algebra approach useful. In many cases we
are dealing with algebraic structures which we require to
satisfy certain equations. Very often they fit into equational
theories, which have a very clear description and which allow
free objects. Freeness is the property that lets one easily
interpret the free object in any other object of the same type.
In the monad setting they are really useful when writing DSLs,
since you will be able to interpret it in any monad, like IO or
some pure monad.
</p>
<h2>References</h2>
<ul>
<li>
<a href="http://www.math.hawaii.edu/~ralph/Classes/619/univ-algebra.pdf">A Course in Universal Algebra</a>
by S. Burris, H.P. Sankappanavar
</li>
<li>
<a href="https://www.springer.com/gp/book/9780387774862">Universal Algebra</a>
by G. Grätzer
</li>
<li>
<a href="http://www.tac.mta.ca/tac/reprints/articles/22/tr22.pdf">Category Theory for Computing Science</a>
by M.Barr and C.Wells
</li>
</ul>
<br />
Note: This post originally appeared <a href="https://marcinszamotulski.me/posts/free-monads.html" target="_blank">here</a>.
<small>Artwork, <a href="https://creativecommons.org/licenses/by/4.0/" target="_blank"><i class="fa fa-creative-commons" aria-hidden="true"></i> <a href="http://www.beeple-crap.com">Mike Beeple</a>
</a></small></p></p></p></div></div>marcin.szamotulski@iohk.ioCardano smart contracts testnet IELE launches2018-07-30T00:00:00+00:002018-07-30T00:00:00+00:00https://iohk.io/blog/cardano-smart-contracts-testnet-iele-launches<p><img src="/images/blog/box.jpg" alt="Air Locked" class="fullwidth" /></p>
<p>Today we launch the second Cardano testnet, which is for the IELE virtual machine (VM) and follows our recent launch of the KEVM testnet. The technology is not only an important step on the Cardano roadmap, but also for the industry – in offering robust and reliable financial infrastructure. Developers now have the opportunity to explore the smart contracts technology that will be offered as part of Cardano, and to give us their feedback, which we look forward to receiving over the coming months.</p>
<h3 id="why-smart-contracts">Why smart contracts?</h3>
<p>In many business processes that involve the exchange of value (including money, property or shares) intermediaries are involved in checking that the terms of the agreements are complete and unambiguous as well as satisfied before the exchange can take place. These intermediaries add to the cost of a transaction. The technology of smart contracts (also known as self-executing contracts or blockchain contracts) has emerged as a way of addressing the need for this verification by reducing the time, third-party involvement, and cost of reliably executing an agreement.</p>
<p>Smart contracts are software programs that are immutably stored on the blockchain. They are executed by virtual machines and store their data in that same immutable infrastructure. Smart contracts offer great benefits to businesses looking to optimize their operations. Many industries – including automotive, supply chain, real estate and healthcare – are investing in research to understand how this technology can make them more competitive.</p>
<h3 id="what-smart-contracts-technology-is-currently-available">What smart contracts technology is currently available?</h3>
<p>There are a few players on the market that offer smart contract capabilities including Hyperledger, NEO and Ethereum. The technology is evolving to meet the market’s demand for platforms that are fast, secure, accurate and can be trusted. Many businesses have tried to deploy broad-scale applications on these platforms and have run into problems (DAO hack, Parity bug and POWH coin to name a few) with these evolving platforms. Despite widespread publicity the most serious bugs <a href="https://arxiv.org/pdf/1802.06038.pdf" title="Finding The Greedy, Prodigal, and Suicidal Contracts at Scale">continue to reappear</a> in smart contracts. There is a lot of room for innovation here and IOHK is working hard to become a leader in this technology.</p>
<h3 id="what-is-iele">What Is IELE?</h3>
<p>IELE (pronounced YELL-eh) is a virtual machine, with an attendant low-level language, designed to execute smart contracts on the Cardano blockchain. It has been developed by Runtime Verification in partnership with IOHK, which provided funding for the project. The word IELE refers to nymphs in Romanian mythology.</p>
<h3 id="how-does-iele-improve-on-smart-contracts-platforms">How does IELE improve on smart contracts platforms?</h3>
<p>IELE is designed to meet the evolving needs of the market for smart contracts by:</p>
<ul>
<li>
<p>Serving as a uniform, lower-level platform for translating and executing smart contracts from higher-level languages. It supports compilation from Solidity and many more languages are set to come.</p>
</li>
<li>
<p>Providing a uniform gas model, across all languages.</p>
</li>
<li>
<p>Making it easier to write secure smart contracts. IELE is ‘<a href="https://testnet.iohkdev.io/goguen/iele/about/semantics-based-compilation/" title="testnet.iohkdev.io">correct by construction</a>’ so many errors discovered after the fact (during code execution) in other VMs are not possible in IELE.</p>
</li>
<li>
<p>Using a register-based as opposed to stack-based architecture.</p>
</li>
</ul>
<h3 id="what-can-i-do-with-iele-that-i-could-not-do-before">What can I do with IELE that I could not do before?</h3>
<p>IELE contains two parts: a correct-by-construction VM designed using the K framework, and a correct by construction, Solidity-to-IELE compiler, also designed using the K framework. When you write your Solidity program and try to compile it using the Solidity-to-IELE compiler, it will catch many of the errors that previously would have been missed and that have caused many smart contracts to fail or be exploited.</p>
<p>In addition, as IELE development progresses, we plan to deliver ‘surface languages’ allowing programmers proficient in Javascript, Python and other languages to have an easy way to integrate smart contracts into their applications.</p>
<h3 id="what-do-i-do-next">What do I do next?</h3>
<p>The IELE language and its VM are completed. It is now in the process of being integrated into Cardano, which will provide a blockchain to store and retrieve data. While the integration is taking place, developers have the opportunity to use the IELE VM along with the Mallet and Remix tools to create and execute smart contracts on the <a href="https://testnet.iohkdev.io/" title="testnet.iohkdev.io">IOHK testnet site</a>.</p>
<p>You can also start getting a feel for the capabilities of both IELE and its VM – and even learn to write IELE code directly!</p>
<h5 id="join-the-conversation"><a href="https://forum.cardano.org/t/iohk-statement-cardano-smart-contracts-testnet-iele-launches/14432" title="IOHK Statement: Cardano smart contracts testnet IELE launches">Join the conversation!</a></h5>
<p> </p>
<h5 id="related-video-updates">Related video updates</h5>
<div class="iohk-video-list youtube" id="latest-videos" rel="PLnPTB0CuBOBz_kyRPBtfIF84ga1U9FJHb" count="20"></div>
<p><small>Artwork, <a href="https://creativecommons.org/licenses/by/4.0/" title="Creative Commons"><i class="fa fa-creative-commons" aria-hidden="true"></i></a> <a href="http://www.beeple-crap.com">Mike Beeple</a></small></p>gerard.moroney@iohk.ioCardano in focus at major international event2018-07-11T00:00:00+00:002018-07-11T00:00:00+00:00https://iohk.io/blog/cardano-in-focus-at-major-international-event<p><img src="/images/blog/compound-blur.jpg" alt="Compound Blur Cafe Latte" class="fullwidth" /></p>
<p>As a third-generation blockchain, Cardano incorporates state of-the-art technology that attracts the interest of computer scientists on the worldwide stage. In the past year, papers describing the consensus algorithm of Cardano have been presented at the leading cryptography conferences, and this month it was the turn of its smart contracts technology to be in the spotlight. </p>
<p><a href="http://fsl.cs.illinois.edu/index.php/Grigore_Rosu" title="Grigore Rosu, fsl.cs.illinois.edu">Grigore Rosu</a>, a professor in computer science at the University of Illinois at Urbana-Champaign, and his startup <a href="https://runtimeverification.com/" title="runtimeverification.com">Runtime Verification</a> have been working with IOHK since June 2017 to develop new technology based on formal semantics for Cardano, including a new virtual machine. He and his colleague Everett Hildenbrandt came to the UK last week to give presentations at the seventh Federated Logic Conference (<a href="http://www.floc2018.org/" title="FLoC">FLoC</a>), which this year is in the city of Oxford and runs from July 6-19 with about 2000 attendees. This major conference is held about every four years in various locations around the world, and under its umbrella stages together nine major international computer science conferences. These cover topics such as formal methods, logic and programming languages. Prominent figures from these worlds come to take part and keynote speeches this year are from <a href="https://www.csail.mit.edu/person/shafi-goldwasser" title="people.csail.mit.edu">Shafi Goldwasser</a> and <a href="https://people.csail.mit.edu/silvio/" title="people.csail.mit.edu">Silvio Micali</a>, the cryptographers and Turing prize winners, and mathematician George Gonthier.</p>
<figure class="alignright">
<img src="/images/blog/rv-grigore.jpg" alt="" width="250" height="" />
<figcaption>Grigore Rosu</figcaption>
</figure>
<p>On Saturday, Grigore had the distinction of giving his first FLoC invited talk, at the “Logical Frameworks and Meta-Languages: Theory and Practice” workshop and his talk was about the K framework. It was a technical presentation, going into detail about the logical formalism underlying K, and matching logic, a first-order logic variant for specifying and reasoning about structure by means of patterns and pattern matching.</p>
<p>This technology, developed by Grigore and his start-up Runtime Verification, has been developed over the past 15 years and is used in mission-critical software that cannot afford to fail. To this end, Runtime Verification has worked with companies including NASA, Boeing and Toyota and many others. His collaboration with IOHK began after he was contacted by CEO Charles Hoskinson, who had spotted that the software vulnerabilities that had resulted in a number of hacks on blockchains and the draining of hundreds of millions of dollars, could have been prevented using the formal methods techniques developed by Grigore and his team.</p>
<p>The K framework was used to formally model the semantics of the Ethereum Virtual Machine, and the knowledge gained from this process was employed to design IELE, the virtual machine for Cardano that will be released in a test format in a few weeks’ time. This is the first time this technology has been deployed within the blockchain industry. Importantly, K is a means to formally verify the code of smart contracts, so they can be automatically checked for the types of flaws that have led to catastrophic financial loss, and must be avoided at all costs.</p>
<p>Grigore said: “We designed IELE from scratch as a formal specification using K and generated the VM automatically from the specification. Therefore, there is nothing to prove about the VM in terms of correctness, because it is correct-by-construction.” He added: “We retrospectively analysed the EVM specification which we defined previously, and looked at all our attempts to verify smart contracts with it and then stepped back to think how should have a virtual machine been designed to overcome all those problems. We came up with IELE. This is an LLVM-like VM for the blockchain. For me as the designer of K, this is a hugely important milestone, and is the first time a real language has been defined in K and its implementation automatically generated.”</p>
<p>On Wednesday afternoon, Grigore will give a second invited talk at FLoC, in the International Conference on Formal Structures for Computation and Deduction (FSCD), about the importance of formal analysis and verification for blockchain and virtual machines. The presentation will be a little less technical than his first talk, and will cover Cardano, and how the tools developed by Runtime Verification allowed the automatic generation of a correct-by-contsruction virtual machine, IELE, from a formal specification.</p>
<figure class="alignright">
<img src="/images/blog/rv-gerard.jpg" alt="" width="250" height="" />
<figcaption>Runtime Verification and IOHK</figcaption>
</figure>
<p>And on Tuesday at the 31st IEEE Computer Security Foundations Symposium, Everett will present on how he and the team developed KEVM. Everett said: “KEVM is a formal specification of the Ethereum Virtual Machine (EVM) in K, which generates not only a VM but also a symbolic execution engine as well as a deductive program verifier for Ethereum smart contracts. There was a big need for such a complete formal EVM specification, because the previous semantics were either too informal or incomplete. Without a formal semantics of EVM the problem of verifying smart contract is meaningless.”</p>
<p>With three talks in total at FLoC in Oxford, it’s great to see this technology recognised at such a major international event. Security is of the utmost importance in designing and producing blockchains and we at IOHK put formal methods at the heart of our design approach. K has already been used to formalise C, Java, JavaScript, PHP, Python, Rust, and using these techniques we also plan to make IELE open to developers of many languages, not just Solidity. We will also offer our own languages, Plutus and Marlowe, which are currently in development with computer scientists including Philip Wadler and Simon Thompson. </p>
<p>Readers who would like to experiment with the <a href="https://iohk.io/blog/first-cardano-testnet-launches-for-smart-contracts/">KEVM testnet</a> can do so through our <a href="https://testnet.iohkdev.io/" title="Cardano Testnets, testnet.iohkdev.io">Cardano testnets website</a>. We are set to release the IELE on this same site in just a few weeks from now. Stay tuned for more updates.</p>
<p><small>Artwork, <a href="https://creativecommons.org/licenses/by/4.0/" title="Creative Commons"><i class="fa fa-creative-commons" aria-hidden="true"></i></a> <a href="http://www.beeple-crap.com">Mike Beeple</a></small></p>jeremy.wood@iohk.ioSelf Organisation in Coin Selection2018-07-03T00:00:00+00:002018-07-03T00:00:00+00:00https://iohk.io/blog/self-organisation-in-coin-selection<p><img src="/images/blog/self-organisation-in-coin-selection/self-organisation-in-coin-selection.jpg" alt="" class="fullwidth" /></p>
<p>The term “self organisation” refers to the emergence of complex behaviour (typically in biological systems) from simple rules and random fluctuations. In this blog post we will see how we can take advantage of self organisation to design a simple yet effective coin selection algorithm.</p>
<p>Coin selection is the process of selecting unspent outputs in a user’s wallet to satisfy a particular payment request (for a recap of UTxO style accounting, see section “Background: UTxO-style Accounting” of my <a href="/blog/semi-formal-development-the-cardano-wallet/" title="Semi-Formal Development: The Cardano Wallet">previous blog post</a>). As Jameson Lopp points out in his blog post <a href="https://medium.com/@lopp/the-challenges-of-optimizing-unspent-output-selection-a3e5d05d13ef" title="The Challenges of Optimizing Unspent Output Selection, medium.com">The Challenges of Optimizing Unspent Output Selection</a>, coin selection is thorny problem. Moreover, there is a surprising lack of academic publications on the topic; indeed, the only scientific study of coin selection appears to be Mark Erhardt’s masters thesis <a href="http://murch.one/wp-content/uploads/2016/11/erhardt2016coinselection.pdf" title="erhardt2016coinselection.pdf, murch.one">An Evaluation of Coin Selection Strategies</a>.</p>
<p>Note: by a slight abuse of nomenclature, throughout this blog post we will refer to a user’s set of unspent outputs as that user’s UTxO.</p>
<h2 id="dust">Dust</h2>
<p>An obvious strategy that many coin selection algorithms use in some form or other is “try to get as close to the requested value as possible”. The problem with such an approach is that it tends to create a lot of dust: small unspent outputs that remain unused in the user’s wallet because they’re not particularly useful. For example, consider the “largest first” algorithm: a simple algorithm which considers all unspent outputs of the wallet in order of size, adding them to a running total until it has covered the requested amount. Here’s an animation of the effect of this algorithm:</p>
<figure>
<img src="/images/blog/self-organisation-in-coin-selection/normal-3to1-largest.gif" alt="Figure 1" />
<img src="/images/blog/self-organisation-in-coin-selection/normal-3to1-largest.png" alt="Figure 1-1" />
<figcaption><strong>Figure 1.</strong> Simulation of largest-first coin selection. Main histogram shows UTxO entries; inset graph shows UTxO balance in blue and UTxO size in red, histogram top-right shows number of inputs per transaction, graph bottom right shows the change:payment ratio (more on that below). Graph at the bottom shows the distribution of deposits (blue, left axis) versus payments (red, right axis). In this case, both are normally distributed with a mean of 1000 and 3000 respectively, and we have a deposit:payment ratio of 3:1; modelling a situation where we have frequent smaller deposits, and less frequent but larger payments (withdrawals). The wallet starts with an initial balance of 1M.</figcaption>
</figure>
<p><br /></p>
<p>There are various things to see in this animation, but for now we want to focus on the UTxO histogram and its size. Note that as time passes, the size of the UTxO increases and increases, up to about 60k entries after about 1M cycles (with 3 deposits and 1 payment per cycle). A wallet with 60k entries is huge, and looking at the UTxO histogram we can see why this happens: virtually all of these entries are dust. We get more and more small outputs, and those small outputs are getting smaller and smaller.</p>
<h2 id="cleaning-up">Cleaning up</h2>
<p>Erhardt makes the following very astute observation:</p>
<blockquote>If 90% of the UTxO is dust, then if we pick an unspent output randomly, we have a 90% change of picking a dust output.
</blockquote>
<p>He concludes that this means that a coin selection algorithm that simply picks unspent outputs at random might be pretty effective; in particular, effective at collecting dust. Indeed, it is. Consider the following simulation:</p>
<figure>
<img src="/images/blog/self-organisation-in-coin-selection/normal-3to1-largeThenRandom.gif" alt="Figure 2" />
<img src="/images/blog/self-organisation-in-coin-selection/normal-3to1-largeThenRandom.png" alt="Figure 2-2" />
<figcaption><strong>Figure 2.</strong> Same distribution and ratio as in Figure 1; we run the largest-first algorithm for 1M cycles, and then random coin selection for another 150k cycles.
</figcaption>
</figure>
<p><br /></p>
<p>Notice quite how rapidly the random coin selection reduces the size of the UTxO once it kicks in. If you watch the inputs-per-transaction histogram, you can see that when the random input selection takes over, it first creates a bunch of transactions with 10 inputs (we limited transaction size to 10 for this simulation), rapidly collecting dust. Once the dust is gone, the number of inputs shrinks to about 3 or 4, which makes perfect sense given the 3:1 ratio of deposits and withdrawals.</p>
<p>We will restate Erhardt’s observation as our first self organisation principle:</p>
<blockquote>
<strong>Self organisation principle 1.</strong> Random selection has a high priobability of picking dust outputs precisely when there is a lot of dust in the UTxO.
</blockquote>
<p>It provides a very promising starting point for an effective coin selection algorithm, but there are some improvements we can make.</p>
<h2 id="active-utxo-management">Active UTxO management</h2>
<p>Consider the following simulation of a pure “select randomly until we reach the target value” coin selection algorithm:</p>
<figure>
<img src="/images/blog/self-organisation-in-coin-selection/normal-1to1-randomOff.gif" alt="Figure 3" />
<img src="/images/blog/self-organisation-in-coin-selection/normal-1to1-randomOff.png" alt="Figure 3-2" />
<figcaption><strong>Figure 3.</strong> Random-until-value-reached, for a 1:1 ratio of deposits and withdrawals, both drawn from a normal distribution with mean 1000.
</figcaption>
</figure>
<p><br /></p>
<p>The first observation is that this algorithm is doing much better than the largest-first policy in terms of the size of the UTxO, which is about 2 orders of magnitude smaller: a dramatic improvement. However, if we look at the UTxO histogram, we can see that there is room for improvement: although this algorithm is good at collecting dust, it’s also still generating quite a bit of dust. The UTxO histogram has two peaks. The first one is approximately normally distributed around 1000, which are the deposits that are being made. The second one is near 0, which are all the dust outputs that are being created.</p>
<p>This brings us to the topic of active UTxO management. In an ideal case, coin selection algorithms should, over time, create a UTxO that has “useful” outputs; that is, outputs that allow us to process future payments with a minimum number of inputs. We can take advantage of self organisation again:</p>
<blockquote>
<strong>Self organisation principle 2.</strong> If for each payment request for value x we create a change output roughly of the same value x, then we will end up with a lot of change outputs in our UTxO of size x precisely when we have a lot of payment requests of size x.
</blockquote>
<p>We will consider some details of how to achieve this in the next section. For now see what the effect of this is on the UTxO:</p>
<figure>
<img src="/images/blog/self-organisation-in-coin-selection/normal-1to1-randomOn.gif" alt="Figure 4" />
<img src="/images/blog/self-organisation-in-coin-selection/normal-1to1-randomOn.png" alt="Figure 4-2" />
<figcaption><strong>Figure 4.</strong> Same deposits and withdrawals as in Figure 3, but now using the "pick randomly until we have a change output roughly equal to the payment" algorithm.
</figcaption>
</figure>
<p><br /></p>
<p>The graph at the bottom right, which we’ve ignored so far, records the change:payment ratio. A value near zero means a very small change output (i.e., dust); a very high value would be the result of using a huge UTxO entry for a much smaller payment. A value around 1 is perfect, and means that we are generating change outputs of equal value as the payments.</p>
<p>Note that the UTxO now follows precisely the distribution of payment requests, and we’re not generating dust anymore. One advantage of this is that because we have no dust, the average number of inputs per transaction can be lower than in the basic algorithm.</p>
<p>Just to illustrate this again, here is the result of the algorithm but now with a 3:1 ratio of deposits and withdrawals:</p>
<figure>
<img src="/images/blog/self-organisation-in-coin-selection/normal-3to1-randomOn.gif" alt="Figure 5" />
<img src="/images/blog/self-organisation-in-coin-selection/normal-3to1-randomOn.png" alt="Figure 5-2" />
<figcaption><strong>Figure 5.</strong> Same algorithm as in Figure 4, but now with 3:1 deposits:payments (i.e., many small deposits, fewer but larger payments).
</figcaption>
</figure>
<p><br /></p>
<p>We have two bumps now: one normally distributed around 1000, corresponding to the the deposits, and one normally distributed around 3000, corresponding to the payment requests that are being made. As before, the median change:payment ratio is a satisfying round 1.0.</p>
<h2 id="the-random-improve-algorithm">The <code>Random-Improve</code> algorithm</h2>
<p>We are now ready to present the coin selection algorithm we propose. The basic idea is simple: we will randomly pick UTxO entries until we have reached the required value, and then continue randomly picking UTxO entries to try and reach a total value such that the the change value is roughly equal to the payment.</p>
<p>This presents a dilemma though. Suppose we have already covered the minimum value required, and we’re trying to improve the change output. We pick an output from the UTxO, and it turns out to be huge. What do we do? One option is to discard it and continue searching, but this would result in coin selection frequently traversing the entire UTxO, resulting in poor performance.</p>
<p>Fortunately, self organisation comes to the rescue again. We can set an upper bound on the size of the change output we still consider acceptable (we will set it to twice the payment value). Then we take advantage of the following property.</p>
<blockquote><strong>Self organisation principle 3.</strong> Searching the UTxO for additional entries to improve our change output is only useful if the UTxO contains entries that are sufficiently small enough. But precisely when the UTxO contains many small entries, it is less likely that a randomly chosen UTxO entry will push the total above the upper bound we set.</blockquote>
<p>In other words, our answer to “what do we do when we happen to pick a huge UTxO entry?” is “we stop trying to improve our selection”. We can now describe the algorithm:</p>
<ol>
<li>Randomly select outputs from the UTxO until the payment value is covered.
(In the rare case that this fails because the maximum number of transaction inputs has been exceeded, fall-back on the largest-first algorithm for this step.)</li>
<li>Randomly select outputs from the UTxO, considering for each output if that output is an improvement. If it is, add it to the transaction, and keep going.
An output is considered an improvement when:
<ul>
<li>It doesn’t exceed the specified upper limit</li>
<li>Adding the new output gets us closer to the ideal change value</li>
<li>It doesn’t exceed the maximum number of transaction inputs.</li>
</ul>
</li>
</ol>
<figure>
<figcaption><strong>Figure 6.</strong> The Random-Improve algorithm. Side note for point (2a): we use twice the value of the payment as the upper limit. Side note for point (2b): it might be that without the new output we are slightly below the ideal value, and with the new output we are slightly above; that is fine, as long as the absolute distance decreases.</figcaption>
</figure>
<h2 id="evaluation">Evaluation</h2>
<p>The algorithm from Figure 6 is deceptively simple. Do the self organisation principles we isolated really mean that order will emerge from chaos? Simulations suggest, yes, it does. We already mentioned how random input selection does a great job at cleaning up dust in Figure 2; what we didn’t emphasize in that section is that the algorithm we simulated there is actually our Random-Improve algorithm. Notice how the median change:payment ratio is initially very low (indicative of a coin selection algorithm that is generating a lot of dust outputs), but climbs rapidly back to 1 as soon as Random-Improve kicks in. We already observed that it does indeed do an excellent job at cleaning up the dust, quickly reducing the size of the UTxO. The simulations in Figures 4 and 5 are also the result of the Random-Improve algorithm.</p>
<p>That said, of course the long term effects of a coin selection algorithm can depend strongly on the nature of the distribution of deposits and payments. It is therefore important that we evaluate the algorithm against a number of different distributions.</p>
<h3 id="normal-distribution-101-depositpayment-ratio">Normal distribution, 10:1 deposit:payment ratio</h3>
<p>We already evaluated <code>Random-Improve</code> against normally distributed payments and deposits with a 1:1 ratio and a 3:1 ratio; perhaps more typical for exchange nodes might be even higher ratios. Here is a 10:1 ratio:</p>
<figure>
<img src="/images/blog/self-organisation-in-coin-selection/normal-10to1-randomOn.gif" alt="Figure 7" />
<img src="/images/blog/self-organisation-in-coin-selection/normal-10to1-randomOn.png" alt="Figure 7-2" />
<figcaption><strong>Figure 7.</strong> Simulation of largest-first coin selection. Main histogram shows UTxO entries; inset graph shows UTxO balance in blue and UTxO size in red, histogram top-right shows number of inputs per transaction, graph bottom right shows the change:payment ratio (more on that below). Graph at the bottom shows the distribution of deposits (blue, left axis) versus payments (red, right axis). In this case, both are normally distributed with a mean of 1000 and 3000 respectively, and we have a deposit:payment ratio of 3:1; modelling a situation where we have frequent smaller deposits, and less frequent but larger payments (withdrawals). The wallet starts with an initial balance of 1M.</figcaption>
</figure>
<p><br /></p>
<p>We see a very similar picture as we did in Figure 5. Since the deposits and payments are randomly drawn (from normal distributions), the UTxO balance naturally fluctuates up and down. What is satisfying to see however is that the size of the UTxO tracks the balance rather precisely; this is about as good as we can hope for. Notice also that the change:payment ratio is a nice round 1, and the average number of transaction inputs covers around 10 or 11, which is what we’d expect for a 10:1 ratio of deposits:payments.</p>
<h3 id="exponential-distribution-11-depositpayment-ratio">Exponential distribution, 1:1 deposit:payment ratio</h3>
<p>What if the payments and deposits are not normally distributed? Here is Random-Improve on exponentially distributed inputs:</p>
<figure>
<img src="/images/blog/self-organisation-in-coin-selection/erlang1-1to1-randomOn.gif" alt="Figure 8" />
<img src="/images/blog/self-organisation-in-coin-selection/erlang1-1to1-randomOn.png" alt="Figure 8-2" />
<figcaption><strong>Figure 8.</strong> Random-Improve, 1:1 deposit:payment ratio, deposits and payments both drawn from an exponential distribution with scale 1000.</figcaption>
</figure>
<p><br /></p>
<p>In an exponential distribution we have a lot of values near 0; for such values it will be hard to achieve a “good” change output, as we are likely to overshoot the range. Partly due to this reason the algorithm isn’t quite achieving a 1.0 change:payment ratio, but at 1.5 it is still generating useful change outputs. Furthermore, we can see that the size of the UTxO tracks the UTxO balance nicely, and the average number of transaction inputs is low, with roughly 53% having just one input.</p>
<p>Moreover, when we increase the deposit:payment ratio to 3:1 and then 10:0, the change:payment ratio drops to about 1.1 and then back to 1.0 (graphs omitted).</p>
<h3 id="erlang">Erlang</h3>
<p>The exponential distribution results in many very small deposits and payments. The algorithm does better on slightly more realistic distributions such as the Erlang-k distributions (for k > 1). Here we show the animation for the 3:1 deposit:payment ratio using the Erlang-3 distribution; the results for other ratios (including 1:1) and other values of k (we additionally simulated for k = 2 and k = 10) are similar.</p>
<figure>
<img src="/images/blog/self-organisation-in-coin-selection/erlang3-3to1-randomOn.gif" alt="Figure 9" />
<img src="/images/blog/self-organisation-in-coin-selection/erlang3-3to1-randomOn.png" alt="Figure 9-2" />
<figcaption><strong>Figure 9.</strong> Random-Improve, 3:1 deposit:payment ratio, deposits drawn from an Erlang-3 distribution with scale 1000 and payments drawn from Erlang-3 distributio with scale 3000.</figcaption>
</figure>
<h3 id="more-payments-than-deposits">More payments than deposits</h3>
<p>We have been focusing on the case where we have more deposits and fewer (but larger) payments. What happens if the ratio is reversed?</p>
<figure>
<img src="/images/blog/self-organisation-in-coin-selection/normal-1to10-randomOn.gif" alt="Figure 10" />
<img src="/images/blog/self-organisation-in-coin-selection/normal-1to10-randomOn.png" alt="Figure 10-2" />
<figcaption><strong>Figure 10.</strong> Random-Improve, 1:10 deposit:payment ratio, deposits and payments drawn from a normal distribution with mean 10k and 1k, respectively. 1M cycles.</figcaption>
</figure>
<p><br /></p>
<p>In this case we are unable to achieve that perfect 1.0 change:payment ratio, but this is expected: when we have large deposits, then we frequently have no choice but to use those, leading to large change outputs. We can see this more clearly when we slow things right down, and remove any source of randomness; here is the same 1:10 ratio again, but now only the first 100 cycles, and all deposits exactly 10k and all payments exactly 1k:</p>
<figure>
<img src="/images/blog/self-organisation-in-coin-selection/constant-1to10-randomOn.gif" alt="Figure 11" />
<figcaption><strong>Figure 11.</strong> Random-Improve, 1:10 deposit:payment ratio, all deposits exactly 10k, all payments exactly 1k (no randomness). First 100 cycles only.</figcaption>
</figure>
<p><br /></p>
<p>We can see the large value being deposited, and then shifting to the left in the histogram as it is getting used for deposits, each time decreasing that large output by 1k. Indeed, this takes 10 slots on average, which makes sense given the 10:1 ratio; moreover, the average value of the “large output” in such a 10-slot cycle is 5k, explaining why we are getting 5.0 change:payment ratio.</p>
<p>The algorithm however is not creating dust outputs; the 1k change outputs it is generating are getting used, and the size of the UTxO is perfectly stable. Indeed, back in Figure 12 we can see that the size of the UTxO tracks the balance perfectly; moreover, the vast majority of transactions only use a single input, which is what we’d expect for a 10:0 deposit:payment ratio.</p>
<h3 id="real-data">Real data</h3>
<h4 id="moneypotcom">MoneyPot.com</h4>
<p>Ideally, of course, we run the simulation against real event streams from existing wallets. Unfortunately, such data is hard to come by. Erhardt was able to find one such dataset, provided by <a href="https://www.moneypot.com/" title="MoneyPot.com">MoneyPot.com</a>. When we run our algorithm on this dataset we get</p>
<figure>
<img src="/images/blog/self-organisation-in-coin-selection/MoneyPot-randomOn.gif" alt="Figure 12" />
<img src="/images/blog/self-organisation-in-coin-selection/MoneyPot-randomOn.png" alt="Figure 12-2" />
<figcaption><strong>Figure 12.</strong> Random-Improve, using the MoneyPot data set. There is a rougly 2:1 deposit:payment ratio. Values have been scaled. NOTE: log scale on the x-axis.</figcaption>
</figure>
<p><br /></p>
<p>A few observations are in order here. First, there are quite a few deposits and payments close to 0, just like in an exponential distribution. Moreover, although we have many values close to 0, we also have some huge outliers; the payments are closely clustered together, but there is a 10xE9 difference between the smallest and largest deposit, and moreover a 10xE5 difference between the largest deposit and the largest payment. It is therefore not surprising that we end up with a relatively large change:payment ratio. Nonetheless, the algorithm is behaving well, with the size of the UTxO tracking the balance nicely, with an average UTxO size of 130 entries. The average number of outputs is 3.03, with 50% of transactions using just one input, and 90% using 6 or fewer.</p>
<h4 id="cardano-exchange">Cardano Exchange</h4>
<p>One of the large Cardano exchange nodes has also helped us with some anonymised data (deposits and payments), similar in nature to the MoneyPot dataset albeit significantly larger. Coming from an exchange node, however, this dataset is very much skewed towards deposits, with a deposit:payment ratio of roughly 30:1. Under these circumstances, of course, coin selection alone cannot keep the UTxO size small.</p>
<figure>
<img src="/images/blog/self-organisation-in-coin-selection/CardanoExchange-randomOn.gif" alt="Figure 13" />
<img src="/images/blog/self-organisation-in-coin-selection/CardanoExchange-randomOn.png" alt="Figure 13-2" />
<figcaption><strong>Figure 13.</strong> Random-Improve, using data set from a large Cardano exchange. There is a rougly 30:1 deposit:payment ratio. Values have been scaled. NOTE: log scale on the x-axis.</figcaption>
</figure>
<h2 id="conclusions">Conclusions</h2>
<p>The choice of coin selection algorithm has far reaching consequences on the long term behaviour of a cryptocurrency wallet. To a large extent the coin selection algorithm determines, over time, the shape of the UTxO. Moreover, the performance of the algorithm can be of crucial importance to high-traffic wallets such as exchange nodes.</p>
<p>In his thesis, Erhardt proposes “Branch and Bound” as his preferred coin selection algorithm. Branch and Bound in essence is a limited backtracking algorithm that tries to find an exact match, so that no change output needs to be generated at all. When the backtracking cannot find an exact match within its bounds, the algorithm then falls back on random selection. It does not, however, attempt our “improvement” step, and instead just attempts to reach a minimum but fixed change size, to avoid generating dust. It is hard to compare the two algorithms directly, but on the MoneyPot dataset at least the results are comparable; Erhardt ends up with a slightly smaller average UTxO (109 versus our 130), and a slightly smaller average number of inputs (2.7 versus our 3.0). In principle we could modify our Random-Improve algorithm to start with bounded backtracking to find an exact match, just like Erhardt does; we have not done this however because it adds complexity to the algorithm and reduces performance. Erhardt reports that his algorithm is able to find exact matches in 30% of the time. This is very high, and at least partly explains why his UTxO and average number of change outputs is lower; in the Cardano blockchain, we would not expect that there exist exact matches anywhere near that often (never mind finding them).</p>
<p>Instead our proposed Random-Improve does no search at all, instead purely relying on self organisation principles, the first of which was stated by Erhardt, and the other two we identified as part of this research. Although in the absence of more real data it is hard to evaluate any coin selection algorithm, we have shown that the algorithm performs well across a large variety of different distributions and deposit:payment ratios. Moreover it is straight-forward to implement and has high performance.</p>
<p>One improvement we may wish to consider is that when there are very large deposits, we could occassionally issue a “reorganisation transaction” that splits those large deposits into smaller chunks. This would bring the change:payment ratio down, which would improve the evolution of the UTxO over time and is beneficial also for other, more technical reasons (it reduces the need for what we call “dependent transactions” in the <a href="https://cardanodocs.com/technical/formal-specification-for-a-cardano-wallet/" title="Formal specification for a Cardano wallet, cardanodocs.com">wallet specification</a>). Such reorganisation is largely orthogonal to this algorithm, however, and can be implemented independently.</p>edsko@well-typed.comThe term “self organisation” refers to the emergence of complex behaviour (typically in biological systems) from simple rules and random fluctuations. In this blog post we will see how we can take advantage of self organisation to design a simple yet effective coin selection algorithm. Coin selection is the process of selecting unspent outputs in a user’s wallet to satisfy a particular payment request (for a recap of UTxO style accounting, see section “Background: UTxO-style Accounting” of my previous blog post). As Jameson Lopp points out in his blog post The Challenges of Optimizing Unspent Output Selection, coin selection is thorny problem. Moreover, there is a surprising lack of academic publications on the topic; indeed, the only scientific study of coin selection appears to be Mark Erhardt’s masters thesis An Evaluation of Coin Selection Strategies. Note: by a slight abuse of nomenclature, throughout this blog post we will refer to a user’s set of unspent outputs as that user’s UTxO. Dust An obvious strategy that many coin selection algorithms use in some form or other is “try to get as close to the requested value as possible”. The problem with such an approach is that it tends to create a lot of dust: small unspent outputs that remain unused in the user’s wallet because they’re not particularly useful. For example, consider the “largest first” algorithm: a simple algorithm which considers all unspent outputs of the wallet in order of size, adding them to a running total until it has covered the requested amount. Here’s an animation of the effect of this algorithm: Figure 1. Simulation of largest-first coin selection. Main histogram shows UTxO entries; inset graph shows UTxO balance in blue and UTxO size in red, histogram top-right shows number of inputs per transaction, graph bottom right shows the change:payment ratio (more on that below). Graph at the bottom shows the distribution of deposits (blue, left axis) versus payments (red, right axis). In this case, both are normally distributed with a mean of 1000 and 3000 respectively, and we have a deposit:payment ratio of 3:1; modelling a situation where we have frequent smaller deposits, and less frequent but larger payments (withdrawals). The wallet starts with an initial balance of 1M. There are various things to see in this animation, but for now we want to focus on the UTxO histogram and its size. Note that as time passes, the size of the UTxO increases and increases, up to about 60k entries after about 1M cycles (with 3 deposits and 1 payment per cycle). A wallet with 60k entries is huge, and looking at the UTxO histogram we can see why this happens: virtually all of these entries are dust. We get more and more small outputs, and those small outputs are getting smaller and smaller. Cleaning up Erhardt makes the following very astute observation: If 90% of the UTxO is dust, then if we pick an unspent output randomly, we have a 90% change of picking a dust output. He concludes that this means that a coin selection algorithm that simply picks unspent outputs at random might be pretty effective; in particular, effective at collecting dust. Indeed, it is. Consider the following simulation: Figure 2. Same distribution and ratio as in Figure 1; we run the largest-first algorithm for 1M cycles, and then random coin selection for another 150k cycles. Notice quite how rapidly the random coin selection reduces the size of the UTxO once it kicks in. If you watch the inputs-per-transaction histogram, you can see that when the random input selection takes over, it first creates a bunch of transactions with 10 inputs (we limited transaction size to 10 for this simulation), rapidly collecting dust. Once the dust is gone, the number of inputs shrinks to about 3 or 4, which makes perfect sense given the 3:1 ratio of deposits and withdrawals. We will restate Erhardt’s observation as our first self organisation principle: Self organisation principle 1. Random selection has a high priobability of picking dust outputs precisely when there is a lot of dust in the UTxO. It provides a very promising starting point for an effective coin selection algorithm, but there are some improvements we can make. Active UTxO management Consider the following simulation of a pure “select randomly until we reach the target value” coin selection algorithm: Figure 3. Random-until-value-reached, for a 1:1 ratio of deposits and withdrawals, both drawn from a normal distribution with mean 1000. The first observation is that this algorithm is doing much better than the largest-first policy in terms of the size of the UTxO, which is about 2 orders of magnitude smaller: a dramatic improvement. However, if we look at the UTxO histogram, we can see that there is room for improvement: although this algorithm is good at collecting dust, it’s also still generating quite a bit of dust. The UTxO histogram has two peaks. The first one is approximately normally distributed around 1000, which are the deposits that are being made. The second one is near 0, which are all the dust outputs that are being created. This brings us to the topic of active UTxO management. In an ideal case, coin selection algorithms should, over time, create a UTxO that has “useful” outputs; that is, outputs that allow us to process future payments with a minimum number of inputs. We can take advantage of self organisation again: Self organisation principle 2. If for each payment request for value x we create a change output roughly of the same value x, then we will end up with a lot of change outputs in our UTxO of size x precisely when we have a lot of payment requests of size x. We will consider some details of how to achieve this in the next section. For now see what the effect of this is on the UTxO: Figure 4. Same deposits and withdrawals as in Figure 3, but now using the "pick randomly until we have a change output roughly equal to the payment" algorithm. The graph at the bottom right, which we’ve ignored so far, records the change:payment ratio. A value near zero means a very small change output (i.e., dust); a very high value would be the result of using a huge UTxO entry for a much smaller payment. A value around 1 is perfect, and means that we are generating change outputs of equal value as the payments. Note that the UTxO now follows precisely the distribution of payment requests, and we’re not generating dust anymore. One advantage of this is that because we have no dust, the average number of inputs per transaction can be lower than in the basic algorithm. Just to illustrate this again, here is the result of the algorithm but now with a 3:1 ratio of deposits and withdrawals: Figure 5. Same algorithm as in Figure 4, but now with 3:1 deposits:payments (i.e., many small deposits, fewer but larger payments). We have two bumps now: one normally distributed around 1000, corresponding to the the deposits, and one normally distributed around 3000, corresponding to the payment requests that are being made. As before, the median change:payment ratio is a satisfying round 1.0. The Random-Improve algorithm We are now ready to present the coin selection algorithm we propose. The basic idea is simple: we will randomly pick UTxO entries until we have reached the required value, and then continue randomly picking UTxO entries to try and reach a total value such that the the change value is roughly equal to the payment. This presents a dilemma though. Suppose we have already covered the minimum value required, and we’re trying to improve the change output. We pick an output from the UTxO, and it turns out to be huge. What do we do? One option is to discard it and continue searching, but this would result in coin selection frequently traversing the entire UTxO, resulting in poor performance. Fortunately, self organisation comes to the rescue again. We can set an upper bound on the size of the change output we still consider acceptable (we will set it to twice the payment value). Then we take advantage of the following property. Self organisation principle 3. Searching the UTxO for additional entries to improve our change output is only useful if the UTxO contains entries that are sufficiently small enough. But precisely when the UTxO contains many small entries, it is less likely that a randomly chosen UTxO entry will push the total above the upper bound we set. In other words, our answer to “what do we do when we happen to pick a huge UTxO entry?” is “we stop trying to improve our selection”. We can now describe the algorithm: Randomly select outputs from the UTxO until the payment value is covered. (In the rare case that this fails because the maximum number of transaction inputs has been exceeded, fall-back on the largest-first algorithm for this step.) Randomly select outputs from the UTxO, considering for each output if that output is an improvement. If it is, add it to the transaction, and keep going. An output is considered an improvement when: It doesn’t exceed the specified upper limit Adding the new output gets us closer to the ideal change value It doesn’t exceed the maximum number of transaction inputs. Figure 6. The Random-Improve algorithm. Side note for point (2a): we use twice the value of the payment as the upper limit. Side note for point (2b): it might be that without the new output we are slightly below the ideal value, and with the new output we are slightly above; that is fine, as long as the absolute distance decreases. Evaluation The algorithm from Figure 6 is deceptively simple. Do the self organisation principles we isolated really mean that order will emerge from chaos? Simulations suggest, yes, it does. We already mentioned how random input selection does a great job at cleaning up dust in Figure 2; what we didn’t emphasize in that section is that the algorithm we simulated there is actually our Random-Improve algorithm. Notice how the median change:payment ratio is initially very low (indicative of a coin selection algorithm that is generating a lot of dust outputs), but climbs rapidly back to 1 as soon as Random-Improve kicks in. We already observed that it does indeed do an excellent job at cleaning up the dust, quickly reducing the size of the UTxO. The simulations in Figures 4 and 5 are also the result of the Random-Improve algorithm. That said, of course the long term effects of a coin selection algorithm can depend strongly on the nature of the distribution of deposits and payments. It is therefore important that we evaluate the algorithm against a number of different distributions. Normal distribution, 10:1 deposit:payment ratio We already evaluated Random-Improve against normally distributed payments and deposits with a 1:1 ratio and a 3:1 ratio; perhaps more typical for exchange nodes might be even higher ratios. Here is a 10:1 ratio: Figure 7. Simulation of largest-first coin selection. Main histogram shows UTxO entries; inset graph shows UTxO balance in blue and UTxO size in red, histogram top-right shows number of inputs per transaction, graph bottom right shows the change:payment ratio (more on that below). Graph at the bottom shows the distribution of deposits (blue, left axis) versus payments (red, right axis). In this case, both are normally distributed with a mean of 1000 and 3000 respectively, and we have a deposit:payment ratio of 3:1; modelling a situation where we have frequent smaller deposits, and less frequent but larger payments (withdrawals). The wallet starts with an initial balance of 1M. We see a very similar picture as we did in Figure 5. Since the deposits and payments are randomly drawn (from normal distributions), the UTxO balance naturally fluctuates up and down. What is satisfying to see however is that the size of the UTxO tracks the balance rather precisely; this is about as good as we can hope for. Notice also that the change:payment ratio is a nice round 1, and the average number of transaction inputs covers around 10 or 11, which is what we’d expect for a 10:1 ratio of deposits:payments. Exponential distribution, 1:1 deposit:payment ratio What if the payments and deposits are not normally distributed? Here is Random-Improve on exponentially distributed inputs: Figure 8. Random-Improve, 1:1 deposit:payment ratio, deposits and payments both drawn from an exponential distribution with scale 1000. In an exponential distribution we have a lot of values near 0; for such values it will be hard to achieve a “good” change output, as we are likely to overshoot the range. Partly due to this reason the algorithm isn’t quite achieving a 1.0 change:payment ratio, but at 1.5 it is still generating useful change outputs. Furthermore, we can see that the size of the UTxO tracks the UTxO balance nicely, and the average number of transaction inputs is low, with roughly 53% having just one input. Moreover, when we increase the deposit:payment ratio to 3:1 and then 10:0, the change:payment ratio drops to about 1.1 and then back to 1.0 (graphs omitted). Erlang The exponential distribution results in many very small deposits and payments. The algorithm does better on slightly more realistic distributions such as the Erlang-k distributions (for k > 1). Here we show the animation for the 3:1 deposit:payment ratio using the Erlang-3 distribution; the results for other ratios (including 1:1) and other values of k (we additionally simulated for k = 2 and k = 10) are similar. Figure 9. Random-Improve, 3:1 deposit:payment ratio, deposits drawn from an Erlang-3 distribution with scale 1000 and payments drawn from Erlang-3 distributio with scale 3000. More payments than deposits We have been focusing on the case where we have more deposits and fewer (but larger) payments. What happens if the ratio is reversed? Figure 10. Random-Improve, 1:10 deposit:payment ratio, deposits and payments drawn from a normal distribution with mean 10k and 1k, respectively. 1M cycles. In this case we are unable to achieve that perfect 1.0 change:payment ratio, but this is expected: when we have large deposits, then we frequently have no choice but to use those, leading to large change outputs. We can see this more clearly when we slow things right down, and remove any source of randomness; here is the same 1:10 ratio again, but now only the first 100 cycles, and all deposits exactly 10k and all payments exactly 1k: Figure 11. Random-Improve, 1:10 deposit:payment ratio, all deposits exactly 10k, all payments exactly 1k (no randomness). First 100 cycles only. We can see the large value being deposited, and then shifting to the left in the histogram as it is getting used for deposits, each time decreasing that large output by 1k. Indeed, this takes 10 slots on average, which makes sense given the 10:1 ratio; moreover, the average value of the “large output” in such a 10-slot cycle is 5k, explaining why we are getting 5.0 change:payment ratio. The algorithm however is not creating dust outputs; the 1k change outputs it is generating are getting used, and the size of the UTxO is perfectly stable. Indeed, back in Figure 12 we can see that the size of the UTxO tracks the balance perfectly; moreover, the vast majority of transactions only use a single input, which is what we’d expect for a 10:0 deposit:payment ratio. Real data MoneyPot.com Ideally, of course, we run the simulation against real event streams from existing wallets. Unfortunately, such data is hard to come by. Erhardt was able to find one such dataset, provided by MoneyPot.com. When we run our algorithm on this dataset we get Figure 12. Random-Improve, using the MoneyPot data set. There is a rougly 2:1 deposit:payment ratio. Values have been scaled. NOTE: log scale on the x-axis. A few observations are in order here. First, there are quite a few deposits and payments close to 0, just like in an exponential distribution. Moreover, although we have many values close to 0, we also have some huge outliers; the payments are closely clustered together, but there is a 10xE9 difference between the smallest and largest deposit, and moreover a 10xE5 difference between the largest deposit and the largest payment. It is therefore not surprising that we end up with a relatively large change:payment ratio. Nonetheless, the algorithm is behaving well, with the size of the UTxO tracking the balance nicely, with an average UTxO size of 130 entries. The average number of outputs is 3.03, with 50% of transactions using just one input, and 90% using 6 or fewer. Cardano Exchange One of the large Cardano exchange nodes has also helped us with some anonymised data (deposits and payments), similar in nature to the MoneyPot dataset albeit significantly larger. Coming from an exchange node, however, this dataset is very much skewed towards deposits, with a deposit:payment ratio of roughly 30:1. Under these circumstances, of course, coin selection alone cannot keep the UTxO size small. Figure 13. Random-Improve, using data set from a large Cardano exchange. There is a rougly 30:1 deposit:payment ratio. Values have been scaled. NOTE: log scale on the x-axis. Conclusions The choice of coin selection algorithm has far reaching consequences on the long term behaviour of a cryptocurrency wallet. To a large extent the coin selection algorithm determines, over time, the shape of the UTxO. Moreover, the performance of the algorithm can be of crucial importance to high-traffic wallets such as exchange nodes. In his thesis, Erhardt proposes “Branch and Bound” as his preferred coin selection algorithm. Branch and Bound in essence is a limited backtracking algorithm that tries to find an exact match, so that no change output needs to be generated at all. When the backtracking cannot find an exact match within its bounds, the algorithm then falls back on random selection. It does not, however, attempt our “improvement” step, and instead just attempts to reach a minimum but fixed change size, to avoid generating dust. It is hard to compare the two algorithms directly, but on the MoneyPot dataset at least the results are comparable; Erhardt ends up with a slightly smaller average UTxO (109 versus our 130), and a slightly smaller average number of inputs (2.7 versus our 3.0). In principle we could modify our Random-Improve algorithm to start with bounded backtracking to find an exact match, just like Erhardt does; we have not done this however because it adds complexity to the algorithm and reduces performance. Erhardt reports that his algorithm is able to find exact matches in 30% of the time. This is very high, and at least partly explains why his UTxO and average number of change outputs is lower; in the Cardano blockchain, we would not expect that there exist exact matches anywhere near that often (never mind finding them). Instead our proposed Random-Improve does no search at all, instead purely relying on self organisation principles, the first of which was stated by Erhardt, and the other two we identified as part of this research. Although in the absence of more real data it is hard to evaluate any coin selection algorithm, we have shown that the algorithm performs well across a large variety of different distributions and deposit:payment ratios. Moreover it is straight-forward to implement and has high performance. One improvement we may wish to consider is that when there are very large deposits, we could occassionally issue a “reorganisation transaction” that splits those large deposits into smaller chunks. This would bring the change:payment ratio down, which would improve the evolution of the UTxO over time and is beneficial also for other, more technical reasons (it reduces the need for what we call “dependent transactions” in the wallet specification). Such reorganisation is largely orthogonal to this algorithm, however, and can be implemented independently.IOHK make a visit to Google’s London offices2018-06-28T00:00:00+00:002018-06-28T00:00:00+00:00https://iohk.io/blog/iohk-make-a-visit-to-googles-london-offices<p><img src="/images/blog/iohk-at-google.jpg" alt="Cityscape" class="fullwidth" /></p>
<p>Cryptocurrency is one of the most discussed topics of the moment, and whatever people think about it, they all want to know what’s next for the technology. An audience that was no different was at Google, where Charles Hoskinson was invited to talk about Cardano and the future of cryptocurrencies. At the meeting, held at Google’s London headquarters last month, Googlers around the world dialled in to hear the presentation and put questions to IOHK’s chief executive, and to Duncan Coutts, IOHK Director of Engineering. As you might expect from a company that has laid much of the ground for today’s technological landscape, Googlers asked some of the most incisive and informed questions Charles has had on his speaking tour this year.</p>
<p>After a brief introduction from Charles on IOHK and Cardano, the floor was opened to questions. Cardano development raised much interest, and Charles explained how its consensus protocol, Ouroboros, uses staking as a means to encourage people to join and help run the network. Development milestones were in focus too, such as one expected in July, when a <a href="https://testnet.iohkdev.io" title="Cardano Testnets">test network</a> will be opened to developers who want to play around with smart contracts on the IELE virtual machine. Later this year, full decentralization of the network is expected, as part of the Shelley phase of development, and Charles explained the background to all these topics.</p>
<p>There followed questions about how developers could get involved with Cardano, and about the K framework, which is part of IOHK’s test smart contract capability; how cryptocurrencies will cater for privacy, and of course, about where cryptocurrencies are headed. After the session, Googlers were kind enough to take the IOHK team up in the glass lifts to the top of the building and on to the roof, to enjoy the spectacular view across London.</p>
<p>Read the conversation, below.</p>
<p><strong>Q. I have a question about Ouroboros and staking: is the number of tokens on offer sufficient to convince people to join the protocol and help run the network?</strong></p>
<p><strong>Charles</strong>: We published a preliminary monetary policy. The ceiling is 45 billion tokens and the current number in circulation is 26 billion so we have a little bit of room to work with inflation there, plus there’s transaction fees as well to subsidise transaction validation.</p>
<p>First, Proof of Stake (PoS) is an extremely cheap protocol to run, especially Ouroboros, if you compare it to mining. The odds are that the operational costs will be so much lower that you really don’t need to pay as much. But it’s a broader and more abstract question: how do you handle fees and incentives and stake pools and delegation and then get all those incentives engineered in a way that you have reasonable game theoretic reasons to believe that the system is going to behave as intended? That’s a really hard question. So we have two individual work streams. One is led by Elias Koutsoupias, an algorithmic game theorist at Oxford and a Gödel prize winner. He’s working on the incentives question, trying to create models and understand [inaudible] first example – if you want to delegate and run a collection of stake pools, how many ought there to be and what are the economics of that?</p>
<figure class="alignright">
<img class="alignright" width="250" src="/images/blog/cardano-google-2.jpg" />
<figcaption>Outside Google HQ, London</figcaption>
</figure>
<p>Then, the other side is, if I’m going to try to convince people to delegate, they ought to get a reward, so how much should that be? And then you have to do some empirical calculations – what is the operational cost of the network? You don’t want to pay too much more [inaudible] but you also want to pay enough to incentivise people to run 24/7 nodes to maintain the system. It’s an interesting question, but with the inflation that we’ve proposed we have more than enough wiggle room to work with. Not only will people participate, they’ll probably make windfall profits relative to operational costs, given the way these markets work.</p>
<p>We opened up registration for stake pools last month and were looking for 100 applicants for a beta test, but got 1,500 applications – 15 times more people expressed interest than we expected.</p>
<p>As with all monetary parameters in a beta system, these things can be adjusted depending on facts and circumstances, but the reality is that the driver here is the price of the underlying asset – the token – and markets tend to converge on that. The short answer is, it’s probably going to work out; the long answer is that we’re probably not going to have the right model to begin with. We’re either going to underpay or overpay and it’s qualitatively going to be pretty obvious, based on participation of the network. The odds are that we’re probably going to overpay in terms of rewards.</p>
<p><strong>Q. On all the projects that you are driving, are there specific milestones that will for sure be completed this year?</strong></p>
<p>Look at <a href="https://cardanoroadmap.com" title="cardanoroadmap.com">cardanoroadmap.com</a>, for Cardano-specific projects. Month by month, it gives an update on where we’re at. We also do weekly reports and we try to be as transparent as possible about where we’re at. Our goal is to release the next major version of Cardano some time this year, called Shelly. We are working really hard towards that. It might slip, but the odds are that it won’t. It’s a difficult project. Shelley is true decentralization of the network. At the moment we’re running our proof of stake protocol in a forced delegation model. So all the PoS mechanics are there and the stake rights have been delegated to nodes that IOHK and two other entities control, so it’s a federated network. We did this because we’re not insane. You don’t go and invent a protocol in the academy, turn it on and say ‘Good luck everybody’. Instead, you have training wheels on your bicycle. You say, ‘Let’s launch this system in a federated format and gradually decentralise the system once we have momentum and assurance that what we’ve done is correct’. And also when we’ve trained up hundreds of people to run stake pools and participate in the staking process so there’s a bit of redundancy and a much more natural unlocking of the system. So, over six to nine months, that process will continue and hopefully all the Shelly mechanics will roll out.</p>
<p>In parallel, we are releasing <a href="https://testnet.iohkdev.io" title="Cardano Testnets">testnets</a> for smart contracts. The first one will be released at the end of the month, and this is done with something called the KEVM. We worked with Runtime Verification at the University of Illinois Urbana-Champaign, and they took the operational semantics of the Ethereum Virtual Machine and wrote them in a meta language called K. What the K framework allows you to do is implement a correct-by-construction version, just from the semantics of the virtual machine. So it’s really cool. What we were able to do is actually take the K semantics, build a VM, connect that to a fork of Ethereum and we’re now turning that on at the end of the month to test that this framework is working and you can run smart contracts on it. We also have another virtual machine that we built specially for Cardano, called IELE. And those semantics are publicly available in GitHub. We have a paper that we are submitting for peer review. That testnet will launch some time in June or July – that gives people who live in the Ethereum world and write smart contracts the chance to play around and deploy code on our system and look at our gas model and get a better understanding of how our system works. And then, over time, testnet iterations will occur and eventually we’ll pull these two systems together.</p>
<figure class="alignright">
<img width="250" src="/images/blog/cardano-google-3.jpg" />
<figcaption>IOHK arriving at Google HQ, London</figcaption>
</figure>
<p>One of the architectural features of Cardano is the separation of accounting and computation. With Ethereum they are bundled together, your peanut butter is in your jelly. And that’s fun from an implementation standpoint; it’s simpler to maintain, but it creates a lot of problems. If you screw up parts of your computational model you’ll also inadvertently block your ability to spend money. Also, computation carries a much higher liability than accounting does. For example, Bitcoin versus Ethereum in terms of transactions. In Bitcoin, if I send Jane a transaction and Philipp a transaction, buying a laptop from Jane and weapons-grade plutonium from Philipp – the miner in the system would have no way of differentiating between those two transactions, they’re fungible. We don’t know the actors, they are just transactions. But if we’re running code, you might be able to differentiate between Crypto Kitties and Silk Road. There is some precedent if you look at Tor exit nodes having legal liability and being arrested for trafficking, child pornography or copyright violations. Computation, if you can discover the difference between what Jane and Philipp are doing, has higher liability. In our view, architecturally, it’s a good idea to separate them and it also gives you a lot of flexibility because you can have multiple computational models, like we’re backwards compatible with Ethereum and we had a different model and we have a functional model and you can do a lot of cool stuff with that. The downside is that you have to maintain the state of many ledgers at the same time and you also have to figure out how to move value between the ledgers, which we’re going to do because of our interoperability mandate. We decided to take this on but it adds complexity to the system, a lot more work to do. That will be gradually rolled out in stages through testnets and it’s quite a bit of work.</p>
<p><strong>Duncan</strong>: There’s the compartmentalization aspect of it. Ethereum is monolithic, it bundles together all of the features, so if one thing breaks the whole thing breaks. If there’s some fundamental flaw you haven’t found, not only have all your ERC20 tokens gone but so has ether itself. There’s no compartmentalization between those things. But if you have, in essence, a Bitcoin-style simple settlement layer and then you do your EVM stuff and equivalent to EVM on different blockchains that are linked you can move money between them but they’re otherwise compartmentalized. If for some fundamental reason there’s a flaw found in the EVM that destroys that, well, that’s very sad but it doesn’t destroy the settlement layer. That’s a big advantage. And it means, as Charles says, you can add new ones of these things that can be somewhat experimental because that lets you evolve the system a bit.</p>
<p><strong>Charles</strong>: We wrote, I think, the first formalization of sidechains. There was a sidechains paper written in 2017. So what a sidechains transaction is for those who don’t know it, I like to call it interledger transactions. So you have a source ledger, a destination ledger and an asset. So basically what you’re trying to do is initiate a transaction, where the destination ledger can answer two questions about the transaction. One, is that the asset does exist from the source ledger, and two, that the asset from the source ledger has not been double spent. The foundational question you’re asking is how much information does the destination ledger need to possess to be able to validate that transaction and verify those two questions? We wrote a model, first for proof of work, called ‘Non-interactive Proofs of Proof of Work’ that explains how to do this, and now we’ve extended that model to Ouroboros and the proof of stake world and we have a paper that we’ve just submitted that contains details on how to construct these types of proofs and what these transactions look like. There are still questions about how large the proofs are going to be, relative to the size of the ledger, and there are questions about validation time, and also generality. The proofs we scoped work with our particular consensus algorithm, but we’d like to make these things work for all ledgers, not just a particular type of ledger, so there’s a lot of work to be done there. But it’s the first time it’s been formalized. The concept has been enumerated in a paper that was written in 2014, by a competitor of ours called Blockstream, but they didn’t write a proper academic paper and submit it for peer review. That’s considerably harder, there’s a lot more to think about when you’re rigorous about these things. In the long term, it’s a great thing to think about for scalability and interoperability, and also testing because you can now deploy versions of your chain with different parameters and it’s easy to move value between them and you can let people vote with their feet.</p>
<p><strong>Q. How will Cardano overcome the first-mover advantage of Ethereum? Do you see multiple smart contract platforms co-existing in the space or will there be one prominent winner?</strong></p>
<p>So how many Java, C++ or Go developers are writing code on Ethereum? You can’t, Ethereum doesn’t support any of these languages. They can’t even run a single viral app on the platform. If you look at the top 10 languages, none of them works on the system, so, by definition, all those developers aren’t developing for the system, they have to go and learn new tools and new stuff. With Cardano, first off, we’re backward-compatible, 100%, we’re running an EVM. So you can take your Solidity code and your Web 3 stuff and all the things you’ve come to know and love about Ethereum, and you can run it on my system, and it’s faster, cheaper and safer to run it on my system because we have a better consensus model. Second, through our work with the University of Illinois, through Runtime Verification – Grigore Rosu and his team – we’re working on something called semantics-based compilation. Should this be successful, we can take any K-defined system and translate it to run on our machine. All you have to do for a new language is just write the semantics in K, one time, and then K framework takes care of the rest of it. It’s a hard, high-risk, high-return project, but at the end of the day, we will end up one way or another supporting mainstream languages. Part of it is backwards compatibility, part of it is supporting mainstream languages, part of it is recognising that the vast majority of real applications aren’t running on Ethereum at the moment. The other thing is that smart contracts are not monolithic and you write and run your entire application on a blockchain and the reality is you have to add a server-client component to it. Let’s think of it like a poker game, maybe you trust random number generation to the server, but other things like player matching and account managing, these things are almost certainly not going to run on your blockchain, you’re dumb if you’re going to do these types of things. They’re probably going to run on some sort of server back-end. It’s more of, I treat a smart contract as a computational service. So it’s silly to say, ‘Oh well, only one platform and one token’s won’, it’s akin to saying Internet Explorer’s won and we all have to be Active X developers, god help us. I’m not loyal to IE, or Amazon Web Services. Rather, I have to ask, what’s the cheapest, best, most secure environment for me to run my computation in for my users? Our strategy is be backwards compatible, support more languages, especially mainstream languages in a better way, have a better user and developer experience, and be smarter about the ecosystem in which these contracts live. So we make it easier for the server to come into play, to use multiple ledgers and have a good app platform to deploy these types of things on, and we’ll definitely get a lot of growth there.</p>
<p>The other thing is that very few people today write smart contracts. They play with these things, but very few people are smart contract developers. If 99% of developers aren’t in the ecosystem, how can you say a person has first-mover advantage? It’s nuts.</p>
<p><strong>Q: In 2014 I played around with the K framework for a university project, but I found it to be extremely slow.</strong></p>
<p><strong>Charles</strong>: Yes, because there is a K to OCaml back-end, but we’re building a K to LLVM back-end, which should be about 100 times faster.</p>
<p><strong>Q: But is that enough? Because it was outright impossible to run a reasonably large project with thousands of lines of code.</strong></p>
<p><strong>Duncan</strong>: This is one of the problems that Grigore is trying to solve. As you say, executing the operational semantics directly is very slow. Runtime Verification are basically trying to do a compiler approach, which they believe will be fast.</p>
<p><strong>Charles</strong>: It’s still a big unknown, exactly how much performance is necessary – and can we get within an order of magnitude of handwritten code? One proof of concept is the testnet that we’re launching at the end of this month running a version of the Ethereum Virtual Machine built in K. You can run smart contracts on the KEVM and compare them to an Ethereum testnet and see the performance delta between the two. But it’s also important to understand that the open source K framework components that you use and the version that Grigore uses are different. Grigore built a second set of software for his private company, Runtime Verification, that he uses for contracts he’s had with Boeing and Nasa, and I think that’s about 100 times faster than the one that you used. But even so, there’s still a big performance delta that needs to be ameliorated. We have quite a large team, there’s 19 people involved in that contract. Some of those people are allocated specifically for performance. Now let’s say that we can’t quite bridge that performance, there’s probably a lot of things we could do to cheat and improve things a bit, including handwriting certain things, or abandoning semantics-based compilation for more traditional techniques. But it’s still a big unknown. This is also why we have a multi-computation model, so in addition to the IELE framework and the K framework, we also have an alternative approach called Plutus.</p>
<p><strong>Duncan</strong>: We think that most of the computation time in most of the smart contracts goes into crypto primitives. So you don’t have to have the world’s fastest interpreter to interpret those smart contracts. Plutus – people like me with a programming language background look at the EVM and Solidity, and say ‘It looks like the people that wrote this didn’t have much experience with programming language design’. There’s an academic discipline to the design of programming languages, and that didn’t seem to inform the design of Solidity at all. That shows up, in things like if you miss one error code then your smart contract loses everybody’s money. So we have two smart contract platforms that Charles has mentioned, the backward compatibility story, byte-for-byte compatibility with the K version of the EVM and then IELE, which is EVM style, but fixing a lot of the obvious problems. That gives us the story of how you compile Solidity programs to IELE and KEVM. In addition, we have a smart contract platform that is based on programming language research, in particular functional programming. We have an approach based on a functional core language, which is actually executed, based on system [inaudible] and then two languages which are compiled into that core language. The core language is what’s executed on the consensus nodes. That’s the [inaudible] equivalent of the EVM. We don’t call it a VM, it’s just an intermediate code, that’s the equivalence, and then two languages initially that compile into that. One is called Plutus, which is a functional, Turing-complete language very similar in many ways to Haskell but simplified and cut down. Then there is a non-Turing-complete DSL (domain specific language) that is aimed specifically at financial applications. It’s based on a paper from around 2001, ‘Financial Smart Contracts’, that lets you express all the normal and even exotic financial contracts that people tend to write, but it does it in a much simpler way so they can be easily analysed and understood. The point is, if your application fits into the domain of that DSL then you would get much shorter and simpler, easier-to-analyse programs, but alternatively, you can go to the general purpose, Turing-complete functional language and in both cases, it’s a two-layer language approach. If you look at existing Ethereum applications they don’t just run on the blockchain, as Charles said, it’s a blob of Javascript that runs on the client, and some Solidity code that runs on the back-end. Your programming model is this two-level thing anyway, but there are two different languages, in many ways like our web stack. Our web stack has grown over time, one language runs on the back-end and another on the front-end, and these multi-language things are not that easy to do, especially when they have grown accidentally. Because we can see that, we are taking a more deliberate approach, and saying, let’s design an explicitly two-level language so this bit will run on the blockchain, this bit will run on the client, but they are very similar languages. So what we’re actually doing is Haskell on the client and Plutus on the blockchain. And Plutus is very similar to Haskell, so what you will see is one program with one file… one program with embedded snippets that run on the chain and an out-of-context [inaudible] layer that happens off-chain. That should give a better, more integrated development experience and we aim to be able to do things like analyzing the stuff that runs on-chain, so you can demonstrate the safety properties of the on-chain code.</p>
<p><strong>Charles</strong>: And performance should be equivalent to Haskell, because they share a common core.</p>
<p><strong>Duncan</strong>: Right, and the Haskell code will be run through the Haskell compiler.</p>
<p><strong>Q: How can a software developer that is excited about your project get best involved with it? Do you have any plans for educating developers or creating developer-relation type of roles at IOHK?</strong></p>
<p><strong>Charles</strong>: I really admire what you guys did with Dart. I love the developer experience effort that Google put into it and there are things to take from that.</p>
<p>It’s difficult with a cryptocurrency that is very rigorous in its approach. You’re starting with white papers written by cryptographers and the notion of formal specifications and you’re trying to implement these and prove correctness, to figure out: when and how do you open that project up to successfully collaborate with one-source developers? We are hiring people specifically to work with the exterior community and try to communicate how we are running a project and how we are writing things and how we welcome third-party contributions. There is a lot of technology in our stack, we’re making material enhancements to K, so anyone who wants to contribute there definitely should. We have Electron in our stack, so we are using Chromium and Node.js for our wallet front-end so there’s a lot of things going on there with the Daedalus wallet, and we’d love contributions there. And, of course, there’s the Haskell back-end. We are reimplementing some of that back-end in Rust, and experimenting with Rust and web assembly in the browser, so a Chrome-based wallet. So there’s a lot of tech there and it depends on the core competency of the person and what exactly they’d like to contribute. We have yet to build an easy-to-use, formal external process, to make it friction-free for external developers to come and assist us and it’s going to be a high priority in the second half of this year to figure out how we do that.</p>
<p>Another thing is that if an open-source project is to be successful, especially with these types of protocols, we do need competition. When I was running Ethereum, multi-client models were very important for us, so we ended up implementing Geth and the C++ client. And then, later on, Gavin Wood split off and created the Parity client for Ethereum. Now, this was great because it really forced us to specify the protocol properly, and we could no longer say, ‘Well, the code is this specification,’ because you’re [inaudible] of ambiguity there. So we worked hard at proper documentation, and we’d like a similar environment to materialize, and it would be great to see some alternative projects grow. But at the moment, the best you can do is go to our forums, go to our GitHub repository, and you can, of course, open issues and email our developers. And if you’re really interested in making open source contributions, we’ll try to find a way to integrate you. And long term, we’ll have a formal process for that’s really easy for people to connect with. And also, again, it’s depending on the particular level you want to contribute to. For example, we do formal specification verification work, so if you’re an Isabelle or Coq developer and you want to work with us, that would be great. If there’s like five people there, we’d probably be able to do that, right? But, levity aside, it would be fun to find people there. And other things, like if you want to build applications in our system. We are launching testnets soon, so it would be much appreciated for people to write software to deploy in our system and try to break it because that helps our system get better. So that’s the best non-answer to your question that I can give!</p>
<p><strong>Duncan</strong>: At the moment, our code is all open, it’s on GitHub. This is one of those projects that’s open source but not yet open, collaborative. It’s not easy at the moment for us to collaborate with people because we’re not yet using GitHub with its issue tracker; we have a private issue tracker at the moment, for example. So this is one of the things where we aim to get there, for there to be documentation that’s easy for other people to look at and understand and make contributions and for us to accept. So the goal is to get there, but we are not there yet. You can go and read all the code, but you can’t see what the open tickets are, the documentation’s a bit patchy. So that is where we’d like to get to, to be able to direct people and accept contributions from anyone really.</p>
<figure class="alignright">
<img width="250" src="/images/blog/google-iohk-videoconference.jpg" />
<figcaption>IOHK videoconferencing with Google HQ</figcaption>
</figure>
<p><strong>Charles</strong>: We are going to try to annotate a lot of the design decisions in the system. For example, we recently released a formal specification for our wallet back-end. I think it’s the first time it’s ever been done for a UTxO wallet, and we’re going to create YouTube lectures, going through section by section on that and putting it up specifically with the aim of educating developers how our design decisions work and what the system’s all about. So as we specify each component of our system, we’re going to try to do that. We have an internal department called <a href="https://iohk.io/education/" title="IOHK Education, iohk.io">IOHK Education</a>, led by a mathematician named Lars Brünjes that specializes in that, so over time, you should see more accessible documentation materialising. Hopefully, that will encourage people who have the capacity to contribute to come in. We are also discussing how we open up our issue tracker. We made the decision to have a private issue tracker in the beginning because there’s a lot of, usually, security concerns or open discussion about what direction to go in because the protocol is still very young. So we just figured, I’ll just leave that all private and not worry too much about it. But we do have a moral obligation, as an open source project, to try to get project management and issue tracking into a more open domain. So there is a lot of open discussion about that. And once those things get into the open domain, it will considerably easier for open source contributions to occur.</p>
<p><strong>Q: We have another question on privacy. Are there any plans to implement private transactions in the style of Monero or Zcash?</strong></p>
<p><strong>Charles</strong>: Yes, so Monero uses a primitive called ring signatures, and Zcash uses a snark primitive. So privacy is a complicated topic because you’re actually talking about three things. You’re talking about the privacy in terms of linkability, so, if I look at a transaction or an address, what’s the probability that I can relate it to a known identity? So basically, go from anonymous or pseudonymous to known, the linkability dimension of it. Then there’s the obfuscation of amounts. So you might not be able to easily connect the addresses or transactions to people, but if it’s a public ledger, you could certainly see what’s the most expensive transaction, and it creates kind of a priority cue for deanonymization. Say, ‘Ah, well, there’s a $10 million transaction that happened today, let’s go and find who has that with a wrench to rob him’. And then there’s the network side of things, so can you obfuscate the use of the protocol or try to prevent people from being able to understand what you’re doing with the protocol? So there are existing solutions in all three of these. Like ring signatures and Zcash cover the first category. Confidential transactions, for example, covers the second category. And the third are covered by technologies like Dandelion. So first, you have to understand that privacy is a spectrum, and also, it is one that carries considerable regulatory discussion. For example, Japan just announced that they’re probably going to de-list all the privacy coins. So if we wish to be in the Japanese markets and we were to embrace Monero-style privacy, there’s a very low probability that Japanese exchanges will list Ada, which is a high priority for us.</p>
<p>On the other hand, privacy is a moral right. If you don’t have privacy in your system, you’re basically creating a system that your entire financial history is publicly known back to the beginning of time, since the system’s inception, which is dystopian to the max. So the best way of resolving this is to develop out some really good privacy options and implement them as improvement proposals, and then take advantage of the governance part of the system. So when voting is available, we can actually have alternative proposals and say, ‘Well, if you wanted to have ring signature style privacy with the whole banana, here’s how we would do that.’ And how much public support do we have for that among Ada holders? And, basically, have a referendum and see which one wins out, and then you can see where in the spectrum you fall. But the important thing is, people need to be informed. If you maximise privacy, you will inadvertently make the protocol illegal within certain jurisdictions and limit market access within certain jurisdictions. If you make the protocol more open, you are inviting more dystopian people to track what you’re doing and use it against you. So we’ll let the community decide that, but we do have active research. For example, on Dandelion, we’ve been funding that team at UIUC, and they’re creating a version of Dandelion that will come out this year. We also have had discussions with people who have been formalising the Monero cryptography primitives and trying to make them more efficient and better bound to security and better bound to privacy. There’s one project out of UCL that was done by Sarah Meiklejohn, and there was one out of China that was done, some professors in Beijing and a few other places – I think it’s called RingCT 2.0. So there’s certainly a lot of good tech, and we know how to implement that technology, but it’s now mostly in the hands of a social phenomenon rather than someone like me making that decision on behalf of the ecosystem.</p>
<p>There’s another thing that’s seldom mentioned, which is the idea of backdoors. We used to live in the [inaudible] debate, or, either you give an unlimited-use private backdoor to a trusted actor like the FBI, or you don’t. But there can be a spectrum with these things. For example, we can put a backdoor in the system, but it has to be constructed with mono-cryptographic primitives, and the use of it requires majority consent of the system, and it’s publicly known if it’s used. So would you be willing to invest in a currency that says only a certain group of people, if they come together and they have near-universal consensus [inaudible] and you know that they’ve used it, and it can only be used on a particular person instead of the entire system, is that a reasonable compromise? So I think that more-nuanced discussion has to be had, and there is certainly a lot of tech that’s being developed to accommodate these types of things. In fact, the inventor of the Snark, Eli Ben-Sasson, has recently started a for-profit company, and he’s developed technology like this to augment Zcash to be able to provide these types of backdoors which are auditable and publicly verifiably when used and, in some cases, one-time used, depending on how they’re deployed. So we’ll certainly be a member of that conversation, and eventually, we’ll settle on something, and it’s [inaudible] of how much privacy’s required. Closely related to it is also the place of metadata and attribution. So under what contexts should you attach mandated transactions, and how do you do that, and how do you share it? This is really important for compliance. If you look at exchanges, they have KYC (know your customer) and AML (anti-money-laundering) reporting requirements, and because of that, they’ve inadvertently become data custodians where they hold tons of personally identifiable information about their customers. They don’t want to hold it, but they have to because of the law. It would be much better having a minimum viable escalation model where you are allowed to have a challenge-response type of a query where you ask questions about your customers, like, are you a US citizen or not? And you can get some verification of that. But you don’t have to have that data necessarily.</p>
<p>The example I like to use is that, in the US, you have to be 21 or older to drink. How we usually verify that is we look at your driver’s licence. And because I see that document, I know your address, I know your driver’s licence number, I know exactly how old you are and how fat you are, how tall you are, the colour of your eyes, the colour of your hair. That’s a bunch of information you don’t need to know, but you inadvertently know because of the way that you’ve done verification. It would be much simpler to be able to just ask a query: are you over the age of 21? And have a response of ‘yes’ and know that that response is correct. Then I’ve learnt nothing about you other than that particular question, and we leave it at that. And the proof itself is the sufficient burden for the merchant. So we’re certainly involved in that conversation. We have a lab at University of Edinburgh that studies these types of things, a privacy and verified computation lab. It’s led by a former Microsoft person, Markulf Kohlweiss, who worked on the Everest project and other things. And privacy is in the remit of the lab, so we’ll come up with some options, and then we’ll democratically validate those options. And whichever one the Ada holders have decided, we’ll implement into the system. And by the way, this takes time, several years.</p>
<p><strong>Q: We actually have a couple of submissions on future outlook. I’m going to read off this. What do you think is the next big thing in the crypto space, outside of your own roadmap? For example, let’s say Hashgraph? And related to that, after scalability and sustainability, which do you think will be the next challenge? And what do you think will be your next big ticket, either in the crypto space outside of your current projects, such as Hashgraph?</strong></p>
<p><strong>Charles</strong>: I think the magic is that there are a lot of concepts that have been sitting on the academic shelf for 25, 30 years, or emerging trends that we’re seeing in hardware or software that can now be dragged into the cryptocurrency space. I’ll give three examples: one is multi-party computation; two is trusted hardware; and three is a lot of PL [programming language] concepts. So in terms of multi-party computation, these protocols or the concept of multi-party computation have been known about since the 1980s, but only recently have the protocols become efficient enough to run on consumer hardware. So, for example, let’s look at poker, that’s kind of the canonical example. This is a perfect example of something that you probably don’t want to run in an Ethereum-style virtual machine. You’re not talking about poker games with 100,000 players; you’re talking about poker games with five players, 10 players, 15 players, so you don’t have a very large set of people. And, really, what do you care about in poker? That people can cheat, and that at the end of the game I’m going to get my winnings and losses. That’s basically it. And you don’t particularly care if that data gets committed to the blockchain. Why should it be there? It’s a game. We’re using blockchain as a broker and a payment system, so we’re using it to find each other and to make sure that we get paid, but outside of that, what I really care about is I played the game. I don’t particularly care about the metadata, or if I do, there’s a way of capturing it that’s far more efficient. So if you look at the Ethereum-style solutions, they say, ‘Oh well, who cares? We’re still going to write it as a smart contract and all the same, your poker game, even though you’re only playing with five people, is the business of every single person on the entire network and competes for resources with every single person.’ So I think there are much more intelligent ways of doing things off-chain, and multi-party computation provides an avenue for that, so we’re exploring it. But then, there are also projects like Enigma, out of MIT Media Lab that are exploring these ideas as well. And so I’m really excited to see the intersection of MPC [multiparty computation] and blockchain, and I think you can do a lot of cool things, not just poker but things like decentralised exchange or even general-purpose computation.</p>
<figure>
<img class="fullwidth" src="/images/blog/cardano-google-4.jpg" />
<figcaption>Charles and Duncan at Google HQ</figcaption>
</figure>
<p>Trusted hardware is really cool too because it’s the closest thing we have to magic. And I think it’s the only way we have for offline, off-chain payments. So let’s say you live in Africa. I was just in Rwanda and I was just before that in Ethiopia, and it would be really crazy for me to go around Addis Ababa and say, ‘I’m going to solve all your problems with my magic internet money which requires you to be online all the time to use, when you don’t even have a stable internet connection.’ They’ll just look at me like, ‘Okay, crazy white person. Go away.’ So the reality is that it would be nice to be able to have a trusted enclave that you can store the private key within that enclave so that you get certain guarantees like the balance associated with that key is correct, you haven’t double spent it, and when you transact it, you can move the private key from one enclave to the other and then get a proof that it was destroyed on the enclave that you had. Now, if you can do this – and trusted hardware can potentially allow you to do this – then you can transact offline. Because what I can do is just walk over to someone who has a cellphone or a hardware device and tap it with my hardware device, and, basically, we just move the private key from one device to the other device and transmit over a proof of destruction, and suddenly, they’ve just got paid. Now, from the blockchain’s perspective, no transaction has happened. It’s basically like staying the same; nobody knows that you’ve swapped the private key. But from the person who’s received that perspective, they’ve just been paid as if I’ve handed them $20. And none of this had to sync, it’s just all offline. So that’s really exciting me because that extends cryptocurrencies into an environment that they’re not meant for and allows us to do things in the maximal privacy because you don’t even know if your transaction’s happened, that’s the most private way of doing a transaction, and there are all kinds of things you can think about. You can even extend that to an offline ATM, for example. So if you can pay people offline, why can’t you have an ATM that’s offline and it can dispense cash and so forth? So trusted hardware, I think, is pretty magical, and there’s a lot of other things you can do with that. But we have a lab that is looking at these things, and there are companies like Rivetz, for example, and Intel’s involved with a protocol called Sawtooth, which is associated with the Hyperledger project, and the range of solutions from IOT (internet of things) all the way down to attestation of data and so forth. So that’s exciting to me.</p>
<p>And then in terms of PL concepts, and Duncan can probably elaborate better than I can on this, but what excites me the most is this idea of making formal specification sexy. There are some projects historically that have had huge impacts, but they’ve been extremely time consuming and incredibly expensive. Like seL4, for example, which was a microkernel that was verified. I think it took like 10 years or something like that?</p>
<p><strong>Duncan</strong>: <em>Not quite that much, but it was a lot larger.</em></p>
<p><strong>Charles</strong>: It was like five years, or a big chunk of time, and then millions of dollars were spent on it. So when you talk about formal verification software, you tend to have this vision that you’re Nasa or you’re SpaceX or you’re Boeing and you have massive budgets and thousands of engineers, and you have horizons of time that almost any rational person wouldn’t invest in normal consumer software, like five years, 10 years, 15 years and so forth. If you look at the smart contract, the smart contract is a small piece of code. Of all canonical examples you’ll be able to pull in Solidity, they’re not very large, they’re like several hundred lines of code, and they have usually pretty well-defined business requirements backing smart contract. So you can have a reasonable discussion about the semantic gap between what you intended on doing and what you actually ended up doing. So this is probably the perfect example of where you’d want to use some form of formal specification approach and verification approach to verify correctness. The other thing is there is high assurance in that if you get them wrong, hundreds of millions to billions of dollars can be lost, and also, you kind of have a very well-defined environment that these things happen to be executing in, so it’s like the perfect storm for formal specification and verification. So I’m very excited about where we can take older techniques, modernise them and drag them into our space in a lightweight way.</p>
<p><strong>Duncan</strong>: And if we’re successful, that not just having everybody write Solidity programs, then we can start with smart contract languages which are amenable to formal analysis because they’re designed with it in mind, based on proper PL research rather than just Solidity, which was accidentally designed.</p>
<p><strong>Charles</strong>: Right. And also, what’s nice is that competition is really brewing in this space. Tezos, for example, with Michelson, and I think it’s Liquidity is the name of the language there, and they’re definitely pushing hard. Simplicity is another language that’s being developed by Blockstream, by proper PL expert Russell O’Connor. And there are other examples in the spaces, end protocol is one, using concepts from F. There was an attempt to port Idris to run on Ethereum, and there’s been some formal verification work that’s occurring around Ethereum, particularly by a guy named Yoichi Hirai using Lem and a few other things. So there’s definitely some effort that’s been put into this, and that’s really exciting. If you study PL, these techniques tend to be on the fringe, and they tend to be isolated, small groups that work on these things, and nobody pays much attention to them, and when they do, they usually roll their own solution and it’s proprietary in-house or it’s for a particular project, and then you move on after it’s done. The fact that we’re now trying to take specification verification concepts and bring them into the mainstream and make them accessible to everyday programmers is a really new concept, and that’s really exciting, and I think it bodes incredibly well for software correctness. And I’m really excited about where that can go. So those are the three things: multi-party computation, trusted hardware, and PL stuff.</p>
<p>In terms of things like Hashgraph, you asked about that, there’s a lot of hype, guys, about DAGs (directed acyclic graphs) and hype about 10,000 transactions or 100,000 transactions per second, and my opinion is, most of this stuff is junk, either because it’s been deployed in a very unrealistic environment or because it doesn’t have a proper trade-off profile or because the protocol was designed without any Byzantine resistance or… you know, you have Byzantine actors or the protocol fails. And I’m very sceptical. There are some real protocols that do shard and have great performances, like, for example, OmniLedger is one and Thunderella is another, and these are protocols built within the academic world by academics, based on protocols that are already proven, or at least have some peer review behind them. So without a doubt, such things can exist, but usually when it’s happening with things like IOTA, which is based on a technology called Tangle or Hashgraph, is you see a wave of hype come along. The ICO comes, pots of money get raised, and then people psychologically buy into it, and no one really bothers to check if what they’ve implemented is right or wrong. The other thing is, you don’t really need to build a system where one chain runs 100,000 or 200,000 transactions per second; to be frank, you’re doing it wrong. Because even if you have that raw processing power, how do you move all that data? You need a network player to do that. And second, where do you store all that? If your security model is everybody has a copy of the blockchain and you’re running 100,000 or 200,000 transactions per second, how big do you think your blockchain is going to be in a year? And so, yeah, Google can store it, or the NSA (US National Security Agency) can store it; pick which one you want. It’s either California or [inaudible], you know, which facility you want your blockchain in. But you necessarily federate the network when you demand this type of performance.</p>
<p>So to me, it makes a lot more sense to have a multi-layered model where you have lots of sidechains. You have overlay protocols, things like payments channels or state channels, and you try to keep your transaction rate within a reasonable envelope. What I am excited about though is the notion of data sharding. I’d like to see ways of breaking up a blockchain and storing it like you would a torrent or storing it like you would a decentralised database. And there are some projects like Siacoin and Filecoin and others that are incredibly well funded, and they have real computer scientists behind them. In fact, Sia, for example, their core paper was given the best paper award at the Eurocrypt conference. They beat our paper; I was pretty upset about that. But that’s a high bar compared to where we were just two or three years ago with the first wave of those technologies, like Storage and MaidSafe. That’s my long-winded answer. Next question.</p>
<p><strong>Q: I think it’s a shorter question. You guys are focusing on the development of the platform, obviously, and making it available for developers as soon as possible, but do you think it might be beneficial to team up maybe with some sort of a real world project to be implemented as a new test case, as a proof of concept, let’s say, of scalability of Cardano, and implementing that application as just basically a showcase?</strong></p>
<p><strong>Charles</strong>: Yeah. It’s kind of like we’re building the Xbox. So the question is, what’s our Halo?</p>
<p><strong>Q: Yeah, exactly.</strong></p>
<p><strong>Charles</strong>: What are our games? What is the catalogue? So we do have a business development arm, and we also have a partner called <a href="https://emurgo.io/" title="emurgo.io">Emurgo</a>, which is a venture fund in Japan that seeks out projects to deploy on Cardano that will act as these proofs of concept. And we’ve had some degree of success. We haven’t been looking, but we’ve seen that there’s an overwhelming demand for pilots. I was just in Ethiopia and talking with the Ministry of Science and Technology, who we signed an MoU with, they have a very strong desire to do supply chain management on blockchain for coffee production. Now, this is a really big market. There are a million and a half farmers, there are massive amounts of stuff going on in those supply chains, so that would be a great stress test for the platform. But then, there’s an open question: what ought to run on the main network and what ought to run on a private permissioned ledger? Hyperledger exists for a reason, and IBM is not dumb. They created a really good product to service the needs of the enterprise. So part of Cardano’s remit is not just to look at what should run in a single universal environment that’s maximally decentralised, but ask, what can you deploy in a private setting? The canonical example I like to look at is the exchanges. So if you right now look at the way that exchanges handle tokens, basically, they have an address, you send your Bitcoin to that address or your Ether to that address and they have some sort of storage policy between hot wallets and cold wallets. They try to arrange it in just the right configuration, so even if the hot wallet’s compromised, the majority of funds stored there aren’t lost, which is fine but it’s not optimal. It would be much better if you used a sidechain transaction and you sent your token to a private ledger. Because you’re already trusting the exchange with your money, so you’re not really getting any extra security by being on the main network; you’re actually losing security because what you’re doing is saying to the exchange, ‘You can’t put in proprietary business logic to better protect my tokens.’ Like, for example, the ability to reverse transactions and so forth in the event that the system gets compromised.</p>
<p>So part of Cardano is to study the relationships between permissioned ledgers and permissionless ledgers and then find linkage points between these two things. So for high-stress applications, things that are … let’s say supply chain management in the coffee industry when you have a million farmers and hundreds of thousands of transactions every day and bloats and tons of data and IOT components putting sensor data on, it makes no sense in hell to run that thing on a big network and have it compete for resources with everybody else. It’s just not economically viable. It makes a lot more sense to just have the washing stations and the government and other people run consensus nodes and have a permissioned ledger there and then create some sort of linkage between the two systems. If anything, you hash the ledger regularly and store it on the main ledger; that can be an example of that. So you kind of have your cake and eat it too. You have a blockchain-esque environment, you get the timestamping, you get the automobility, you get the immutability and the record within the reason of decorum, but then you also get very low-cost operation and predictable performance of operation and a much easier time custom-tuning the ledger towards your business logic in your system. So we are looking for pilots like that, but we’re also looking for smart contracts to come on to our system, and when Plutus is fully available, we’ll be pushing like hell to get lots of people to come and do things. The advantage that we have, because we chose Haskell as our code basis, we kind of work with everybody in the Haskell space. I take it there’s like, what, four or five major Haskells that we don’t work with at this point. We work with Tweak and FP Complete, and Well-Typed and others. So we kind of know everybody there, and we have a very good brand footprint in the Haskell space. So once Plutus is available, there’s going to be probably a large group of Haskell developers who are curious to be able to write smart contracts in a language like Plutus. So that will create a wave of innovation.</p>
<p>We can also spark innovation by creating cohorts of people that are trained developers in our system. So we have an education arm, and we do classes. We did one in Athens, we did one in Barbados, and we’re soon doing one in Ethiopia. By the way, that class will be all-female developers; the government requested that. And we teach them Haskell and smart contracts. So once they learn these things, the target platform, they’ll write software for us on our system, and they’ll write lots of prototypes and so forth. So it’s a collection of these types of things. Some of the things you need to do to brand and showcase the system, but they don’t necessarily need to be run on the main network, they should be run on permissioned ledgers. Some things you do need to run on the main network, and you have to also have channels of them with which to attract developers to your system, and we have been pursuing that as a company, and our partners have also been pursuing that as a company. But it’s important to point out there’s overwhelming demand; there are lots of people who want to be doing things with us, and we have to tell them ‘no’ because we don’t have the capacity at the moment.</p>
<p><strong>Q: I’m curious … this is kind of a fuzzy area, but what do you see as the role of cryptocurrency in society? Right now, there’s so much talk of ‘Lambo’ and ‘to the moon’, and a lot of what’s going on is purely speculative work. Are we going to have our grandmothers using this on a day-to-day basis? Is this going behind the scenes? What do you think?</strong></p>
<p><strong>Charles</strong>: Well remember, not too long ago the internet revolution was treated the same way. People were making fabulous amounts of money from vaporware products, and the market collapsed. But on the ashes of that, you got the Googles and the Facebooks and the YouTubes and so forth. So cryptocurrency technology, when you decompose it into its fundamental components, is a discussion about trust and coordination, identity, reputation, the representation of value and the movement of value. These are the core components of cryptocurrency technology. So if you look at society and markets, all of these things require an opinion on how these things ought to fit together. Now what ends up happening is, you look at the default configurations we have, almost always there is some sort of middle-men of necessity, not of desire. You don’t put Bob in charge, make him the gatekeeper, because you like Bob and he happened to have built the business around it, and because his business is operating well, it becomes a core component of the way the market structure works. A great example would be eBay. Nobody really likes eBay, but eBay is eBay, you kind of have to deal with them. So you need a marketplace, you need a reputation system, you need a way of organising your buyers and sellers, you need a payment system and so forth, and eBay happened to be the winner of that fight, now they have a monopoly and they just kind of run that. Similarly, with YouTube, there’s more than one person, sorry guys, that have some issues with YouTube and getting demonetised and so forth, and maybe you guys feel you’re doing a good job – some people don’t, but no matter what, YouTube’s YouTube, it’s just of scale. Uber’s another example of that.</p>
<p>So there are hundreds of these things that when you start really looking carefully, you notice, in the flow of money, in the flow of commerce, in the flow of insurance, in the flow of commodities in every facet of life. So what cryptocurrency tech is all about is saying, can we reinvent this marketplace where we can get rid of that central broker and directly connect the buyer and seller, allow them to somehow coordinate, somehow trust each other and have a successful commercial transaction? Now a lot goes into that. You need to have things like reputation, you need to have things like insurance, you need to have a payment system, you need to have a value stable currency, which we still don’t have, you need to have credit, you need to have the ability to do settlements that you don’t even have the full amount of money for – it’s called eventual settlement, where I’m going to pay you but I don’t have the money on hand to pay you quite yet. It happens quite a lot in the world. You need to have an invoicing system. You need to have lots of stuff. But just because we don’t have all the basic components doesn’t mean there’s no merit to the system. It’s just like with the internet, you needed to have the cookie, you needed to have the certificate, you needed to have the browser, you needed to have good web languages, you need to have high-speed internet to be able to get things quickly to people. Eventually, you needed to have mobile internet and so forth. But once these infrastructural components got put in, the internet got great and changed our lives. And so, I view cryptocurrencies much the same way.</p>
<p>Now there is the natural question to ask of where does the token fit in all of this? Because everything that I’ve mentioned doesn’t really require Bitcoin; it just requires protocols and rails and so forth. So what the token acts as is some sort of incentive mechanism for people to maintain these particular types of systems. It’s not necessarily what the instrument of value’s going to be. The token just decides who gets to be in control at the end of the day, especially in proof-of-stake systems; it’s very explicit in this respect. I think any of the proof-of-work systems are eventually going to die out because they’re not sustainable or competitive in the long haul, but on proof of stake the token’s about who gets the vote, who gets to decide who’s in control. And I think what’s going to end up happening is you’re going to have the liquification of value and the tokenization of value, and we’re going to converge to a universal wallet notion. So in life, you have lots of stores of value: you have commodities and you have stocks and you have bonds, you have real estate. I have a lot of airline miles; I’m saving up enough to buy a jet. I travel 200 days a year; I’ve been to 63 countries, so I’m a pretty tired guy. We have currencies, you have whatever, you have hundreds of these stores of value. Now, we tend not to look at them as the same thing. We tend to look at them as silos, right? My gold is not the same as my airline miles. Why is that? They’re just value, they’re just wealth, and you put them all together, that’s the portfolio of your wealth. So the capstone I think of where the cryptocurrency movement is going is to remove the walls between these tokens of value and give you ways of representing them that are interoperable with each other. So you can turn your gold into silver. You can turn your silver into currency, turn your currency into airline miles. And because all the payment systems are now programmable, thanks to you guys and many others, the merchant gets paid whatever the hell the merchant wants to get paid, now. So I can walk over to Starbucks, and I can have my house tokenized, and I can sell it. There’s a market maker that lives in between that, and I sell one-millionth of my home and I can buy that cup of coffee, somebody bought that from me and the merchant gets paid in dollars or pounds.</p>
<p>So that’s where I think this is all going, that it’s going to have this ecosystem of tokens that live. The tokens that are connected to the command and control of the protocols I think will survive because they become basically like a prediction market in the sense of the utility and value of that protocol to society. It’s almost as if you could tokenize BitTorrent or something like that and say, ‘Okay, here you go. This is the value of BitTorrent to society.’ And that’s one thing, but then you’re going to have tokenization of concepts, you’re going to have tokenization of stores of value. And that’s what you’re going to end up spending on the system. So those can be government-issued, they can be synthetic in that you somehow create a value-stable cryptocurrency, they’re called stable coins and they’re traded as if they’re dollars, even though they’re not issued by a government. They can represent loyalty, you can be selfish, like you can organise your labor as an engineer and say, ‘I can take 40 hours of pre-paid labor, and people can buy it on the open market.’ You can do all these types of things. That’s where we’re going, I think, with this entire system. And that’s really cool to have a single unified global market, you have a much finer-grained way of handling trust and coordination, having universal identity and reputation space, so people can get backed and get into the system. You can know how to do business with your counterparty in a safe way, and then to have value become liquified and fluid. Now, who’s going to win that fight? I have no idea. I’m betting on myself, you know, with my protocol, but I’m not insane enough to believe that we’ll probably universally win, nor should we. It should be diverse [inaudible].</p>
<p><strong>Q: Great, Charles. Well, this has been very informative. Thank you so much for joining us at Google today and taking time out of your busy schedule.</strong></p>
<p><strong>Charles</strong>: Oh, one thing I forgot to mention, real quickly, we’re passing out these flyers. We don’t have them in your offices. Unfortunately, Richard hasn’t mastered teleportation yet.</p>
<p><strong>Richard</strong>: We have a little meetup going on tomorrow – The <a href="https://iohk.io/projects/symphony/ “Symphony of Blockchains”">Symphony of Blockchains</a> that we mentioned to you. If any of you guys on the screens are coming to London tomorrow, we’d love you to come and join in if you’ve got free time tomorrow afternoon. Charles will be talking, Lars Brünjes as well on the delegation scheme, and we’ll also be showing some art. Free bar, so come and join us if you can.</p>
<p><strong>Charles</strong>: And I want to explain real quickly that IOHK has a secret third division. So we’re an engineering company and a science company; we’re also a design company, and our head of our design office is right there. His name is Richard Wild. So it was really important to me that in addition to getting the science right and getting the engineering right that we actually find ways of representing and visualising what we’re doing. So it’s not just good enough to write code, the code has to be alive. You have to see it and be able to touch it. So our art department tries to visualise things. And so, we started with Bitcoin and we said, ‘How do we visualise a block explorer?’ So block explorer just tells you the history – what’s in this block and how many transactions were in it, how many fees were paid, etc. Wouldn’t it be so cool to put that into a three-dimensional, navigable art piece that’s on our website and eventually VR’d and you can walk through it and so forth? It makes it far more accessible to the general public because these concepts, as much as I happen to love them, are really boring. But if you have a piece of 3D art, then you can put that in a gallery and people can see this. And also, it gives you a visual way of representing things. Like, we asked a question of, what is a healthy network? Before Bitcoin Cash came, we had a big crisis in Bitcoin where every block was full, you had to wait hours for transactions to be able to confirm, sometimes days, fees were very high, it cost you $25 to buy a cup of coffee in some cases, so it’s not a very healthy state of affairs. So it would be nice to visualise that. You could just look at a blockchain and say, ‘Boy, it’s a red day. It’s not a good day. Normally we have green days, but, no, this is bad. Bitcoin’s not healthy right now.’ So it would be really cool to do that and allow people to be inspired and understand that.</p>
<p>Also, when we talk about concepts like IOTA or Hashgraph, which are directed acyclic graph-based structures versus a blockchain-based structure, how are they different, conceptually speaking? It’s one thing again to talk about the graph theory, it’s another thing to actually show people a picture or show people a model. And so, all the art we do is interactive. We use Three.js and WebGL and these types of things. So if you go to our website, you can see some of the art we’ve done, including the Symphony of Blockchains. And the event we’re having today is the first of its kind, and certainly not the last. And there are goals to build up a large portfolio of art. And eventually, we’re going to try and create incentive systems for the general public to be able to create these types of things and be able to make money from it and so forth. So you’re all welcome to come. Free beer and interesting people to meet.</p>
<p><strong>Q: Great. Thanks a lot Charles.</strong></p>
<p><strong>Charles</strong>: Cheers.</p>
<p>The above is an edited transcript of the discussion at Google’s London offices on May 14, 2018.</p>jane.wild@iohk.io