Daedalus Mantis 1.1 Released for Ethereum Classic
Software update delivers performance improvements
30 April 2018 5 mins read
It's the end of April and it already feels like a long time since February, when we announced version 1.0 of Mantis, our Ethereum Classic client built in Scala. After the success of Mantis 1.0 some of the Grothendieck team got temporarily drafted onto other projects. That, coupled with the two-month full-time Haskell training course some of the team were on earlier this year, meant that Team Grothendieck has been short-handed recently. In fact, internally this release is sometimes called "The Konrad Release" as it was developer Konrad Staniec who kept the 1.1 candle burning and the performance pull requests (PRs) coming.
For review of the PRs we did of course leverage the expertise of the whole team, and special mention to Alan Verbner and Nico Taller for their efforts in review and testing of the performance improvements to the complex pruning functionality.
While I'm listing contributors, many thanks to Lukasz Gasior and Radek Tkaczyk for taking time to review PRs early on and especially to Carlos Montero and Jeremy Townsend, two new IOHK developers who jumped head first into the Mantis code just as the test cycle was starting and were invaluable in reviewing PRs and testing the JSON RPC API. Also the testing team's Alan McNicholas for giving the release candidate a whirl and finding an installer bug! That's quite a lot of shoulders to the wheel.
The objective of the 1.0 release was to create a working product, while the 1.1 release aimed to take the working code and find and remove the performance bottlenecks. The most painful bottleneck identified was in synchronizing the blockchain. This was complicated by the fact that tuning that performance depends on quite a few factors, the speed of the network, the type of hard disk on the machine and the number and type of peers at the time of synchronizing.
For example on MacOs with an SSD and 8Gb of RAM Konrad was consistently getting about 17 hours for a full synchronization, whereas on an "small" EC2 instance this could take over a week! One of the reasons for this is AWS t2.small instances can be "limited" or "unlimited" referring to their CPU credits per hour. Once the limit of CPU credits has been reached, the synchronization slows down considerably. Our developer Jeremy Townsend wrote this up and it turns out this can be improved by using a "compute optimised" EC2 instance because the software now spends most of its time actually verifying blocks of transactions and those crypto functions are compute expensive!
Apart from performance, the "difficulty bomb" ECIP 1041 has been disabled – just in time too as the block where this becomes important rolls around in early May.
There have been no changes to the wallet interface in this release. A couple of fairly minor changes were considered but the Daedalus team is flat out and while they are on the team’s backlog list they did not rise sufficiently in priority for inclusion in this release. The only real implication of that in this release is how the block download progress is reported. It was quite confusing for users to see a 0.0% progress bar for such a long time. The reason is the synchronization for ETC is different to the synchronization for ADA, in that ETC implements 'fast sync' and ADA does not.
Fast sync downloads the state trie and the blocks in the blockchain. Ada has no fast sync and downloads only the blocks. In 1.0 the state trie was downloaded before the blockchain and when the state trie was fully downloaded the blocks began to download. The progress bar is only aware of the blocks and so continued to show "0.0%" while the state trie was being downloaded. In 1.1 the situation is better but not perfect. In 1.1, as a result of performance testing and improvements the block downloading begins straight away and the state trie downloads towards the end of the synchronization. The user will see the progress bar update as expected, however towards the end of the synchronization the progress will appear to stall. This happens while the state trie downloads. While this is not perfect and needs to be fixed, on the plus side the whole process is faster and so the frustration should be less. Thank you in advance for your patience with this.
The next release will be Mantis 2.0, currently slated for the end of the year, around Q4. This will be a significant release with significant new functionality. The Project Manager for this release, Ravi Patel, and a full-strength team will be introducing this functionality as it is prioritized.
On a general note, I feel the progress in ETC is picking up pace. Observing the external ETC community it seems there has been a lot of good organisational work done and dedicated people involved. I'm more optimistic than ever about the future!
A Brief Update on Cardano Development
Processes are evolving under the project management team
9 April 2018 10 mins read
After returning from my yearly global sojourn, I wanted to update the Cardano community on the status of the project. Since the beginning of the year a lot has happened. Cardano continues to grow at a rapid pace and the project is evolving into a new stage. The Byron release back in September of 2017 was an experiment for IOHK. It's the first cryptocurrency we have launched as a company. It's the first time we've had to manage public release cycles, segregated stakeholders (the general public, exchanges, developers, etc). Furthermore, the Cardano project has half a dozen software companies collaborating on it, thus we have to invest a huge amount of time in coordinating, communication and timezone overheads.
Since the September release, we've learned a huge amount about all of these processes and also in dealing with the needs of our broader community. We certainly haven't achieved a Nirvana-like state of perfection, but processes have definitely improved.
I'd like to share some of these improvements as they are mostly hidden from the general public, yet have a huge impact on our ability to deliver a long-term roadmap. First, since September, IOHK has built a tremendous amount of project management capacity, led by Elieen Fitzgerald.
Under her department, Eileen has managed to capture business requirements, draft project charters, improve our resource allocation and budgeting processes, improve development estimates, get better weekly reporting and manage the inter-dependencies between projects. We have also had an increasingly easier time managing third-party relationships, like our partnership with Runtime Verification for the K framework, IELE, and smart contract research.
Some of the outputs of these processes are that we are moving to regular release cycles for Cardano, with the first cycle starting next week on Friday. Our hope is to cut a develop branch for release and then run the release through a rigorous QA cycle currently planned for one month. Over time, this cycle can be shortened through automation and parallel processes speeding up delivery. Updates will become more frequent, higher quality and encumber less user disruptions.
The goal of the project management department is to ensure when we give a date for delivering a feature, a release or a major update that the date and quality are delivered. This goal is one of the hardest to achieve for a software company and much more so given the nature of the software we build, but it's also incredibly important for those who rely upon us for their own commercial interests.
Over time our project management methodology will become increasingly public and eventually will be ported into a public GitHub repository. We'd like IOHK to follow a creative commons process that other software companies and projects in our space can benefit from, and add to, as they pursue their own projects. We'd like to explore lighter weight versions of the processes for DApp development so that our developer community can follow best practices.
Second, working with exchanges and others, such as Ledger, we've been systematically redesigning Cardano's architecture, APIs and other components to be more friendly for those who wish to use our software. For example, you can see our new APIs here.
Another output has been moving our development towards a specification-driven process. The first component of the Cardano Settlement Layer to be ported into this process is our wallet backend, with the following formal specification:
Figure 1: Cardano SL's Formal Wallet Specification
The goal is that each part of Cardano will be specified in a format similar to the one above. These specifications are implementation independent, will eventually be analyzed using formal methods and can be used as a basis for test suites and improvement proposals.
We are aware that many want to build their own mobile clients or modified software. To this end, we've been exploring the best way of discussing a unified backend architecture. I would personally like to see dozens of wallets and great user experiences materialize for Cardano, but I'd also like to make sure that these experiences are useful, secure and easy to deploy.
Over the coming months, large chunks of Cardano's code will be ported over to, or replaced with, these specification-driven designs. This likely won't be directly noticed by our users. Rather, users will experience indirect symptoms, for example things getting faster, like wallet recovery from seed, fewer issues connecting to the network, a smaller memory and disk footprint as well as other improvements.
As we now have the talent, processes and clear roadmap, Byron will continue to rapidly improve and become more feature-rich. The release we are cutting next week to QA will include paper wallets, much faster wallet restoration and numerous fixes. We expect it to clear QA in mid-May and for updates to be released to our users monthly thereafter.
Third, the most anticipated upcoming release is Shelley. Shelley is a massive project with many workstreams and scientific dependencies. It also contains many social processes involving community coordination and management. Effectively, Shelley is about turning over the network fully to the users thereby decentralizing as much as possible.
The work we have done with Byron has given us a great deal of knowledge operationally, about the best way of iterating towards Shelley; however, special care has to be placed on consensus. IOHK has developed a custom, proof of stake protocol called Ouroboros for Cardano. It has never been used in a cryptocurrency before and has a completely original design.
Thus we have been extremely focused on a proper deployment of Ouroboros to the general public. Byron is running a version of Ouroboros with delegation locked to core nodes under the control of IOHK, the Cardano Foundation and Emurgo, and block rewards turned off, but when Shelley comes this cannot continue. Staking rights will be returned to Ada holders and delegation will be fully under their control.
To be clear, Ouroboros isn't a forced, delegated, proof of stake protocol like EOS or Bitshares. It's a pure, proof of stake protocol where for epoch elections every active Ada account is factored in. Anyone who holds Ada in a normal address in the global UTXO has a probability of being elected as a slot leader regardless of the amount of Ada they hold.
However, the reality is that most will not want or have the ability to host consensus nodes and consistently show up to fill the slots they have been elected to commit. Thus, we developed a delegation system and the concept of stake pools for those users.
Briefly, anyone can run a stake pool. There isn't a minimum threshold of Ada or a special club. Rather, there will be a blockchain-based registration system and a special transaction type to register a stake pool on-chain. Registered pools will be listed in the delegation center of Daedalus and pulled directed from the Cardano blockchain thereby preventing censorship or bias.
Over the last few months, we've had to invest a huge amount of careful design and security thought into the process of delegation. There are dozens of factors and scenarios to consider, from cold staking to automation of rewards. But we have converged on a reasonable design for the Shelley release, and a paper will be released soon on eprint.
The summary is that Ada holders can create a delegation certificate for the Ada they hold and register it on the Cardano blockchain. This process in effect separates stake rights from the spending keys for their Ada addresses. Thus the delegation certificates can live in Daedalus, but the spending keys could be kept offline on a paper wallet or ledger device for example.
Delegation will be done through a special transaction and from a user experience viewpoint, via the delegation center in Daedalus. A user can find a stake pool they wish to delegate to, select it, and click a "delegate" button. It's just that simple. As we launch Shelley testnets, we'll experiment with different user experience flows, from length of delegation to partial delegation (splitting stake between pools).
Another advantage of this process is that unlike Bitcoin mining pools, as our protocol natively understands delegation, it can automatically pay rewards to those who delegated without having to trust the pool operator. At the end of each epoch, our goal is to close the reward pool through a special transaction, paying all those who delegated proportionally to their delegation amounts.
Some benchmarks and threshold will have to be conducted over the coming months to optimize for space and prevent penny-flooding attacks. We will also need to explore different user experiences including notifications and other user interface considerations.
There are natural questions to ask. How will we ensure that when Shelley launches there will be enough stake pools to ensure a reasonable amount of decentralization? How will these pools establish their brand and reputation? We considered these concerns and decided to open enrollment for a collection of beta testing stake pool operators.
The goal with this process is to identify a set of 50 to 100 independent entities that are geographically well distributed who would like to run a stake pool as a business. The process will progress as follows:
- Collect as many applications as possible at https://staking.cardano.org/ until the end of April
- Process and winnow applications until we have a good set of 50 to 100 candidates
- Invite the candidates into the IOHK slack and begin discussions about hardware configurations, deployment strategy, docker images, etc
- When the Shelley testnet is launched, invite the stake pools to register and work closely with them to beta test various scenarios and experiences
These beta testers will not get a special advantage or consideration when Shelley launches. They are necessary to test Shelley's design and ensure our assumptions and choices are reasonable as well as improve deployment strategy and documentation. When Shelley launches, there will be a grace period where all those who desire to register a stake pool can do so and Ada holders will be free to choose to delegate to anyone they want.
Once the grace period expires, auto-delegation will be turned off and rewards turned on: Cardano will be fully decentralized.
In closing, Cardano is a huge project. There are so many brilliant minds, great engineers and parallel efforts that it's difficult to capture all of it in a single post, much less just convey our progress. What's amazing to me is that we have really great processes established and are daily moving forward (a fan made a great website showing our daily commits: https://cardanoupdates.com/).
What's also amazing to me is how quickly our scientific research is moving from the lab to code. Ouroboros has gone through over a dozen revisions and now is converging to a state where we can bootstrap from the genesis block without a checkpoint (an industry first for proof of stake). Our sidechains research is state of the art and a paper is coming in May.
We have also brought game theorists and programming language theory experts together. The output has been incredibly innovative with new accounting languages like Marlowe and an increasingly richer theory for incentives for stake pools, network maintenance and other topics requiring an honest majority for cryptocurrencies to run properly.
Figure 2a: The Marlowe Programming Language
Figure 2b: The Marlowe Programming Language
I'm astounded how we are able to think in systems now and by the quality of the people on the Cardano project. It's taken years to build this team and go from dream to regular status reports. I look forward to achieving our milestones and seeing Cardano change the world.
Artwork, Mike Beeple
Haskell and Cryptocurrency Course in Barbados
Two graduates share their experience of IOHK's functional programming course
7 April 2018 7 mins read
The IOHK Haskell and Cryptocurrency course in Barbados brought together students and professionals who were interested in learning the Haskell programming language. The course ran for eight weeks at the University of West Indies. Barbados was the second time the programme was offered, after a successful inaugural course held in Athens last year. The goal of the course is to extend the computer science training of participants and introduce them to Haskell, an elegant functional programming language, in the context of the cryptocurrency industry. Haskell is the language used in the Cardano cryptocurrency, and it was chosen because of the security benefits it offers.
The course was designed and taught by Haskell and functional programming experts, including myself, and Dr. Andres Löh of Well Typed and Dr. Marcin Szamotulski, Haskell Developer at IOHK. Visiting lecturers included Prof. Philip Wadler, one of the creators of Haskell, and IOHK Area Leader in Programming Languages, and Cardano SL developer, Darryl McAdams. The instructors aimed to strike a balance between theory and practice, teaching the students both theoretical background of functional programming in Haskell in particular (Lambda Calculus, System F, Category Theory, etc), but also introducing them to popular libraries and important techniques for solving real-world problems (networking, parsing, resource handling, and more). At the end of each course, students are given the opportunity to apply for positions at IOHK and continue their professional career in Haskell.
Here are the stories of two students who attended the IOHK Haskell and Cryptocurrency course in Barbados:
Jordan Millar - Jordan is currently a chemist with an M.Sc. in organic chemistry from Oxford University.
Upon hearing that IOHK was offering a free Haskell and Cryptocurrency course in Barbados I decided it would be foolish to not enroll. The stars aligned for me as Barbados is a stone’s throw away from Trinidad and Tobago, my home, and I was acutely aware of IOHK’s approach to cryptocurrency.
My programming experience prior to this course was limited; I predominantly had written scripts in Python over the last two years. Only a few weeks prior to the course, I had become interested in Haskell through a functional programmer I happened to meet.
IOHK piqued my curiosity because at the time, they were the only company in the cryptocurrency space driven by research. This resonated with me as I had first-hand experience with research and development, albeit in a different field. I work in organic chemistry, and am currently working with a cleaning product manufacturer on their formulations. I appreciate the importance of developing a model, trying to break it yourself, and then having people smarter than you also trying to break it. In particular, if these networks are destined to hold hundreds of billions of dollars, you cannot afford to construct these networks in the ad hoc manner we have often seen before. With this in mind, I set off to Barbados to learn more about Haskell and IOHK’s methodology.
The course was a whirlwind tour of Haskell starting with data types all the way to type families and everything in between. We charged through a series of topics and on a weekly basis were given a problem sheet to complete. Dr. Lars Brünjes and Dr. Marcin Szamotulski were exceptional teachers. They delivered the content effectively and gave additional clarification when required.
The course was challenging, especially because I had missed the first week due to logistical issues. That being said, I came to find that Haskell is an elegant and concise programming language. IOHK flew Philip Wadler to Barbados to give a couple of lectures! What stood out to me the most was his demonstration of the Curry-Howard correspondence. It was fascinating to see the link between mathematical proofs and programs. IOHK’s Darryl McAdams also visited us in Barbados and took us through compilers in Haskell. The elegance of Haskell really shone here in my opinion as she effortlessly created a simply typed programming language during class.
It wasn’t all work and no play; IOHK sponsored many dinners and outings while we were in Barbados. This rounded off the course nicely as everybody had time to socialize outside of the classroom. To top it off, Charles Hoskinson took time out of his incredibly busy schedule to come and see us in Barbados. Having only seen Charles in YouTube videos, it was a privilege to hear him share his future plans and vision for IOHK in person. If you thought his conviction transmitted effectively through his interviews online, wait until you hear him in person.
All in all, it was a great experience. To anybody reading this, I highly recommend enrolling in this course if you get the chance to!
Rob Cohen - Rob divides his time between his information security consulting firm Callidus Security and serving in the cyber security field in the US military. He graduated from Columbia University in 2015 with a BA in Computer Science and Mathematics.
I was fortunate to be invited to attend IOHK's eight-week course, "Haskell and Cryptocurrency" in Barbados in early 2018. Prior to the course, my only experience with functional programming had been in a Compilers course I had taken in college and some toy projects I had built as experiments. As a result I knew this course would be challenging, and it turned out to be exactly the kind of deep-dive crash course on Haskell and functional programming that I was hoping for. This course was demanding, requiring dedication in the classroom and diligence in applying those concepts in our assigned homeworks and group projects.
The course covered a variety of topics, from the basics of IO and higher-order functions, to lambda calculus, optics, free monads, GADTS and Generic Programming concepts (and much more). Beyond the theory, what I found most rewarding in the course was working with my fellow classmates on our group projects. One project my team worked on involved building the early stages of a working Bitcoin client in Haskell! Working on larger-scale projects like that really helped Haskell come alive for me. Furthermore, guest lectures by Phil Wadler and Darryl McAdams from IOHK's Plutus team were pleasant surprises as well.
This course was not easy; there were times where I really felt my mettle was challenged. However, I stuck with it and pushed through. Now I not only have a deeper understanding and appreciation of the theoretical underpinnings of Haskell, but I came out as a developer with a stronger constitution as well. I am grateful to Lars and Marcin (the course instructor and teacher assistant) for all their hard work on the course. They put a tremendous amount of effort into the course materials and it showed. There were frequently times where their diagrams and explanations were better than Haskell's own official documentation!
The highlight of the course was the opportunity to work closely with such incredible developers from all around the world. My classmates hailed from Japan, Ireland, Argentina, Poland, Germany, Sweden, the USA, and the Caribbean. It was truly remarkable to have so many people from all over the world working together in one room trying to figure out how to learn Haskell. To my fellow classmates, I am deeply grateful for your support and camaraderie during the course. I salute you all for your dedication and hard work. I also look forward to our (spearfishing) reunion someday!
I can safely say that I've been bitten by the Haskell bug, and I look forward to being a member of the Haskell/functional programming community for years to come. I highly recommend this course to anyone who is interested in learning about Haskell and functional programming.