2017.43 Hyper lands, racing…

The past week saw reliable support for .hyper and .race arrive, as described in Jonathan Worthington‘s blog post earlier this year. Now some operations can be made about 3x faster with just a few keystrokes. To give an example, these are some ways of obtaining the prime numbers below 100000:

$ time perl6 -e 'my @primes = ^100000 .grep: *.is-prime'
real 0m26.428s
user 0m26.440s

Where real is the wallclock time that passed, and user is the amount of CPU-time that was used. Now, by adding a simple .hyper:

$ time perl6 -e 'my @primes = ^100000 .hyper.grep: *.is-prime'
real 0m9.288s
user 0m31.354s

We go from 26 to 9 seconds wallclock time, which is almost 3x as fast. And if you’re not interested in the order of the final result, you can use .race:

$ time perl6 -e 'my @primes = ^100000 .race.grep: *.is-prime'
real 0m8.335s
user 0m32.106s

Which is well over 3x as fast.

A more real-world example is the test-t.pl test of Text::CSV. The --hyper version typically runs for 1.14 seconds, whereas the serial (normal) version runs for 2.57 seconds (timings all made on the quadcore machine that yours truly is using, so YMMV).

To execute code that only needs to be run once in a batch, you can use a once block:

$ perl6 -e '^100 .hyper.map: { once { say "started with $_" } }'
started with 0
started with 64

Unfortunately, it is not yet possible to use phasers inside hypered/raced blocks, mainly because we still need to figure out when e.g. a LAST phaser should fire: at the end of a batch? Or after the final call of the final batch?

Casual Contributors Welcome!

Ann Barcomb describes a number of useful strategies for managing what she calls “episodic contributors” in her blog post: “How to manage casual contributors to open source projects. I think we can all learn from this, so recommended reading!

More Perl 6 Videos

The uploads page of TPCiA videos shows some more Rakudo Perl 6 related videos this week:

Also, Rakudo Perl 6 – A Primer by Simon Proctor (from the last London Perl Mongers Technical Meeting). Enjoy!

2017.10 Release Delayed

Some showstoppery issues with Rakudo are delaying the 2017.10 release. As this release will most likely be used for the next Rakudo Star release, and thus will be used longer and more extensively, it was considered better to wait until these issues are dealt with, at least temporarily. So stay tuned!

Other Core Developments

As promised last week, a roundup of other core developments of the past 2 weeks:

  • Several issues with quoting in is-run on Windows were fixed by Zoffix Znet, allowing make test to run cleanly again.
  • Samantha McVey made IO::Path.parent take an Int to indicate how many levels to go up (default: 1).
  • David Warring made indexing into NativeCall‘s TypedPointer (aka AT-POS) at least 4x as fast.
  • Daniel Green improved the speed with which variables are interpolated in regular expressions (aka the internal INTERPOLATE subroutine).
  • Zoffix Znet fixed IO::Pipe.close so that the Proc object is always returned, instead of only by the .close on the last open handle. He also fixed some issues with Promise.in, and Promise.at, and very low values / negative values / values in the past.
  • Timo Paulssen added Promise.kept and Promise.broken as a quick way to generate a kept or broken Promise without needing to call Promise.new.
  • Zoffix Znet also fixed precedence issues with the Unicode versions , , and of the ASCII operators <=, >= and !=.
  • Elizabeth Mattijsen fixed an issue caused by a mismatch in the layout of the BUILDPLAN between nqp and Perl 6. She also improved the speed of initialization of native arrays and fixed issues with the handling of Failures by some set operators.
  • And many, many, many more smaller fixes and improvements, both internally as externally visible. Interested in seeing more? Check the ChangeLog or the latest commits.

Blog Posts

Meanwhile on Twitter

Meanwhile on StackOverflow

Meanwhile on perl6-users

Meanwhile on PerlMonks

Winding Down

Alas, no recent module information on CPAN just yet. We’ve been all too busy fixing bugs before the 2017.10 release. But you should check out the new MetaCPAN-like search functionality of modules.perl6.org! And while you all do that, I’ll be preparing for next week’s Perl 6 Weekly. See you then!


2017.42 Taking Ticketing Seriously

Aleks-Daniel Jakimenko-Aleksejev has been ticketing away! The result can be seen in an overview of status changes in RT tickets of the past week or so. He explains it like this:

The high number of [REGRESSION] tickets is caused by the concentrated effort to detect any unintentional changes in Rakudo Perl 6. I extracted all messages to the evalbot (e.g. “m:” on the #perl6 channel) from the last few years, and passed these code snippets to different versions of Rakudo Perl 6. These snippets were filtered by comparing the output and exit code, which resulted in a list of thousands of samples that produce different results on different versions. Most of these differences are due to bug fixes or other harmless reasons (e.g. current time being shown in the output), but there were some notable regressions. All regressions noticed during the manual inspection are now filed, and some of them are already resolved (with tests, of course). This is the second time we see dedicated effort to find otherwise hard-to-notice regressions, first time being in December 2016.

So it should be easier to keep the sanity of Rakudo Perl 6 in check!

One more TPCiA Rakudo Perl 6 Video

Samantha McVey‘s High End Unicode in Rakudo Perl 6 has arrived! Of course, you can still keep checking for more uploads in the coming weeks.

Blog Posts

Meanwhile on Twitter

  • Please welcome, CPAN6 by builtinperl.
  • Rakudo Perl 6 BuildPack by brian d foy.
  • 20% shaved off by Zoffix Znet.
  • Debugging an Amazon issue by Moritz Lenz.
  • Whole day Rakudo Perl 6 Tutorial at TPCiG by Jeff Goff.
  • !perl6 at DuckDuckGo by Zoffix Znet.
  • int32.new shouldn’t give you an Int by Zoffix Znet.
  • Meanwhile on StackOverflow

    Meanwhile on perl6-users

    Meanwhile on PerlMonks

    Lately, there have been some Rakudo Perl 6 related Meditations on PerlMonks. Yours truly feels they should be mentioned in the Perl 6 Weekly as well, although some troll hugging may be needed:

    Ecosystem Additions

    This week will probably be the last time you will see the ecosystem additions mentioned in the Perl 6 Weekly. Thanks to Zoffix Znet and Aleks-Daniel Jakimenko-Aleksejev we will soon have a page on that will show the recent uploads to CPAN (the new preferred way putting your module out into the wild). So here goes for the last time:

    Winding Down

    Being too tired to investigate the latest Core Developments of the past week, I will keep them from you until next week. Which will also coincide with the 2017.10 release of Rakudo Perl 6! So please check in again next week!

    2017.41 The Case for Empathy

    Yours truly is glad she can finally link to Ruth Holloway‘s impressive The Case for Empathy keynote at The Perl Conference in Amsterdam. Please excuse the poor sound quality for the first 7 minutes or so: kudos to Amanda who managed to recover and process the sound from that period and supply the subtitles.

    Other videos of TPCiA related to Rakudo Perl 6 are:

    It is expected that more videos will be uploaded in the coming week, so please check again from time to time!

    CPAN6 is here

    Zoffix Znet did a lot of work on getting Rakudo Perl 6 modules out of CPAN, just as he promised last week. He explains it all in his blog post CPAN6 is here (Reddit comments). So now we finally have a front-end to browse the available Rakudo Perl 6 distributions that have been uploaded to CPAN!

    The Second Bug Squashathon

    Last weekend saw the second Community Bug Squashathon, this time focused on fixing problems in the core of Rakudo Perl 6. And the results are in! Looking forward to the next Squashathon, on 4 November 2017!

    Inside the head of Larry Wall

    Joe Armstrong conducted a Q&A with Larry Wall at the Erlang & Elixir Factory SF Bay Area 2017 conference (Hacker News comments). Also recommended as a very nice intro to Rakudo Perl 6 and the thoughts that have gone into the 100 year language.

    Rakudo Perl 6 performance analysis tooling

    Timo Paulssen‘s grant proposal for Rakudo Perl 6 performance analysis tooling has been approved by the TPF Grant Committee. Yours truly can hardly wait to see the results of this grant come to fruition!

    Call for Rakudo Perl 6 Papers!

    This is the time to propose your Rakudo Perl 6 presentations! Please select one or more conferences:

    Core Developments

    • The new JIT was merged into MoarVM, and consequently into Rakudo’s main branch. Some teething issues were already fixed by Jonathan Worthington. The new JIT hasn’t yet given us any large speed improvements, but now more people can work on making the JIT smarter and thereby making Rakudo Perl 6 faster! Yours truly is looking forward to be able to help here. Interested in working on this as well? Check out the documentation.
    • Zoffix Znet, with some help of Dagfinn Ilmari Mannsåker made EVAL/EVALFILE aware of Buf and the utf8-c8 encoding, as was originally stipulated in S29. He also worked on making .gist on several classes a lot faster, and re-enabled the :bin parameter on slurp.
    • Daniel Green made sure that enum values are now also included in errors suggesting identifiers when an identifier could not be found.
    • Larry Wall improved several parsing error messages, specifically on the likely location of the error, e.g. by catching missing expressions on statement modifiers earlier. Christian Bartolomäus and Aleks-Daniel Jakimenko-Aleksejev did similar work on improving the error message formerly known as “Malformed my”.
    • Zoffix Znet added a --repl-mode parameter to Rakudo Perl6 6, specifically meant to make testing the REPL easier. He also created new REPL tester logic and adapted the tests to use the new REPL tester logic.
    • Aleks-Daniel Jakimenko-Aleksejev also improved many error messages related to mismatched quotes.
    • Jonathan Stowe provided several fixes related to NativeCall and also provided tests for those fixes.
    • Elizabeth Mattijsen made object creation where attributes have compile-time default values (such as has $.foo = 42) about 40% faster. She also improved bare startup time by making code, that would normally run during startup, be pre-compiled in the setting.
    • Zoffix Znet, with some persistent help of Aleks-Daniel Jakimenko-Aleksejev, made array slices with a lazy infinite list (such as @a[3..*]) about 4x faster.
    • And many more smaller fixes and improvements.

    Other Blog Posts

    Meanwhile on Twitter

    Meanwhile on perl6-users

    Winding Down

    Quite an eventful week again, with the JIT being merged and the Squashathon going on. More than 100 commits in the rakudo repository alone. Please check in again next week for more Rakudo Perl 6 news!

    2017.40 Unicode Granted

    Samantha McVey has completed her work on the “Improving the Robustness of Unicode Support” Grant. In an extensive blog post she describes what she has done in the past months, how all achievable goals were achieved, and how quite a number of bonus features also made it. You can leave any congratulations, support or other comments on the official TPF page about the completion of the grant. I think we can now safely say that Rakudo Perl 6 has the most complete Unicode support of any programming language in the world. Kudos to Samantha!

    Object creation 1.5x faster

    Elizabeth Mattijsen (with a lot of help by Timo Paulssen and Jonathan Worthington) completed her work on auto-generating the BUILDALL method for each class. This is an internal method that is called to create an object using the default .new method. Instead of interpreting a build plan of the class every time an object of that class is created, a custom BUILDALL method is created directly using QAST ops (basically writing source code in Abstract Syntax Trees and then “EVALling” them). This makes each call to .new 1.5x faster, making our canary about 8% faster!

    Nativecall also much faster

    Stefan Seifert also finished his work on auto-generating native call function bodies, making the interface between Rakudo Perl 6 internals and Inline::Perl5 much faster. This also made the Inline::Perl5 version of our canary about 10% faster. The race continues to be on!

    Roast Versioning

    Rakudo Perl 6 has a file that contains a list of test-files (roast) it should run (spectest.data) when doing a so-called spectest. A single file used to be used for all language versions, which is bad because:

    • New files get added to roast master and roast runner complains they’re missing when running tests for older language versions.
    • Files in master get restructured, but deleted files still have to live as dummy files when tests for another language version get run.

    Zoffix Znet implemented roast versioning by making a separate list of files for each language version so that Rakudo Perl 6 can run tests for each language version seperately. Roast runner just looks at VERSION file in roast to figure out which version of the list of test-files to load (the file list is now in spectest.data and spectest.data.6.c; for a future release a spectest.data.6.d file will be added).

    The language that will not be named

    Zoffix Znet rekindled the discussion about the naming of Rakudo Perl 6 again, this time suggesting 6lang (to be pronounced slang). Again, quite some discussion on blogs.perl.org, /r/perl6, /r/perl and Twitter. Yours truly hopes we can settle this once and for all before Rakudo Perl 6 language version 6.d comes out (or whatever we will call it then).

    Hacktoberfest is here again

    It’s the month of October again and your chance to win a T-shirt! Simply sign up, have a look at the issues at hand, fix only 4 of them and profit!

    Community Bug SQUASHathon

    On Saturday 7 October we will have yet another Community Bug SQUASHathon. Please join us on #perl6-dev on freenode and help us squash those pesky core bugs.

    Texas to ASCII

    To avoid any political connotations, it was felt that we should be moving towards documenting operators consisting purely of ASCII characters as “ASCII” operators (such as (<=)), and operators that consist of at least 1 non-ASCII character (such as ) as “Unicode” operators. Internal documentation has already been adapted, external documentation will follow shortly unless consensus changes.

    Other Core Developments

    • Zoffix Znet made chainable operators (such as <: 2 < 3 < 4) between 6x and 36x times faster (depending on number of items in the chain). He also made cmp on Version objects about 7x faster. Which is nice as we will need to do a lot more language version comparisons in the future!
    • Jonathan Worthington fixed several race conditions and deadlocks with the new ThreadPoolScheduler. He also made sure that all aspects of the new ThreadPoolScheduler are now used on 6.c, except the part of await no longer blocking the thread it is awaiting in.
    • Samantha McVey fixed an issue with uniname and non-unique Unicode names. The noncharacter property is now also returned for noncharacters.
    • Zoffix Znet also fixed and improved the ** regex quantifier and made sure that chdir respects the :CWD named parameter.
    • And quite some other smaller fixes and improvements.

    Other Blog Posts

    Meanwhile on Twitter

    Meanwhile on StackOverflow

    Meanwhile on perl6-users

    Ecosystem Additions

    Winding Down

    Bart Wiegmans has created a Pull Request for all of the work that has done on the new JIT: this is now going through the final reviewing process. This work will be merged any day now, which should allow for quite some more “so many percent faster” items in the next issue. So please check in again next week!

    2017.39 Smarting up the Pool

    Jonathan Worthington elaborates on his new ThreadPool implementation in an extensive blog post, which was made possible by the sponsoring of Vienna.pm. A recommended read if you want to keep up to date on the features and performance of asynchronous processing with Rakudo Perl 6! In short, there are now 3 queues internally: one for timers, one for workers that sequentially process, and one for general purpose activity. There’s now also a “Sufficiently Smart Supervisor” thread that looks at the state of the other threads about 100x per second and decides if adding more threads would make sense or not. Exciting times!

    Performance analysis tooling

    Timo Paulssen submitted a grant proposal to improve the performance analysis tools of Rakudo Perl 6. Judging by the comments so far, he’s not the only one wanting to see this happen. Please leave your comments if you also have an opinion about this proposal!

    Other Core Developments

    Jonathan Worthington was responsible for the most of last weeks commits. I guess everybody else was just watching in awe, or busy with things that didn’t make it into the main rakudo repo just yet. Notable exceptions are:

    • Samantha McVey finally fixed the last bugs in the utf8-c8 encoding: this is the encoding that tries to encode as utf8, but which creates temporary synthetic codepoints for those sequences that are not valid UTF-8, instead of giving up.
    • David Warring made sure that subtypes actually have an .isa method.
    • And some other minor fixes and improvements.

    Other Blog Posts

    Meanwhile on Twitter

    Meanwhile on StackOverflow

    Meanwhile on perl6-users

    Ecosystem Additions

    Winding Down

    Apart from all the visible work Jonathan and Samantha have done this week, there was a lot going on behind the scenes. One of them promises to make object creation in Rakudo Perl 6 at least 10% faster. Too bad we couldn’t report about it this week. So be sure to check out the next issue of the Perl 6 Weekly!

    2017.38 Color Me Booked

    There’s been a lot of book activity in the Perl 6 world lately. Andrew Shitov announced his new book Perl 6 Deep Dive (preliminary table of contents). And Moritz Lenz also continued working on his “Parsing with Perl 6 Regexes and Grammars” book. To top it off this week, Zoffix Znet announced the Rakudo Book Project – a plan to write some Rakudo books (/r/perl and /r/perl6 comments).
    Butterflies Galore!
    Please check out his plans and support him in any way you can!

    AlexDaniel++ for his second release

    Aleks-Daniel Jakimenko-Aleksejev has done his second Rakudo compiler release! The announcement for Rakudo Perl 6 2017.09 shows quite a number of fixes and improvements again this month. Please note there are currently no plans for creating a Rakudo Star release for this compiler release.

    London Perl Workshop – 25 November

    Saturday 25 November will see another London Perl Workshop 2017. And yours truly would love to see a lot of Rakudo Perl 6 presentations there: so please submit your presentation. Hope to see you there!

    New ThreadPoolScheduler implementation

    Jonathan Worthington started work on a new thread pool scheduler (which got merged after the 2017.09 release because of possible ecosystem fallout). This implementation has separate general and timer queues with separate workers, and also introduces affinity queues, which are intended for cases where events will be fed into a Supply, and thus there’s no point having lots of threads competing over them only to immediately stumble over each other. The separate timer queue helps when timer events are being delayed, for example if a process is producing a load of output.

    This implementation also adds a supervisor, which is where the smarts on how many threads to have in the pool will be put. For now, it is already smart enough to start a lot less threads than the previous scheduler when they obviously aren’t needed. This helps with memory consumption. And it can add more threads on demand when needed to break deadlocks. The default maximum number of threads has been raised to 64, now that the scheduler does not start up the maximum number of threads even when they don’t have any work to do.

    For debugging, the RAKUDO_SCHEDULER_DEBUG and RAKUDO_SCHEDULER_DEBUG_STATUS environment variables can be set.

    This work has been kindly sponsored by the Vienna Perl Mongers.

    Other Core Developments

    These features made it to the 2017.09 compiler release.

    • Samantha McVey marked the .collate method, coll and unicmp infix functions as no longer experimental. The dynamic variable $*COLLATION, which allows you to configure the sort, will remain as experimental for now.
    • Elizabeth Mattijsen made simple (one attribute) object creation about 25% faster, and reduced memory requirements for classes.
    • Philippe Bruhat worked on .succ and .pred on enums. This resulted in more attention to this subject, with the outcome a .WHICH on enums that is O(1).
    • Timo Paulssen fixed a closure issue with permutations and combinations.
    • Aleks-Daniel Jakimenko-Aleksejev fixed needless buffering in Test, causing test information not be updated continuously when running the test in a harness.
    • And quite a few smaller fixes and improvements.

    Other blog posts

    Meanwhile on Twitter

    Meanwhile on StackOverflow

    Meanwhile on perl6-users

    Ecosystem Additions

    Quite a nice catch this week!

    Winding down

    Between the rain and the wind, quite a lot happened in the Rakudo Perl 6 world yet again. Sometimes we forget how many ways we found how not to make Perl 6. With that in mind, see you next week for more Rakudo Perl 6 goodies!

    2017.37 Collating Sorted

    Samantha McVey completed the Unicode Collation Algorithm support in Rakudo Perl 6 on the MoarVM backend. She describes the functionality in the commit that did the merge back into the main branch of MoarVM. She also gave a less complete review on the #perl6 channel. Also check out the temporary documentation! A code example may tell more than a thousand words:

    say <9 ① 1 ② 8 one hi>.sort;    # (1 8 9 hi one ① ②)
    use experimental :collation;     # activate collation
    say <9 ① 1 ② 8 one hi>.collate; # (1 ① ② 8 9 hi one)

    TPF Grant Proposals (Sep 2017)

    Will Coleda tells us we have only a few days left for this round of TPF Grant Proposals. So if you would like to do some development work on Rakudo Perl 6, write a proposal, do your thing and profit! Well, be sure to write a good proposal that will get accepted, of course 🙂

    Handling of unclosed files has changed

    The ecosystem started showing issues (RT #132030) after non-TTY handles had buffering switched on by default last week. Solution #1 (as suggested by Jonathan Worthington) was subsequently implemented by Elizabeth Mattijsen. What does this mean?

    • If you don’t care about keeping resources tied up, just open your files for reading and writing. When the program exits, any buffers to be written will be automatically flushed before actually returning to the OS. This approach is fine for most applications, unless the application is about opening a lot of files: in that case you run the risk of running into the “too many open handles” OS resource error. Please note that if you’re using any of the IO::Path methods for reading or writing, or use slurp() or spurt() you do not have to worry about this, as those methods/subs will close the handle for you when they’re done.
    • If you do care about your resources, make sure that any open file will be closed as soon as you’re done with it. This can be as simple as having a LEAVE phaser in the scope where the file is opened, or by adding the will leave trait to the variable containing the opened handle.

    Please note that this behaviour only applies to the MoarVM backend. The old semantics of only automatically closing files when they are garbage collected, still applies to the JVM backend. But since there is no output buffering on the JVM backend, you only run the risk of running out of open handles.

    Blog Posts

    Other Core Developments

    • Christian Bartolomäus started unbitrotting the JVM backend. Pretty sure he would appreciate more eyes and more hands on that!
    • Zoffix Znet made sure that .make can now pass on type objects again.
    • Jonathan Worthington fixed the scoping issues on "foo{$x}bar", which could show up with recursive calls inside the {} and/or when being run by multiple threads at the same time.
    • John Harrison fixed an issue with IO::Notification.watch-path($dir).
    • Elizabeth Mattijsen cleaned up program end / exit() handling to ensure they are identical, and that any END blocks are guaranteed to be only run once by a single thread. She also reduced the memory usage of the BUILDALLPLAN, which affects the memory footprint of each class: this may also make building objects slightly faster in some situations.
    • Timo Paulssen continued his work on the heap analyzer and also re-instated the literal junction optimizer.
    • And many other smaller fixes and improvements.

    Meanwhile on Twitter

    Meanwhile on StackOverflow

    Meanwhile on perl6-users

    Ecosystem Additions

    Winding down

    An exciting week once again! Next weekend will see the 2017.09 release of Rakudo Perl 6. After the release, a number of optimizer and JIT improvements will find their way into the main branch again. So check back next week for more exciting Rakudo Perl 6 news!