2017.42 Taking Ticketing Seriously

Aleks-Daniel Jakimenko-Aleksejev has been ticketing away! The result can be seen in an overview of status changes in RT tickets of the past week or so. He explains it like this:

The high number of [REGRESSION] tickets is caused by the concentrated effort to detect any unintentional changes in Rakudo Perl 6. I extracted all messages to the evalbot (e.g. “m:” on the #perl6 channel) from the last few years, and passed these code snippets to different versions of Rakudo Perl 6. These snippets were filtered by comparing the output and exit code, which resulted in a list of thousands of samples that produce different results on different versions. Most of these differences are due to bug fixes or other harmless reasons (e.g. current time being shown in the output), but there were some notable regressions. All regressions noticed during the manual inspection are now filed, and some of them are already resolved (with tests, of course). This is the second time we see dedicated effort to find otherwise hard-to-notice regressions, first time being in December 2016.

So it should be easier to keep the sanity of Rakudo Perl 6 in check!

One more TPCiA Rakudo Perl 6 Video

Samantha McVey‘s High End Unicode in Rakudo Perl 6 has arrived! Of course, you can still keep checking for more uploads in the coming weeks.

Blog Posts

Meanwhile on Twitter

  • Please welcome, CPAN6 by builtinperl.
  • Rakudo Perl 6 BuildPack by brian d foy.
  • 20% shaved off by Zoffix Znet.
  • Debugging an Amazon issue by Moritz Lenz.
  • Whole day Rakudo Perl 6 Tutorial at TPCiG by Jeff Goff.
  • !perl6 at DuckDuckGo by Zoffix Znet.
  • int32.new shouldn’t give you an Int by Zoffix Znet.
  • Meanwhile on StackOverflow

    Meanwhile on perl6-users

    Meanwhile on PerlMonks

    Lately, there have been some Rakudo Perl 6 related Meditations on PerlMonks. Yours truly feels they should be mentioned in the Perl 6 Weekly as well, although some troll hugging may be needed:

    Ecosystem Additions

    This week will probably be the last time you will see the ecosystem additions mentioned in the Perl 6 Weekly. Thanks to Zoffix Znet and Aleks-Daniel Jakimenko-Aleksejev we will soon have a page on that will show the recent uploads to CPAN (the new preferred way putting your module out into the wild). So here goes for the last time:

    Winding Down

    Being too tired to investigate the latest Core Developments of the past week, I will keep them from you until next week. Which will also coincide with the 2017.10 release of Rakudo Perl 6! So please check in again next week!

    Advertisements

    2017.41 The Case for Empathy

    Yours truly is glad she can finally link to Ruth Holloway‘s impressive The Case for Empathy keynote at The Perl Conference in Amsterdam. Please excuse the poor sound quality for the first 7 minutes or so: kudos to Amanda who managed to recover and process the sound from that period and supply the subtitles.

    Other videos of TPCiA related to Rakudo Perl 6 are:

    It is expected that more videos will be uploaded in the coming week, so please check again from time to time!

    CPAN6 is here

    Zoffix Znet did a lot of work on getting Rakudo Perl 6 modules out of CPAN, just as he promised last week. He explains it all in his blog post CPAN6 is here (Reddit comments). So now we finally have a front-end to browse the available Rakudo Perl 6 distributions that have been uploaded to CPAN!

    The Second Bug Squashathon

    Last weekend saw the second Community Bug Squashathon, this time focused on fixing problems in the core of Rakudo Perl 6. And the results are in! Looking forward to the next Squashathon, on 4 November 2017!

    Inside the head of Larry Wall

    Joe Armstrong conducted a Q&A with Larry Wall at the Erlang & Elixir Factory SF Bay Area 2017 conference (Hacker News comments). Also recommended as a very nice intro to Rakudo Perl 6 and the thoughts that have gone into the 100 year language.

    Rakudo Perl 6 performance analysis tooling

    Timo Paulssen‘s grant proposal for Rakudo Perl 6 performance analysis tooling has been approved by the TPF Grant Committee. Yours truly can hardly wait to see the results of this grant come to fruition!

    Call for Rakudo Perl 6 Papers!

    This is the time to propose your Rakudo Perl 6 presentations! Please select one or more conferences:

    Core Developments

    • The new JIT was merged into MoarVM, and consequently into Rakudo’s main branch. Some teething issues were already fixed by Jonathan Worthington. The new JIT hasn’t yet given us any large speed improvements, but now more people can work on making the JIT smarter and thereby making Rakudo Perl 6 faster! Yours truly is looking forward to be able to help here. Interested in working on this as well? Check out the documentation.
    • Zoffix Znet, with some help of Dagfinn Ilmari Mannsåker made EVAL/EVALFILE aware of Buf and the utf8-c8 encoding, as was originally stipulated in S29. He also worked on making .gist on several classes a lot faster, and re-enabled the :bin parameter on slurp.
    • Daniel Green made sure that enum values are now also included in errors suggesting identifiers when an identifier could not be found.
    • Larry Wall improved several parsing error messages, specifically on the likely location of the error, e.g. by catching missing expressions on statement modifiers earlier. Christian Bartolomäus and Aleks-Daniel Jakimenko-Aleksejev did similar work on improving the error message formerly known as “Malformed my”.
    • Zoffix Znet added a --repl-mode parameter to Rakudo Perl6 6, specifically meant to make testing the REPL easier. He also created new REPL tester logic and adapted the tests to use the new REPL tester logic.
    • Aleks-Daniel Jakimenko-Aleksejev also improved many error messages related to mismatched quotes.
    • Jonathan Stowe provided several fixes related to NativeCall and also provided tests for those fixes.
    • Elizabeth Mattijsen made object creation where attributes have compile-time default values (such as has $.foo = 42) about 40% faster. She also improved bare startup time by making code, that would normally run during startup, be pre-compiled in the setting.
    • Zoffix Znet, with some persistent help of Aleks-Daniel Jakimenko-Aleksejev, made array slices with a lazy infinite list (such as @a[3..*]) about 4x faster.
    • And many more smaller fixes and improvements.

    Other Blog Posts

    Meanwhile on Twitter

    Meanwhile on perl6-users

    Winding Down

    Quite an eventful week again, with the JIT being merged and the Squashathon going on. More than 100 commits in the rakudo repository alone. Please check in again next week for more Rakudo Perl 6 news!

    2017.40 Unicode Granted

    Samantha McVey has completed her work on the “Improving the Robustness of Unicode Support” Grant. In an extensive blog post she describes what she has done in the past months, how all achievable goals were achieved, and how quite a number of bonus features also made it. You can leave any congratulations, support or other comments on the official TPF page about the completion of the grant. I think we can now safely say that Rakudo Perl 6 has the most complete Unicode support of any programming language in the world. Kudos to Samantha!

    Object creation 1.5x faster

    Elizabeth Mattijsen (with a lot of help by Timo Paulssen and Jonathan Worthington) completed her work on auto-generating the BUILDALL method for each class. This is an internal method that is called to create an object using the default .new method. Instead of interpreting a build plan of the class every time an object of that class is created, a custom BUILDALL method is created directly using QAST ops (basically writing source code in Abstract Syntax Trees and then “EVALling” them). This makes each call to .new 1.5x faster, making our canary about 8% faster!

    Nativecall also much faster

    Stefan Seifert also finished his work on auto-generating native call function bodies, making the interface between Rakudo Perl 6 internals and Inline::Perl5 much faster. This also made the Inline::Perl5 version of our canary about 10% faster. The race continues to be on!

    Roast Versioning

    Rakudo Perl 6 has a file that contains a list of test-files (roast) it should run (spectest.data) when doing a so-called spectest. A single file used to be used for all language versions, which is bad because:

    • New files get added to roast master and roast runner complains they’re missing when running tests for older language versions.
    • Files in master get restructured, but deleted files still have to live as dummy files when tests for another language version get run.

    Zoffix Znet implemented roast versioning by making a separate list of files for each language version so that Rakudo Perl 6 can run tests for each language version seperately. Roast runner just looks at VERSION file in roast to figure out which version of the list of test-files to load (the file list is now in spectest.data and spectest.data.6.c; for a future release a spectest.data.6.d file will be added).

    The language that will not be named

    Zoffix Znet rekindled the discussion about the naming of Rakudo Perl 6 again, this time suggesting 6lang (to be pronounced slang). Again, quite some discussion on blogs.perl.org, /r/perl6, /r/perl and Twitter. Yours truly hopes we can settle this once and for all before Rakudo Perl 6 language version 6.d comes out (or whatever we will call it then).

    Hacktoberfest is here again

    It’s the month of October again and your chance to win a T-shirt! Simply sign up, have a look at the issues at hand, fix only 4 of them and profit!

    Community Bug SQUASHathon

    On Saturday 7 October we will have yet another Community Bug SQUASHathon. Please join us on #perl6-dev on freenode and help us squash those pesky core bugs.

    Texas to ASCII

    To avoid any political connotations, it was felt that we should be moving towards documenting operators consisting purely of ASCII characters as “ASCII” operators (such as (<=)), and operators that consist of at least 1 non-ASCII character (such as ) as “Unicode” operators. Internal documentation has already been adapted, external documentation will follow shortly unless consensus changes.

    Other Core Developments

    • Zoffix Znet made chainable operators (such as <: 2 < 3 < 4) between 6x and 36x times faster (depending on number of items in the chain). He also made cmp on Version objects about 7x faster. Which is nice as we will need to do a lot more language version comparisons in the future!
    • Jonathan Worthington fixed several race conditions and deadlocks with the new ThreadPoolScheduler. He also made sure that all aspects of the new ThreadPoolScheduler are now used on 6.c, except the part of await no longer blocking the thread it is awaiting in.
    • Samantha McVey fixed an issue with uniname and non-unique Unicode names. The noncharacter property is now also returned for noncharacters.
    • Zoffix Znet also fixed and improved the ** regex quantifier and made sure that chdir respects the :CWD named parameter.
    • And quite some other smaller fixes and improvements.

    Other Blog Posts

    Meanwhile on Twitter

    Meanwhile on StackOverflow

    Meanwhile on perl6-users

    Ecosystem Additions

    Winding Down

    Bart Wiegmans has created a Pull Request for all of the work that has done on the new JIT: this is now going through the final reviewing process. This work will be merged any day now, which should allow for quite some more “so many percent faster” items in the next issue. So please check in again next week!

    2017.39 Smarting up the Pool

    Jonathan Worthington elaborates on his new ThreadPool implementation in an extensive blog post, which was made possible by the sponsoring of Vienna.pm. A recommended read if you want to keep up to date on the features and performance of asynchronous processing with Rakudo Perl 6! In short, there are now 3 queues internally: one for timers, one for workers that sequentially process, and one for general purpose activity. There’s now also a “Sufficiently Smart Supervisor” thread that looks at the state of the other threads about 100x per second and decides if adding more threads would make sense or not. Exciting times!

    Performance analysis tooling

    Timo Paulssen submitted a grant proposal to improve the performance analysis tools of Rakudo Perl 6. Judging by the comments so far, he’s not the only one wanting to see this happen. Please leave your comments if you also have an opinion about this proposal!

    Other Core Developments

    Jonathan Worthington was responsible for the most of last weeks commits. I guess everybody else was just watching in awe, or busy with things that didn’t make it into the main rakudo repo just yet. Notable exceptions are:

    • Samantha McVey finally fixed the last bugs in the utf8-c8 encoding: this is the encoding that tries to encode as utf8, but which creates temporary synthetic codepoints for those sequences that are not valid UTF-8, instead of giving up.
    • David Warring made sure that subtypes actually have an .isa method.
    • And some other minor fixes and improvements.

    Other Blog Posts

    Meanwhile on Twitter

    Meanwhile on StackOverflow

    Meanwhile on perl6-users

    Ecosystem Additions

    Winding Down

    Apart from all the visible work Jonathan and Samantha have done this week, there was a lot going on behind the scenes. One of them promises to make object creation in Rakudo Perl 6 at least 10% faster. Too bad we couldn’t report about it this week. So be sure to check out the next issue of the Perl 6 Weekly!

    2017.38 Color Me Booked

    There’s been a lot of book activity in the Perl 6 world lately. Andrew Shitov announced his new book Perl 6 Deep Dive (preliminary table of contents). And Moritz Lenz also continued working on his “Parsing with Perl 6 Regexes and Grammars” book. To top it off this week, Zoffix Znet announced the Rakudo Book Project – a plan to write some Rakudo books (/r/perl and /r/perl6 comments).
    Butterflies Galore!
    Please check out his plans and support him in any way you can!

    AlexDaniel++ for his second release

    Aleks-Daniel Jakimenko-Aleksejev has done his second Rakudo compiler release! The announcement for Rakudo Perl 6 2017.09 shows quite a number of fixes and improvements again this month. Please note there are currently no plans for creating a Rakudo Star release for this compiler release.

    London Perl Workshop – 25 November

    Saturday 25 November will see another London Perl Workshop 2017. And yours truly would love to see a lot of Rakudo Perl 6 presentations there: so please submit your presentation. Hope to see you there!

    New ThreadPoolScheduler implementation

    Jonathan Worthington started work on a new thread pool scheduler (which got merged after the 2017.09 release because of possible ecosystem fallout). This implementation has separate general and timer queues with separate workers, and also introduces affinity queues, which are intended for cases where events will be fed into a Supply, and thus there’s no point having lots of threads competing over them only to immediately stumble over each other. The separate timer queue helps when timer events are being delayed, for example if a process is producing a load of output.

    This implementation also adds a supervisor, which is where the smarts on how many threads to have in the pool will be put. For now, it is already smart enough to start a lot less threads than the previous scheduler when they obviously aren’t needed. This helps with memory consumption. And it can add more threads on demand when needed to break deadlocks. The default maximum number of threads has been raised to 64, now that the scheduler does not start up the maximum number of threads even when they don’t have any work to do.

    For debugging, the RAKUDO_SCHEDULER_DEBUG and RAKUDO_SCHEDULER_DEBUG_STATUS environment variables can be set.

    This work has been kindly sponsored by the Vienna Perl Mongers.

    Other Core Developments

    These features made it to the 2017.09 compiler release.

    • Samantha McVey marked the .collate method, coll and unicmp infix functions as no longer experimental. The dynamic variable $*COLLATION, which allows you to configure the sort, will remain as experimental for now.
    • Elizabeth Mattijsen made simple (one attribute) object creation about 25% faster, and reduced memory requirements for classes.
    • Philippe Bruhat worked on .succ and .pred on enums. This resulted in more attention to this subject, with the outcome a .WHICH on enums that is O(1).
    • Timo Paulssen fixed a closure issue with permutations and combinations.
    • Aleks-Daniel Jakimenko-Aleksejev fixed needless buffering in Test, causing test information not be updated continuously when running the test in a harness.
    • And quite a few smaller fixes and improvements.

    Other blog posts

    Meanwhile on Twitter

    Meanwhile on StackOverflow

    Meanwhile on perl6-users

    Ecosystem Additions

    Quite a nice catch this week!

    Winding down

    Between the rain and the wind, quite a lot happened in the Rakudo Perl 6 world yet again. Sometimes we forget how many ways we found how not to make Perl 6. With that in mind, see you next week for more Rakudo Perl 6 goodies!

    2017.37 Collating Sorted

    Samantha McVey completed the Unicode Collation Algorithm support in Rakudo Perl 6 on the MoarVM backend. She describes the functionality in the commit that did the merge back into the main branch of MoarVM. She also gave a less complete review on the #perl6 channel. Also check out the temporary documentation! A code example may tell more than a thousand words:

    say <9 ① 1 ② 8 one hi>.sort;    # (1 8 9 hi one ① ②)
    use experimental :collation;     # activate collation
    say <9 ① 1 ② 8 one hi>.collate; # (1 ① ② 8 9 hi one)

    TPF Grant Proposals (Sep 2017)

    Will Coleda tells us we have only a few days left for this round of TPF Grant Proposals. So if you would like to do some development work on Rakudo Perl 6, write a proposal, do your thing and profit! Well, be sure to write a good proposal that will get accepted, of course 🙂

    Handling of unclosed files has changed

    The ecosystem started showing issues (RT #132030) after non-TTY handles had buffering switched on by default last week. Solution #1 (as suggested by Jonathan Worthington) was subsequently implemented by Elizabeth Mattijsen. What does this mean?

    • If you don’t care about keeping resources tied up, just open your files for reading and writing. When the program exits, any buffers to be written will be automatically flushed before actually returning to the OS. This approach is fine for most applications, unless the application is about opening a lot of files: in that case you run the risk of running into the “too many open handles” OS resource error. Please note that if you’re using any of the IO::Path methods for reading or writing, or use slurp() or spurt() you do not have to worry about this, as those methods/subs will close the handle for you when they’re done.
    • If you do care about your resources, make sure that any open file will be closed as soon as you’re done with it. This can be as simple as having a LEAVE phaser in the scope where the file is opened, or by adding the will leave trait to the variable containing the opened handle.

    Please note that this behaviour only applies to the MoarVM backend. The old semantics of only automatically closing files when they are garbage collected, still applies to the JVM backend. But since there is no output buffering on the JVM backend, you only run the risk of running out of open handles.

    Blog Posts

    Other Core Developments

    • Christian Bartolomäus started unbitrotting the JVM backend. Pretty sure he would appreciate more eyes and more hands on that!
    • Zoffix Znet made sure that .make can now pass on type objects again.
    • Jonathan Worthington fixed the scoping issues on "foo{$x}bar", which could show up with recursive calls inside the {} and/or when being run by multiple threads at the same time.
    • John Harrison fixed an issue with IO::Notification.watch-path($dir).
    • Elizabeth Mattijsen cleaned up program end / exit() handling to ensure they are identical, and that any END blocks are guaranteed to be only run once by a single thread. She also reduced the memory usage of the BUILDALLPLAN, which affects the memory footprint of each class: this may also make building objects slightly faster in some situations.
    • Timo Paulssen continued his work on the heap analyzer and also re-instated the literal junction optimizer.
    • And many other smaller fixes and improvements.

    Meanwhile on Twitter

    Meanwhile on StackOverflow

    Meanwhile on perl6-users

    Ecosystem Additions

    Winding down

    An exciting week once again! Next weekend will see the 2017.09 release of Rakudo Perl 6. After the release, a number of optimizer and JIT improvements will find their way into the main branch again. So check back next week for more exciting Rakudo Perl 6 news!

    2017.36 Blogs-a-reading

    In his compelling blog post titled “On Troll Hugging, Hole Digging, and Improving Open Source Communities (How to be Better)”, Zoffix Znet describes how to make Open Source communities even better, building on Audrey Tang’s “hug a troll” concept.

    …here’s a well-shaped hole. Everyone’s more connected. It’s easy to get in and start digging. And even those who dug the deepest can still go and help out those who are about to start.

    Example of a well shaped hole

    The hole digging metaphor isn’t just about the shape of the hole. It’s also about people’s position within it.

    He also introduces the seven Hugs for a better community:

    1. Gift a Shovel
    2. Feed The Hand That Bites You
    3. We All Leave Footprints
    4. Speak Up
    5. Simply a Hug
    6. Love Others
    7. Go For The Third Option

    Zoffix shows in his examples that we can all learn to become better. And that he finds himself to be a lot happier because of this learning process:

    …I’ve been applying the ideas I discussed in this article for about a week. I think they have something real behind them, as I feel a lot happier now than a week ago and I see some positive changes around me that I think I could attribute to these ideas

    A well recommended read!

    P6::Journey

    Thanks to the pingback functionality, yours truly was pointed to a new blog about Rakudo Perl 6: P6::Journey by p6steve. His blog posts so far:

    All yours truly can add is: Welcome to the Rakudo Perl 6 Blogoverse!

    Unicode Grant update #4

    Samantha McVey continued working on the Unicode support in Rakudo Perl 6 and reports on her progress in a blog post. It describes how using the Knuth-Morris-Pratt String Search Algorithm improved string searching, how looking for a single grapheme can be 9x faster in some situations, that the database is updated to Unicode 10, and that several bugs were fixed and documentation improved. Among many other things. A good read if you want to keep up-to-date in that area of Rakudo Perl 6!

    Swiss Perl Workshop in review

    Steve Mynott created an excellent diary-like blog-post about the Swiss Perl Workshop, with lots of pictures. Looking forward to many more of these in the future!

    In related news, the videos of the Swiss Perl Workshop have been made available by Lee Johnson. The Rakudo Perl 6 related presentations for which videos were made, are:

    Other blog posts

    Other videos

    Some more recent than others, but these somehow slipped through the cracks the past weeks.

    Other core developments

    This week yours truly was reminded that a lot can happen in two weeks. Let me try to recap this in a way that is not overly long without letting important things fall through the cracks:

    • Andrew Ruder fixed several issues with Supply.batch. Elizabeth Mattijsen continued on that work by making sure Lists are returned rather than Arrays, which saves an extra copying step.
    • andreoss made it possible to pipe Perl 6 source code to perl6 without having the REPL being invoked.
    • Daniel Green made the handling of the ignorecase and ignoremark when interpolating strings into regexes, much faster thanks to the new nqp::eqatic, nqp::eqatim and nqp::eqaticim opcodes added by Samantha McVey.
    • Jonathan Worthington did a lot of work on MoarVM, about which he will surely elaborate in a soon to be seen blog post. Meanwhile, he also made sure that --ll-exception flag is also honoured in the case of a broken Promise returned by a start block. And he activated output buffering for non-TTY handles by default. This (and other optimizations) makes the writing a million lines benchmark only 1.2x slower than the Pumpking Perl 5 version.
    • Elizabeth Mattijsen made sure that you can now concatenate a string to a Junction, or concatenate two Junctions. This also implies that you can now interpolate a Junction into a string, and that List.join is no longer guaranteed to return a Str (because it will return a Junction if one of the elements of the List was a Junction).
    • And many other fixes and improvements that aren’t not really visible yet. Such as the merging of a lot of the work that Bart Wiegmans has done on the new JIT, and the work that Timo Paulssen is doing on speeding up the MoarVM heap analyzer.

    Meanwhile on FaceBook

    • Jeff Goff said: 0th cut of perl6-Grammar-Common up on GitHub – https://github.com/drforr/perl6-Grammar-Common. Add also does Grammar::Common::Expression::Infix to your grammar, supply a token and you get an rule that you can drop anywhere you want a simple but complete mathematical expression. It provides both grammar rules and some simple actions to generate a basic .ast consisting of nested hashes that you can walk on your own, or override the provided methods and generate your own object tree.
    • Anushka Jodha asked: “Can we use Rakudo Perl 6 with Angular 2 like typescript and scala.js ???”. Jeff Goff answered: “Depends upon what you mean by “use” – It’s always been possible for Pumpking Perl 5 and Angular to coexist, what this project does is adds routes to Bailador that lets it serve your Angular 2 code. What I intend to do with this is add a Rakudo Perl 6 wrapper around whatever tool that generates the Angular 2 code which puts a Bailador route in place, then shells out to the Angular toolkit.”

    Meanwhile on Twitter

    Meanwhile on StackOverflow

    Meanwhile on perl6-users

    Ecosystem Additions

    Winding Down

    Hope I didn’t miss too much this week. Please let me know if I did. Hope to see you all again next week for more Rakudo Perl 6 news!